AWS CloudWatch Logs

The AWS CloudWatch Logs sink is used to publish log events collected by Kron TP to Amazon CloudWatch Logs. It is designed for production use and provides at-least-once delivery semantics, batching for efficiency, and optional end-to-end acknowledgements.

This sink operates in a stateless manner and is intended for log data only.

When enabled, Kron TP batches log events and sends them to CloudWatch Logs using the AWS Logs API. Events are grouped into log groups and log streams. Each log stream must have exactly one active writer at any time.

If multiple worker instances are deployed, stream naming must guarantee uniqueness per instance to avoid write conflicts.

group_name Specifies the CloudWatch Log Group where logs are written. This value supports templating, allowing dynamic group assignment based on event fields. The log group must already exist unless automatic creation is enabled.

stream_name Specifies the CloudWatch Log Stream within the log group. This value also supports templating. Because CloudWatch allows only one writer per stream, stream names must be unique per worker instance, host, or container.

create_missing_group When enabled, worker automatically creates the log group if it does not already exist. This is enabled by default.

create_missing_stream When enabled, worker automatically creates the log stream if it does not already exist. This is enabled by default.

retention Controls whether a retention policy is applied when a new log group is created. If enabled, logs are automatically deleted after the configured number of days. This setting has no effect on already existing log groups.

kms_key Specifies a customer-managed AWS KMS key ARN used to encrypt log data at rest in CloudWatch Logs.

tags Defines key-value tags applied to the log group and log stream. Tags can be used for cost allocation, governance, or automation.

The sink supports all standard AWS authentication mechanisms and follows the AWS SDK credential resolution order unless explicitly overridden.

Static credentials Access key ID, secret access key, and optional session token can be provided directly. This is typically discouraged for production environments.

Credentials file and profile Allows selecting credentials from a shared AWS credentials file using a named profile.

Assume role Allows Kron TP to assume an IAM role via STS. An optional external ID can be provided for cross-account access. A session name can be specified or auto-generated.

Instance Metadata Service (IMDS) When running on EC2, ECS, or EKS, Kron TP can retrieve credentials automatically from the instance or pod role. IMDS timeout and retry behavior can be tuned.

Region Defines the AWS region used for CloudWatch Logs and STS calls. If not explicitly set, Kron TP uses the region associated with the target service.

Custom endpoint Allows sending logs to AWS-compatible services or private endpoints, such as LocalStack or AWS Outposts.

The encoding configuration determines how log events are converted into bytes before being sent to CloudWatch Logs.

codec Defines the serialization format. Common options include JSON, plain text, logfmt, CSV, GELF, CEF, Avro, Protobuf, OTLP, and raw message passthrough.

JSON encoding Can be configured for compact or pretty output. Timestamp format can be customized.

Text and raw message encoding Uses the event’s message field directly. If the message field is removed upstream, the output may be empty.

Structured formats (CEF, GELF, Avro, Protobuf) Require additional schema or field mappings. These are typically used for SIEM or security platforms that expect strict formats.

Field inclusion and exclusion Allows explicitly selecting which event fields are included or excluded from the output.

Worker batches events before sending them to CloudWatch Logs to improve efficiency and reduce API calls.

Batching behavior is controlled by maximum batch size in bytes, maximum number of events, and maximum batch age. A batch is flushed when any of these thresholds is reached.

This batching is applied before compression and serialization.

The sink supports both in-memory and disk-based buffering.

Memory buffering Provides high performance but does not survive restarts or crashes.

Disk buffering Persists events to disk and provides durability across restarts. This mode requires a minimum buffer size and periodically syncs data to disk.

When full behavior Controls whether the worker blocks upstream components when the buffer is full or drops incoming events. Blocking preserves data at the cost of backpressure, while dropping favors throughput.

The sink provides at-least-once delivery semantics. Duplicate events are possible if retries occur.

Acknowledgements When enabled, worker waits for CloudWatch Logs to acknowledge successful ingestion before acknowledging events upstream. This allows end-to-end delivery guarantees when sources support acknowledgements.

Sink-level acknowledgement settings override global acknowledgement configuration.

Kron TP includes advanced outbound request controls to protect both CloudWatch Logs and itself.

Adaptive concurrency Automatically adjusts the number of concurrent requests based on observed latency. This is the default and recommended mode.

Fixed concurrency Allows explicitly setting a concurrency limit or forcing single-request operation.

Retries Failed requests are retried using exponential backoff with optional jitter. Retry limits and maximum backoff duration can be configured.

Rate limiting Limits the number of requests per time window to avoid API throttling.

Timeouts Defines how long the worker waits before aborting a request. This should not be set below AWS service timeouts.

Optional compression can be applied to batched payloads before transmission. Supported algorithms include gzip, zlib, snappy, and zstd. Compression reduces bandwidth usage but increases CPU consumption.

Outbound traffic can be routed through HTTP or HTTPS proxies. Proxying can be selectively disabled for specific hosts or CIDR ranges.

TLS settings control how worker validates CloudWatch endpoints.

Certificate verification and hostname verification are enabled by default and should not be disabled unless absolutely necessary. Custom CA certificates and client certificates can be provided for advanced environments.

CloudWatch Logs enforces strict API limits and sequencing rules. Each log stream accepts events in strict timestamp order and only one writer is allowed. Always ensure stream naming guarantees uniqueness.

For high-throughput environments, prefer multiple streams over large batches, and use adaptive concurrency.