AWS SQS

The AWS SQS sink publishes worker log events to Amazon Simple Queue Service (SQS) as individual messages (stream egress). This is a good fit when you want to decouple producers from consumers, absorb bursts, and hand off logs to downstream processors (Lambda, ECS/EKS workers, SIEM forwarders, ETL jobs) with a queue-based reliability model.

Because SQS is a queue (not a database/search backend), this sink is typically used as a transport/ingress layer in a larger pipeline rather than the final storage or query system.

Logs

Before configuring this sink, you typically need:

An existing SQS queue and its Queue URL AWS credentials available to Kron TP(instance role, task role, IRSA, env vars, or credential file) IAM permissions allowing at minimum SendMessage to the queue (and possibly additional permissions depending on how credentials are resolved) A decision on whether you’re using a Standard queue or a FIFO queue (FIFO impacts ordering/dedup options)

Queue URL (required) The target SQS queue URL where worker sends messages.

This is the primary routing parameter and uniquely identifies the destination queue in AWS.

Region (optional) The AWS region of the target SQS queue.

If omitted, the region is typically inferred from the environment/credentials configuration, but setting it explicitly can prevent surprises in multi-region deployments.

Endpoint (optional) A custom AWS endpoint used for AWS-compatible services or private/test environments.

This is mainly used for non-default routing (e.g., AWS-compatible stacks, local testing, or constrained network paths).

Auth (optional) Controls how worker obtains AWS credentials and (optionally) assumes roles.

Common patterns include:

Using the default credential chain (instance/task/IRSA) for production Using explicit access keys for development (less preferred operationally) Assuming a role for cross-account queue publishing

The key design goal here is: make worker's identity least-privileged (only send to the intended queue).

Encoding (required) Controls how worker serializes each log event into the SQS message body.

This is a critical choice because your downstream consumer determines what “good” looks like:

JSON encoding is typical for structured pipelines Text/raw message encoding is common when you want minimal transformation and downstream parsing

Practical rule: pick an encoding that your consumers can process without ambiguity, and keep it stable (changing formats can break consumers).

If your destination is a FIFO queue, SQS introduces ordering and deduplication semantics, and the sink exposes knobs to populate the related fields:

Message Group ID (optional) Defines the “ordering lane” for FIFO queues. Messages within the same group are processed in order.

Use this when you want ordering per tenant/service/source (for example: per namespace, per app, per customer) while still allowing parallelism across groups.

Message Deduplication ID (optional) Provides a deduplication identifier so SQS can detect duplicates over the FIFO deduplication window.

This is useful when retries or upstream replay could produce repeated messages and you prefer consumers not to see duplicates.

Buffer (optional) Controls how worker buffers events before publishing them to SQS.

This is how you absorb short outages or throttling without immediately dropping data:

Memory buffering: faster, but volatile across restarts Disk buffering: more durable for transient AWS/network issues, at the cost of disk I/O

Buffer behavior also determines whether worker blocks (backpressure) or drops when the buffer is full, which is a loss-vs-latency tradeoff.

Proxy (optional) Routes SQS traffic through HTTP/HTTPS proxies, and supports bypass rules.

This is important in enterprise environments where outbound traffic must traverse controlled egress points.

TLS (optional) Controls certificate verification, custom CAs, and related secure transport settings.

In most production cases you keep verification enabled, and only customize when you have private endpoints, custom trust bundles, or unusual certificate chains.

Queue as an ingestion bus Publish logs into SQS, then have multiple consumers: one for enrichment, one for indexing, one for long-term archive.

Burst absorption Let SQS absorb spikes while downstream processing scales horizontally.

Cross-account / multi-tenant pipelines Use role assumption + per-tenant queues to isolate workloads and blast radius.

FIFO for ordered streams Use FIFO with message group IDs when consumer logic depends on per-entity ordering (careful: ordering usually reduces maximum parallel throughput compared to standard queues).