AWS SQS
AWS SQS Source
The AWS SQS source collects events from Amazon Simple Queue Service (SQS) and converts queued messages into observability signals. Depending on the payload and decoding configuration, this source can emit logs, metrics, or traces.
It is designed to run in an aggregator role and is commonly used to centralize data produced by distributed systems, serverless workloads, and decoupled application components.
Collection Model
- Pull-based polling of SQS queues
- Concurrent consumers fetch messages from the queue
- Messages are decoded into structured events
- Successfully processed messages are deleted from the queue
The source operates statelessly and relies on SQS visibility timeouts to guarantee reliable delivery.
Queue Configuration
queue_url (required, string)
Specifies the URL of the SQS queue to poll for messages.
Operational considerations:
- The queue must exist and be accessible
- Permissions must allow ReceiveMessage, DeleteMessage, and ChangeMessageVisibility
- FIFO and standard queues are both supported
Authentication and Authorization
auth (optional, object)
Defines how the source authenticates with AWS services.
Supported authentication strategies include:
- Static credentials (access key and secret key)
- IAM roles with assume_role
- AWS credentials files and named profiles
- Instance Metadata Service (IMDS)
The source follows AWS best practices and supports the standard AWS credential resolution chain.
IMDS Configuration
When running on EC2, ECS, or EKS, credentials can be automatically retrieved via IMDS.
IMDS-related settings control:
- Connection timeout
- Read timeout
- Retry attempts when fetching credentials
Region and Endpoint Resolution
region (optional, string)
Defines the AWS region for the target SQS service.
If not set, the region is inferred from:
- The queue URL
- Environment configuration
- AWS SDK defaults
endpoint (optional, string)
Allows overriding the AWS endpoint.
This is typically used for:
- AWS-compatible services
- Local testing environments
- Private cloud or emulated SQS endpoints
Message Polling Behavior
poll_secs (optional, uint)
Controls how long the source waits while polling the queue.
Notes:
- Messages are consumed immediately when available
- Increasing this value does not slow down consumption
- Generally should not be modified unless explicitly required
client_concurrency (optional, uint)
Controls the number of concurrent polling tasks.
Higher values may be beneficial when:
- Message throughput is high
- Messages are small
- CPU resources are underutilized
visibility_timeout_secs (optional, uint)
Defines how long a message remains invisible after being received.
If processing exceeds this duration:
- The message becomes visible again
- Another consumer may reprocess it
- This can lead to duplicate events
The timeout should be aligned with worst-case processing time.
Message Lifecycle Control
delete_message (optional, bool)
Controls whether messages are deleted after successful processing.
Disabling deletion can be useful for:
- Debugging
- Validation during initial setup
- Replay scenarios
In production environments, this should remain enabled.
Decoding Configuration
decoding (optional, object)
Defines how raw message payloads are decoded into events.
The decoding stage determines:
- Event type (log, metric, trace)
- Field structure
- Timestamp parsing
- Schema validation behavior
Supported Codecs
The source supports multiple decoding codecs, including:
- Raw bytes
- JSON
- Syslog
- GELF
- InfluxDB line protocol
- Protobuf
- Avro
- OTLP (logs, metrics, traces)
- Native Vector formats
- VRL-based custom decoding
Some codecs are experimental and may evolve over time.
VRL Decoding
When using VRL decoding:
- Each message is passed through a VRL program
- The final .target defines the decoded event
- Compilation or runtime errors cause decoding failures
This approach enables advanced transformation and enrichment during ingestion.
Framing Configuration
framing (optional, object)
Controls how raw byte streams are split into individual messages.
Supported framing methods include:
- Byte-based framing
- Character-delimited
- Newline-delimited
- Length-delimited
- Octet-counting
- Chunked GELF
- Varint length-delimited
Framing configuration is critical when messages contain multiple events or non-trivial binary formats.
Proxy Support
proxy (optional, object)
Configures HTTP(S) proxy usage for AWS API requests.
This is useful in:
- Restricted network environments
- Controlled egress architectures
- Enterprise proxy setups
Proxy bypass rules can be defined per host or network.
TLS Configuration
tls (optional, object)
Controls TLS behavior for outgoing connections.
Capabilities include:
- Custom CA certificates
- Client certificate authentication
- ALPN protocol configuration
- Certificate and hostname verification
Disabling certificate or hostname verification is strongly discouraged in production.
Acknowledgements and Delivery Semantics
- End-to-end acknowledgements are supported
- Source-level acknowledgement settings are deprecated
- Reliability is managed at the global or sink level
- Duplicate delivery is possible under retry conditions
Downstream systems should be idempotent.
Security Considerations
- IAM permissions should follow least-privilege principles
- Queue access should be restricted to authorized producers and consumers
- TLS verification should remain enabled
- Visibility timeouts should be carefully tuned to avoid message replay storms
Common Use Cases
- Centralized log ingestion from distributed producers
- Decoupled observability pipelines
- Serverless and event-driven architectures
- OTLP signal ingestion via SQS
- Buffered ingestion for bursty workloads