---
title: AWS SQS
slug: telemetry-pipeline/aws-sqs
docTags: 
createdAt: 2025-04-21T00:48:43.940Z
---

## AWS SQS Source

The AWS SQS source collects events from **Amazon Simple Queue Service (SQS)** and converts queued messages into observability signals. Depending on the payload and decoding configuration, this source can emit **logs, metrics, or traces**.

It is designed to run in an **aggregator** role and is commonly used to centralize data produced by distributed systems, serverless workloads, and decoupled application components.

## Collection Model

- Pull-based polling of SQS queues 
- Concurrent consumers fetch messages from the queue 
- Messages are decoded into structured events 
- Successfully processed messages are deleted from the queue 

The source operates statelessly and relies on SQS visibility timeouts to guarantee reliable delivery.

## Queue Configuration

### queue\_url (required, string)

Specifies the URL of the SQS queue to poll for messages.

Operational considerations:

- The queue must exist and be accessible 
- Permissions must allow ReceiveMessage, DeleteMessage, and ChangeMessageVisibility 
- FIFO and standard queues are both supported 

## Authentication and Authorization

### auth (optional, object)

Defines how the source authenticates with AWS services.

Supported authentication strategies include:

- Static credentials (access key and secret key) 
- IAM roles with assume\_role 
- AWS credentials files and named profiles 
- Instance Metadata Service (IMDS) 

The source follows AWS best practices and supports the standard AWS credential resolution chain.

### IMDS Configuration

When running on EC2, ECS, or EKS, credentials can be automatically retrieved via **IMDS**.

IMDS-related settings control:

- Connection timeout 
- Read timeout 
- Retry attempts when fetching credentials 

## Region and Endpoint Resolution

### region (optional, string)

Defines the AWS region for the target SQS service.

If not set, the region is inferred from:

- The queue URL 
- Environment configuration 
- AWS SDK defaults 

### endpoint (optional, string)

Allows overriding the AWS endpoint.

This is typically used for:

- AWS-compatible services 
- Local testing environments 
- Private cloud or emulated SQS endpoints 

## Message Polling Behavior

### poll\_secs (optional, uint)

Controls how long the source waits while polling the queue.

Notes:

- Messages are consumed immediately when available 
- Increasing this value does not slow down consumption 
- Generally should not be modified unless explicitly required 

### client\_concurrency (optional, uint)

Controls the number of concurrent polling tasks.

Higher values may be beneficial when:

- Message throughput is high 
- Messages are small 
- CPU resources are underutilized 

### visibility\_timeout\_secs (optional, uint)

Defines how long a message remains invisible after being received.

If processing exceeds this duration:

- The message becomes visible again 
- Another consumer may reprocess it 
- This can lead to duplicate events 

The timeout should be aligned with worst-case processing time.

## Message Lifecycle Control

### delete\_message (optional, bool)

Controls whether messages are deleted after successful processing.

Disabling deletion can be useful for:

- Debugging 
- Validation during initial setup 
- Replay scenarios 

In production environments, this should remain enabled.

## Decoding Configuration

### decoding (optional, object)

Defines how raw message payloads are decoded into events.

The decoding stage determines:

- Event type (log, metric, trace) 
- Field structure 
- Timestamp parsing 
- Schema validation behavior 

### Supported Codecs

The source supports multiple decoding codecs, including:

- Raw bytes 
- JSON 
- Syslog 
- GELF 
- InfluxDB line protocol 
- Protobuf 
- Avro 
- OTLP (logs, metrics, traces) 
- Native Vector formats 
- VRL-based custom decoding 

Some codecs are experimental and may evolve over time.

### VRL Decoding

When using VRL decoding:

- Each message is passed through a VRL program 
- The final .target defines the decoded event 
- Compilation or runtime errors cause decoding failures 

This approach enables advanced transformation and enrichment during ingestion.

## Framing Configuration

### framing (optional, object)

Controls how raw byte streams are split into individual messages.

Supported framing methods include:

- Byte-based framing 
- Character-delimited 
- Newline-delimited 
- Length-delimited 
- Octet-counting 
- Chunked GELF 
- Varint length-delimited 

Framing configuration is critical when messages contain multiple events or non-trivial binary formats.

## Proxy Support

### proxy (optional, object)

Configures HTTP(S) proxy usage for AWS API requests.

This is useful in:

- Restricted network environments 
- Controlled egress architectures 
- Enterprise proxy setups 

Proxy bypass rules can be defined per host or network.

## TLS Configuration

### tls (optional, object)

Controls TLS behavior for outgoing connections.

Capabilities include:

- Custom CA certificates 
- Client certificate authentication 
- ALPN protocol configuration 
- Certificate and hostname verification 

Disabling certificate or hostname verification is strongly discouraged in production.

## Acknowledgements and Delivery Semantics

- End-to-end acknowledgements are supported 
- Source-level acknowledgement settings are deprecated 
- Reliability is managed at the global or sink level 
- Duplicate delivery is possible under retry conditions 

Downstream systems should be idempotent.

## Security Considerations

- IAM permissions should follow least-privilege principles 
- Queue access should be restricted to authorized producers and consumers 
- TLS verification should remain enabled 
- Visibility timeouts should be carefully tuned to avoid message replay storms 

## Common Use Cases

- Centralized log ingestion from distributed producers 
- Decoupled observability pipelines 
- Serverless and event-driven architectures 
- OTLP signal ingestion via SQS 
- Buffered ingestion for bursty workloads