Aws S3
AWS S3 Source
Collect logs, metrics, and traces from objects stored in Amazon S3. This source ingests data from S3 buckets by consuming S3 event notifications delivered through Amazon SQS, enabling reliable and scalable batch-based ingestion.
It is commonly used for ingesting archived logs, cloud service outputs, and object-based telemetry pipelines.
Collection Model
- S3 bucket events are delivered to an SQS queue
- Worker polls the queue for object creation notifications
- Referenced objects are fetched from S3
- Object contents are decoded into events
- Events are forwarded downstream with acknowledgement support
This model decouples object creation from ingestion, enabling high durability and retry safety.
Requirements
- An S3 bucket configured to emit event notifications
- An SQS queue subscribed to those notifications
- Appropriate IAM permissions to:
- Read from S3
- Poll and manage messages in SQS
- Assume roles or retrieve credentials as configured
Typical Architecture Patterns
- S3-based log archival ingestion
- Cloud service log pipelines (CloudTrail, ELB, VPC Flow Logs)
- Batch-to-stream telemetry conversion
- Disaster recovery or delayed ingestion
- Multi-account AWS ingestion pipelines
Acknowledgement Handling
acknowledgements (deprecated)
Source-level acknowledgement configuration is deprecated.
Important notes:
- Acknowledgement behavior is controlled globally or at the sink level
- Enabling or disabling acknowledgements at the source has no effect
- Ensures consistent end-to-end delivery semantics across pipelines
Authentication Model
The AWS S3 source supports multiple AWS authentication strategies, aligned with standard AWS SDK behavior.
Supported Authentication Methods
- Static credentials (access key / secret key)
- Temporary credentials with session tokens
- IAM role assumption (STS)
- Instance Metadata Service (IMDS)
- Shared credentials files and profiles
Credential loading follows the configured strategy and supports timeout and retry controls.
IAM Role Assumption
When assuming a role:
- A role ARN must be provided
- Optional external IDs can be used for cross-account access
- Session names are auto-generated if not specified
This is the recommended approach for production-grade AWS deployments.
AWS Region and Endpoints
region
Specifies the AWS region used for service interactions.
If not set:
- Defaults to the region configured for the service
- Or is inferred from the execution environment
endpoint
Allows specifying a custom S3-compatible endpoint.
Useful for:
- AWS-compatible object storage
- Private S3 endpoints
- Local or on-prem S3-compatible systems
Object Retrieval and Addressing
force_path_style
Controls whether:
- Bucket name appears in the hostname (virtual-hosted style)
- Or as part of the request path (path-style)
Path-style addressing improves compatibility with some S3-compatible services.
Compression Handling
compression
Controls how objects are decompressed after retrieval.
Supported modes:
- Automatic detection
- Explicit compression formats
- No decompression
Compression is inferred from metadata and object key suffixes when auto mode is enabled.
Event Decoding
The AWS S3 source supports a wide range of decoders to transform raw object bytes into structured events.
Supported Decoding Types
- JSON
- Syslog
- GELF
- Avro
- Protobuf
- OTLP
- InfluxDB Line Protocol
- Native Vector formats
- Raw bytes
- VRL-based custom decoding
Decoding may emit:
- Logs
- Metrics
- Traces depending on the selected codec.
Signal Type Detection
For multi-signal formats:
- The decoder attempts parsing in priority order
- Signal detection can be restricted for performance optimization
Framing Model
Framing defines how individual events are separated within object content.
Supported framing methods include:
- Newline-delimited
- Character-delimited
- Length-delimited
- Octet-counting
- Varint length-delimited
- Chunked GELF
- Raw byte frames
Frame length limits can be enforced to protect against malformed or unbounded inputs.
Multiline Aggregation
Multiline processing allows multiple lines to be combined into a single logical event.
Supported aggregation modes:
- Continue through
- Continue past
- Halt before
- Halt with
This is commonly used for:
- Stack traces
- Multi-line application logs
- Structured log blocks
Timeouts ensure buffered messages are eventually flushed.
SQS Integration
Queue Consumption Model
- Messages are polled in batches
- Each message references one or more S3 objects
- Visibility timeouts protect against duplicate processing
- Messages are deleted after successful ingestion
Concurrency and Throughput
- Polling concurrency scales with CPU by default
- Batch size and poll intervals are tunable
- Designed to balance ingestion throughput and SQS visibility constraints
Deferred Processing
The source can defer processing of older events:
- Events exceeding a defined age threshold
- Forwarded to an alternate SQS queue
- Enables delayed or backfill pipelines
Reliability Characteristics
- At-least-once delivery semantics
- Integrated retry handling via SQS visibility timeouts
- End-to-end acknowledgements supported
- Stateless ingestion model
This makes the AWS S3 source suitable for durable, large-scale ingestion pipelines.
TLS and Proxy Support
- TLS can be configured for AWS API communication
- Custom CA bundles and certificates are supported
- HTTP and HTTPS proxying is supported with fine-grained bypass rules
Common Use Cases
- Ingesting CloudTrail, ELB, ALB, and VPC Flow Logs
- Processing archived logs stored in S3
- Batch-to-stream observability pipelines
- Cross-account or cross-region ingestion
- Hybrid cloud and on-prem object storage ingestion