--- title: Aws S3 slug: telemetry-pipeline/i2Ks-aws-s3 docTags: createdAt: 2025-04-21T00:47:07.681Z --- ## AWS S3 Source Collect logs, metrics, and traces from objects stored in **Amazon S3**. This source ingests data from S3 buckets by consuming **S3 event notifications delivered through Amazon SQS**, enabling reliable and scalable batch-based ingestion. It is commonly used for ingesting **archived logs**, **cloud service outputs**, and **object-based telemetry pipelines**. ## Collection Model - S3 bucket events are delivered to an **SQS queue** - Worker polls the queue for object creation notifications - Referenced objects are fetched from S3 - Object contents are decoded into events - Events are forwarded downstream with acknowledgement support This model decouples object creation from ingestion, enabling **high durability and retry safety**. ## Requirements - An **S3 bucket** configured to emit event notifications - An **SQS queue** subscribed to those notifications - Appropriate **IAM permissions** to: - Read from S3 - Poll and manage messages in SQS - Assume roles or retrieve credentials as configured ## Typical Architecture Patterns - **S3-based log archival ingestion** - **Cloud service log pipelines** (CloudTrail, ELB, VPC Flow Logs) - **Batch-to-stream telemetry conversion** - **Disaster recovery or delayed ingestion** - **Multi-account AWS ingestion pipelines** ## Acknowledgement Handling ### acknowledgements (deprecated) Source-level acknowledgement configuration is deprecated. Important notes: - Acknowledgement behavior is controlled globally or at the sink level - Enabling or disabling acknowledgements at the source has no effect - Ensures consistent end-to-end delivery semantics across pipelines ## Authentication Model The AWS S3 source supports multiple AWS authentication strategies, aligned with standard AWS SDK behavior. ### Supported Authentication Methods - Static credentials (access key / secret key) - Temporary credentials with session tokens - IAM role assumption (STS) - Instance Metadata Service (IMDS) - Shared credentials files and profiles Credential loading follows the configured strategy and supports timeout and retry controls. ### IAM Role Assumption When assuming a role: - A role ARN must be provided - Optional external IDs can be used for cross-account access - Session names are auto-generated if not specified This is the recommended approach for **production-grade AWS deployments**. ## AWS Region and Endpoints ### region Specifies the AWS region used for service interactions. If not set: - Defaults to the region configured for the service - Or is inferred from the execution environment ### endpoint Allows specifying a custom S3-compatible endpoint. Useful for: - AWS-compatible object storage - Private S3 endpoints - Local or on-prem S3-compatible systems ## Object Retrieval and Addressing ### force\_path\_style Controls whether: - Bucket name appears in the hostname (virtual-hosted style) - Or as part of the request path (path-style) Path-style addressing improves compatibility with some S3-compatible services. ## Compression Handling ### compression Controls how objects are decompressed after retrieval. Supported modes: - Automatic detection - Explicit compression formats - No decompression Compression is inferred from metadata and object key suffixes when auto mode is enabled. ## Event Decoding The AWS S3 source supports a wide range of decoders to transform raw object bytes into structured events. ### Supported Decoding Types - JSON - Syslog - GELF - Avro - Protobuf - OTLP - InfluxDB Line Protocol - Native Vector formats - Raw bytes - VRL-based custom decoding Decoding may emit: - Logs - Metrics - Traces depending on the selected codec. ### Signal Type Detection For multi-signal formats: - The decoder attempts parsing in priority order - Signal detection can be restricted for performance optimization ## Framing Model Framing defines how individual events are separated within object content. Supported framing methods include: - Newline-delimited - Character-delimited - Length-delimited - Octet-counting - Varint length-delimited - Chunked GELF - Raw byte frames Frame length limits can be enforced to protect against malformed or unbounded inputs. ## Multiline Aggregation Multiline processing allows multiple lines to be combined into a single logical event. Supported aggregation modes: - Continue through - Continue past - Halt before - Halt with This is commonly used for: - Stack traces - Multi-line application logs - Structured log blocks Timeouts ensure buffered messages are eventually flushed. ## SQS Integration ### Queue Consumption Model - Messages are polled in batches - Each message references one or more S3 objects - Visibility timeouts protect against duplicate processing - Messages are deleted after successful ingestion ### Concurrency and Throughput - Polling concurrency scales with CPU by default - Batch size and poll intervals are tunable - Designed to balance ingestion throughput and SQS visibility constraints ### Deferred Processing The source can defer processing of older events: - Events exceeding a defined age threshold - Forwarded to an alternate SQS queue - Enables delayed or backfill pipelines ## Reliability Characteristics - At-least-once delivery semantics - Integrated retry handling via SQS visibility timeouts - End-to-end acknowledgements supported - Stateless ingestion model This makes the AWS S3 source suitable for **durable, large-scale ingestion pipelines**. ## TLS and Proxy Support - TLS can be configured for AWS API communication - Custom CA bundles and certificates are supported - HTTP and HTTPS proxying is supported with fine-grained bypass rules ## Common Use Cases - Ingesting CloudTrail, ELB, ALB, and VPC Flow Logs - Processing archived logs stored in S3 - Batch-to-stream observability pipelines - Cross-account or cross-region ingestion - Hybrid cloud and on-prem object storage ingestion