---
title: AWS S3
slug: telemetry-pipeline/aws-s3
docTags: 
createdAt: 2025-04-21T00:56:41.050Z
---

## AWS S3 sink 

The **AWS S3 sink **writes observability events (logs / metrics / traces) into **Amazon S3 objects**. It’s a **batch** sink, so it accumulates events and periodically uploads them as object files.

### Destination

- **bucket**
   The target S3 bucket name. This is the only “where” that is mandatory. 
- **region**
   Which AWS region S3 API calls go to. If wrong, you’ll see auth/signing errors or it’ll try the wrong regional endpoint. 
- **endpoint**
   Overrides the AWS S3 endpoint. Used for: 
  - S3-compatible storage (MinIO, Ceph RGW, etc.) 
  - special AWS partitions / private endpoints
     If set, worker talks to *that* URL instead of AWS’s default. 
- **force\_path\_style**
   Controls URL style: 
  - **true**: endpoint/bucket/key (path-style) 
  - **false**: bucket.endpoint/key (virtual-hosted style)
     Many S3-compatible systems prefer path-style; AWS generally supports both (but behavior can vary with TLS/DNS). 

### Object naming (this defines your “folder structure”)

- **key\_prefix**
   Prefix prepended to every object key. Think “directory”.
   Best used to partition by date/env/cluster/tenant to keep listing/querying sane. 
- **filename\_time\_format**
   Controls the timestamp portion in the object name (strftime-like).
   If you want predictable rotation and time-based grouping, this is the knob. 
- **filename\_append\_uuid**
   If enabled, appends a UUID to the generated object name.
   Meaning: avoids collisions when multiple writers create objects in the same second/time-bucket.
   Tradeoff: harder to visually scan names, but safer at scale. 
- **filename\_extension**
   Forces a specific file extension (e.g., .json, .log, .txt).
   Useful if downstream tooling relies on extensions. Otherwise, optional. 

### Batching (how often objects are uploaded + how big they are)

- **batch.max\_bytes**
   Upper bound on batch size (pre-compression).
   Larger → fewer, bigger objects (cheaper + more throughput).
   Smaller → more objects (more PUT/list overhead, lower latency). 
- **batch.max\_events**
   Upper bound on number of events per object.
   This protects you from huge objects when events are small. 
- **batch.timeout\_secs**
   Maximum time worker waits before it “flushes” the batch even if size/event limits aren’t hit.
   Smaller → more near-real-time uploads.
   Larger → better efficiency, more latency. 

### Buffering (what happens if S3 is slow/unreachable)

- **buffer.type** (memory or disk) 
  - memory: fastest, but you lose buffered data on crash/restart 
  - disk: slower, but survives restarts 
- **buffer.max\_size**
   Hard cap of buffer storage. When you hit this, you’re “out of cushion”. 
- **buffer.max\_events** *(only relevant for memory buffer)*
   Limit by event count instead of bytes. 
- **buffer.when\_full** (block or drop\_newest) 
  - block: apply backpressure upstream (safer, no intentional loss) 
  - drop\_newest: intentionally drop newest events to keep running (only if you accept loss) 

### Compression + HTTP metadata

- **compression** (gzip, zstd, snappy, zlib, none)
   Controls how payload is compressed before upload. 
  - gzip: common & widely compatible 
  - zstd: often best compression ratio/speed balance, but some tooling expects gzip 
  - none: bigger objects, higher cost 
- **content\_encoding**
   Forces the HTTP Content-Encoding metadata on the object.
   Normally this matches your compression; override only if you have a special consumer that expects a specific value. 
- **content\_type**
   Sets object Content-Type.
   Useful for browsers/tools or certain pipelines; not critical for most log pipelines. 

### Storage class + encryption + access control

- **storage\_class**
   Sets the S3 storage tier of uploaded objects (STANDARD, IA, Glacier variants, etc.).
   This affects cost + retrieval latency. Many teams leave STANDARD and use lifecycle rules later. 
- **server\_side\_encryption** (AES256 or aws\:kms)
   Enables server-side encryption: 
  - AES256 = SSE-S3 (AWS managed keys) 
  - aws\:kms = SSE-KMS (KMS keys/policies/audit) 
- **ssekms\_key\_id**
   Which KMS key to use when server\_side\_encryption = aws\:kms.
   Critical for compliance / cross-account setups. 
- **acl** (canned ACL)
   Applies an S3 canned ACL to created objects/bucket.
   In modern AWS setups, bucket policies + IAM are preferred; ACL is for legacy/cross-account corner cases. 
- **grant\_read / grant\_full\_control / grant\_read\_acp / grant\_write\_acp**
   Adds explicit ACL grants to specific grantees.
   Only use when you *must* share object-level permissions via ACLs (rare today). 
- **tags**
   Adds object tags. Useful for lifecycle policies, cost allocation, and governance. 

### Outbound request behavior (performance + resilience knobs)

- **request.concurrency** (none, fixed number, or adaptive)
   How many uploads/requests worker can have in-flight.
   More concurrency → more throughput, more pressure on S3/network. 
- **request.rate\_limit\_num / request.rate\_limit\_duration\_secs**
   Caps request rate. Used to avoid throttling or to be nice to S3-compatible backends. 
- **request.timeout\_secs**
   Per-request timeout. Too low can cause retries and duplicates; too high can stall recovery. 
- **request.retry\_attempts / retry\_strategy**
   How aggressively to retry failures.
   retry\_strategy.type controls whether to retry everything, nothing, or a custom set (like only certain HTTP codes). 
- **request.retry\_initial\_backoff\_secs / request.retry\_max\_duration\_secs / request.retry\_jitter\_mode**
   Shapes the retry backoff curve (how fast it ramps and whether retries are randomized to avoid retry storms). 
- **request.adaptive\_concurrency.\***
   Controls worker's “auto-tuning” concurrency. Usually leave defaults unless you’re tuning around a flaky endpoint. 

### Proxy + TLS (network path / trust)

- **proxy.enabled / proxy.http / proxy.https / proxy.no\_proxy**
   Routes requests via HTTP(S) proxy; no\_proxy bypass list. 
- **tls.ca\_file / tls.crt\_file / tls.key\_file / tls.key\_pass**
   TLS trust and client cert configuration. 
- **tls.verify\_certificate**
   Whether to verify the certificate chain (should generally be on). 
- **tls.verify\_hostname**
   Whether to verify hostname matches cert (should generally be on). 
- **tls.server\_name**
   Override SNI / hostname used for TLS verification (useful behind proxies or S3-compatible endpoints). 
- **tls.alpn\_protocols**
   Advanced TLS negotiation detail; rarely needed unless a proxy/backend requires specific ALPN. 

### Timezone (only affects templated date fields)

- **timezone**
   Controls what timezone is used when your templated strings include date/time specifiers.
   Important if you partition by day and you want day boundaries in UTC vs local time.