---
title: Aws Kinesis Firehose
slug: telemetry-pipeline/aws-kinesis-firehose
docTags: 
createdAt: 2025-04-21T00:46:46.618Z
---

## AWS Kinesis Firehose Source

Collect logs, metrics, and traces delivered by **AWS Kinesis Firehose** using its HTTP destination integration.

This source allows worekr to act as an **HTTP endpoint** for Firehose delivery streams, enabling direct ingestion of AWS-managed streaming data into observability pipelines.

## Collection Model

- AWS Kinesis Firehose delivers records via **HTTP POST** requests 
- worker exposes an HTTP endpoint to receive these requests 
- Incoming payloads are decoded into events 
- Events are processed in batches and forwarded downstream 
- Acknowledgements ensure delivery guarantees back to Firehose 

This model supports **high-throughput managed ingestion** directly from AWS services.

## Requirements and Network Considerations

### HTTP-Only Delivery

AWS Kinesis Firehose **only supports HTTP delivery**.

As a result:

- TLS termination must be handled explicitly 
- worker must either: 
  - Be fronted by a **load balancer** (ALB / NLB) 
  - Or be configured with **native TLS support** 

Without TLS, Firehose delivery will fail.

## Typical Architecture Patterns

- AWS-native log delivery pipelines 
- CloudWatch Logs → Firehose → Worker 
- Managed streaming ingestion without agents 
- Centralized observability aggregation 
- Cross-account AWS telemetry ingestion 

## Authentication and Access Control

### Access Key Validation

Firehose can attach a **user-defined access key** to each HTTP request.

Worker supports:

- A single access key (deprecated) 
- Multiple access keys for rotation and multi-stream support 

If access keys are configured:

- Only requests with matching keys are accepted 
- Requests without valid keys are rejected 

If no access keys are configured:

- All incoming requests are accepted 

### Access Key Storage

When enabled:

- Incoming access keys are stored securely in **event secrets** 
- The key is exposed under a reserved secret name 
- Enables downstream enrichment, auditing, or routing logic 

## Acknowledgement Handling

### acknowledgements (deprecated)

Source-level acknowledgement configuration is deprecated.

Important notes:

- Acknowledgements are controlled globally or at the sink level 
- Source-level configuration has no effect 
- Ensures consistent end-to-end delivery semantics 

## Event Decoding

Firehose delivers raw byte records that must be decoded before processing.

### Supported Codecs

- JSON 
- Syslog (RFC 3164 / RFC 5424) 
- GELF 
- Avro 
- Protobuf 
- OTLP (logs, metrics, traces) 
- InfluxDB Line Protocol 
- Worker native formats 
- Raw bytes 
- VRL-based custom decoding 

Some codecs can automatically determine the output signal type.

### Signal Type Detection

For multi-signal formats:

- Logs, metrics, and traces can be auto-detected 
- Parsing order can be restricted to optimize performance 
- Duplicate signal attempts are automatically removed 

## Record-Level Compression

### record\_compression

Controls decompression of individual Firehose records.

This is especially important when ingesting:

- CloudWatch Logs 
- AWS services that gzip records before delivery 

Supported modes:

- Automatic detection via magic bytes 
- Explicit gzip 
- No decompression 

Incorrect compression handling may result in corrupted events.

## Framing Model

Framing determines how individual events are extracted from raw byte streams.

Supported framing strategies include:

- Byte-based framing 
- Newline-delimited 
- Character-delimited 
- Length-delimited 
- Octet counting 
- Varint length-delimited 
- Chunked GELF 

Frame size limits can be enforced to prevent unbounded memory usage.

## Multiline and Structured Event Handling

Framing and decoding can be combined to:

- Reassemble multiline logs 
- Decode structured payloads 
- Safely handle malformed or partial records 

Timeouts and buffer limits ensure memory safety under load.

## Keepalive and Connection Management

HTTP keepalive parameters allow fine-grained control over:

- Maximum connection lifetime 
- Connection reuse 
- Jittered connection termination to avoid thundering herds 

This is critical for **high-volume Firehose streams**.

## TLS Support

The source supports full TLS configuration, including:

- Server certificates and private keys 
- Custom CA bundles 
- ALPN protocol negotiation 
- Certificate and hostname verification 

TLS is mandatory when exposing the endpoint directly to AWS Firehose.

## Reliability Characteristics

- At-least-once delivery semantics 
- Batch-based ingestion 
- Backpressure-aware acknowledgement handling 
- Stateless processing model 

These properties make the Firehose source suitable for **large-scale, production-grade AWS ingestion pipelines**.

## Common Use Cases

- CloudWatch Logs ingestion 
- AWS service telemetry pipelines 
- Centralized observability aggregation 
- Managed streaming ingestion without agents 
- Secure, TLS-enforced AWS log delivery