Aws Kinesis Firehose
AWS Kinesis Firehose Source
Collect logs, metrics, and traces delivered by AWS Kinesis Firehose using its HTTP destination integration.
This source allows worekr to act as an HTTP endpoint for Firehose delivery streams, enabling direct ingestion of AWS-managed streaming data into observability pipelines.
Collection Model
- AWS Kinesis Firehose delivers records via HTTP POST requests
- worker exposes an HTTP endpoint to receive these requests
- Incoming payloads are decoded into events
- Events are processed in batches and forwarded downstream
- Acknowledgements ensure delivery guarantees back to Firehose
This model supports high-throughput managed ingestion directly from AWS services.
Requirements and Network Considerations
HTTP-Only Delivery
AWS Kinesis Firehose only supports HTTP delivery.
As a result:
- TLS termination must be handled explicitly
- worker must either:
- Be fronted by a load balancer (ALB / NLB)
- Or be configured with native TLS support
Without TLS, Firehose delivery will fail.
Typical Architecture Patterns
- AWS-native log delivery pipelines
- CloudWatch Logs → Firehose → Worker
- Managed streaming ingestion without agents
- Centralized observability aggregation
- Cross-account AWS telemetry ingestion
Authentication and Access Control
Access Key Validation
Firehose can attach a user-defined access key to each HTTP request.
Worker supports:
- A single access key (deprecated)
- Multiple access keys for rotation and multi-stream support
If access keys are configured:
- Only requests with matching keys are accepted
- Requests without valid keys are rejected
If no access keys are configured:
- All incoming requests are accepted
Access Key Storage
When enabled:
- Incoming access keys are stored securely in event secrets
- The key is exposed under a reserved secret name
- Enables downstream enrichment, auditing, or routing logic
Acknowledgement Handling
acknowledgements (deprecated)
Source-level acknowledgement configuration is deprecated.
Important notes:
- Acknowledgements are controlled globally or at the sink level
- Source-level configuration has no effect
- Ensures consistent end-to-end delivery semantics
Event Decoding
Firehose delivers raw byte records that must be decoded before processing.
Supported Codecs
- JSON
- Syslog (RFC 3164 / RFC 5424)
- GELF
- Avro
- Protobuf
- OTLP (logs, metrics, traces)
- InfluxDB Line Protocol
- Worker native formats
- Raw bytes
- VRL-based custom decoding
Some codecs can automatically determine the output signal type.
Signal Type Detection
For multi-signal formats:
- Logs, metrics, and traces can be auto-detected
- Parsing order can be restricted to optimize performance
- Duplicate signal attempts are automatically removed
Record-Level Compression
record_compression
Controls decompression of individual Firehose records.
This is especially important when ingesting:
- CloudWatch Logs
- AWS services that gzip records before delivery
Supported modes:
- Automatic detection via magic bytes
- Explicit gzip
- No decompression
Incorrect compression handling may result in corrupted events.
Framing Model
Framing determines how individual events are extracted from raw byte streams.
Supported framing strategies include:
- Byte-based framing
- Newline-delimited
- Character-delimited
- Length-delimited
- Octet counting
- Varint length-delimited
- Chunked GELF
Frame size limits can be enforced to prevent unbounded memory usage.
Multiline and Structured Event Handling
Framing and decoding can be combined to:
- Reassemble multiline logs
- Decode structured payloads
- Safely handle malformed or partial records
Timeouts and buffer limits ensure memory safety under load.
Keepalive and Connection Management
HTTP keepalive parameters allow fine-grained control over:
- Maximum connection lifetime
- Connection reuse
- Jittered connection termination to avoid thundering herds
This is critical for high-volume Firehose streams.
TLS Support
The source supports full TLS configuration, including:
- Server certificates and private keys
- Custom CA bundles
- ALPN protocol negotiation
- Certificate and hostname verification
TLS is mandatory when exposing the endpoint directly to AWS Firehose.
Reliability Characteristics
- At-least-once delivery semantics
- Batch-based ingestion
- Backpressure-aware acknowledgement handling
- Stateless processing model
These properties make the Firehose source suitable for large-scale, production-grade AWS ingestion pipelines.
Common Use Cases
- CloudWatch Logs ingestion
- AWS service telemetry pipelines
- Centralized observability aggregation
- Managed streaming ingestion without agents
- Secure, TLS-enforced AWS log delivery