HTTP
HTTP Sink
Overview
The HTTP sink sends observability events to an arbitrary HTTP endpoint. It’s the “generic egress” option when you want to push logs, metrics, and traces into:
- your own ingestion API (custom collector / gateway),
- an internal event intake service,
- a vendor endpoint that accepts HTTP payloads,
- or a lightweight bridge that forwards to Kafka/Loki/ClickHouse/etc.
It supports batching, buffering, configurable encoding + framing, and multiple auth methods (basic, bearer, custom header, AWS signing).
Supported Input Types
- Logs
- Metrics
- Traces
Core Configuration Parameters
URI (required)
The full destination URI to send requests to.
Because it’s templateable, you can route per-event to different paths/hosts (careful: this can explode the number of distinct upstream targets, which affects connection reuse and performance).
Authentication
Auth (optional)
Defines how worker authenticates requests.
Supported strategies:
- basic: username + password (base64 in Authorization)
- bearer: token passed as-is in Authorization: Bearer <token>
- custom: arbitrary authorization header value inserted as Authorization: <value>
- aws: AWS signing (SigV4-style) using access key/secret, optional assume role, etc.
Practical notes:
- Use HTTPS with any header-based auth (basic/bearer/custom), otherwise credentials travel in cleartext.
- AWS auth is for cases where the destination is an AWS service that expects signed requests.
Payload Encoding
Encoding (required)
Controls how events are serialized into bytes and also determines which input types are supported for a chosen codec.
Common choices in practice:
- json: easiest for custom collectors and modern ingest APIs
- text / raw_message: when your endpoint expects plain text lines
- otlp: when the receiver expects OpenTelemetry Protocol (protobuf)
- protobuf / avro: when you have a strict schema and want compact payloads
You can also constrain output fields:
- encoding.only_fields: whitelist
- encoding.except_fields: blacklist
- encoding.timestamp_format: how timestamps are represented
Framing
Framing (optional)
Defines how multiple encoded events are delimited inside the request body.
Key methods:
- newline_delimited: typical for “one JSON per line” ingestion endpoints
- character_delimited: delimiter is configurable (e.g., ,), used when upstream expects a specific separator
- length_delimited / varint_length_delimited: useful for binary/protobuf-style transports
- bytes: no delimiter (you usually pair this with a batch that is already a single blob)
If your receiver expects NDJSON, the classic pairing is:
- encoding.codec = "json"
- framing.method = "newline_delimited"
Delivery Shaping
Batch (optional)
Controls when worker flushes a set of events into one HTTP request:
- batch.max_bytes: cap by size (uncompressed size before serialization/compression)
- batch.max_events: cap by event count
- batch.timeout_secs: flush by time
Batch tuning is the main lever for:
- throughput (bigger batches → fewer requests),
- latency (shorter timeout → quicker delivery),
- and upstream load (request rate).
Buffer (optional)
Controls local buffering for backpressure / durability:
- buffer.type: memory or disk
- buffer.max_size: total buffer memory/disk budget
- buffer.max_events: only relevant for memory buffers
- buffer.when_full: block vs drop_newest
This is the main control for “what happens when the endpoint slows down”.
Compression
Compression (optional)
Compresses the HTTP request body:
- gzip, snappy, zlib, zstd, or none
Use compression when bandwidth is expensive or payload volume is high. It trades CPU for network savings and can help if the receiver supports it efficiently.
HTTP Request Customization
Method (optional)
HTTP method used for sending requests. Default is POST.
Request Headers (optional)
request.headers lets you attach arbitrary headers to every request.
Both header names and values can be templateable, so you can do patterns like:
- per-tenant API keys,
- dynamic routing hints,
- correlation identifiers.
Note: the older headers option is deprecated in favor of request.headers.
Proxy and TLS
Proxy (optional)
Standard proxy config with:
- proxy.http, proxy.https
- proxy.no_proxy list/patterns
TLS (optional)
TLS settings for secure delivery:
- custom CA (tls.ca_file)
- client identity (mTLS: tls.crt_file + tls.key_file)
- hostname verification, cert verification controls
- SNI server name override (tls.server_name)
Advanced Shaping: JSON Envelope
Payload Prefix / Suffix (optional)
payload_prefix and payload_suffix allow wrapping character-delimited JSON payloads into a larger JSON object.
This is used when the receiver expects a JSON envelope like:
worker can emit the delimiters + wrapper as long as the prefix/suffix form valid JSON together.