Kafka

Collect logs, metrics, and traces from Apache Kafka topics using a consumer group–based ingestion model.

This source allows worker to act as a Kafka consumer, reading observability data from one or more topics and integrating Kafka-native streaming pipelines into centralized telemetry architectures.

This model aligns with Kafka-native scalability and fault tolerance patterns.

This enables:

On restart or rebalance:

During shutdown or partition revocation:

This is particularly useful in environments with high topic churn.

Kafka messages are delivered as raw bytes and must be decoded.

Some codecs can automatically infer the event signal type.

The Kafka source supports:

For OTLP and native formats:

Framing defines how events are extracted from Kafka message payloads.

Supported strategies include:

Frame size limits prevent malformed messages from exhausting memory.

Kafka-specific metadata can be attached to each event, including:

These fields enable:

Worker uses librdkafka internally, enabling:

Low-level Kafka parameters can be passed directly when needed.

Supported mechanisms:

For advanced mechanisms (e.g. Kerberos):

This allows compatibility with enterprise Kafka deployments.

Optional metrics expose:

These metrics are critical for:

These properties make the Kafka source suitable for high-volume, continuous telemetry ingestion.