The Kafka sink enables worker to publish observability data to Apache Kafka topics. It supports both log and metric data and is designed for scalable, distributed data pipelines where Kafka acts as a central event backbone.

This sink integrates seamlessly with existing Kafka ecosystems, enabling downstream processing by stream processors, analytics platforms, and long-term storage systems.

Before configuring the Kafka sink, ensure the following requirements are met:

Bootstrap servers define the initial Kafka brokers worker uses to connect to the cluster. These servers allow properties.content_encoding Sets the content encoding metadata. Use it when your consumers expect a specific encoding marker. This is especially relevant if you apply compression or special serialization conventions at the application level. properties.expiration_ms Sets a per-message TTL in milliseconds. When used, messages may be dropped by the broker after the expiration window. This is useful for volatile telemetry where late data is worthless, but dangerous for audit/compliance logs. properties.priority Sets message priority. It can be fixed or templated per event, and must fit within broker constraints. Message priority only has effect if the target queues are configured to support priorities, and it may increase broker overhead. Buffering and Backpressure The sink supports memory or disk buffering. Memory buffering provides higher performance but loses buffered data on crash or restart. Disk buffering persists buffered data and survives restarts but requires sufficient disk capacity and meets minimum size requirements. Disk buffering is recommended when you cannot tolerate losing in-flight logs during agent restarts or node failures. when_full behavior controls whether worker blocks upstream components (preserving data but applying backpressure) or drops new events (preserving system responsiveness but losing data under pressure). For security or compliance logs, blocking is usually the safer default. Acknowledgements Although the sink includes an acknowledgements configuration section, this sink is documented as not supporting acknowledgements. In practice, do not rely on end-to-end acknowledgements behavior for delivery correctness with this sink. If you need strong end-to-end acknowledgement semantics, you typically need a sink that explicitly supports them, or you need an architecture where the broker publish is confirmed and integrated into your ingestion guarantees. TLS and Secure Transport TLS configuration is used when connecting over amqps or when you need to customize certificate validation behavior. ca_file is used when the broker certificate is signed by a private CA or when your environment injects TLS interception certificates. crt_file and key_file are used for client certificate authentication when the broker requires mutual TLS. verify_certificate and verify_hostname should remain enabled for production. Disabling them removes key protections against impersonation and man-in-the-middle attacks. server_name can be set when the broker is behind a TLS endpoint that requires SNI to route correctly. Operational Notes and Common Pitfalls Single point of failure versus broker durability AMQP is often used as a durability layer, but delivery guarantees depend on the broker configuration. If your consumers require persistent messages, you must ensure the broker exchange/queues and consumers are configured accordingly; otherwise, data may be lost despite worker retrying. Dynamic exchange/routing_key risks Templated exchange or routing key values make routing flexible, but increase the chance of silent drops if bindings don’t exist. Treat this like a contract between producers and broker topology. At-least-once duplicates If the broker or network is unstable, retries can result in duplicates. Downstream consumers should be idempotent where possible, or you should include stable identifiers in events to enable deduplication. Credential handling Because the connection string embeds user/password, avoid storing it in plaintext config repos. Use secret injection mechanisms.to discover all brokers and partitions within the Kafka cluster.

Multiple bootstrap servers should be provided to ensure resiliency and high availability.

The topic defines the Kafka destination where events are published.

The topic value can be static or dynamically derived from event fields, enabling:

The Kafka sink supports batching to improve throughput and reduce network overhead.

Batching can be controlled using:

Batching is strongly recommended for high-volume workloads and metric streams.

The sink supports both memory-based and disk-based buffering.

Buffer limits and overflow behavior can be tuned to apply backpressure or drop events when capacity is exceeded.

The Kafka sink supports multiple encoding formats, allowing events to be serialized in a way that aligns with downstream consumers.

Common use cases include:

Encoding configuration determines how logs and metrics are transformed into Kafka messages.

Message compression can be enabled to reduce network usage and improve throughput.

Supported compression options allow you to balance:

Compression is particularly effective for high-volume log pipelines.

Kafka message keys can be derived from event fields. Keys influence partition selection and ordering guarantees within Kafka.

Typical use cases include:

Event fields can be mapped to Kafka headers. Headers provide lightweight metadata that can be consumed without parsing the message payload.

This is useful for:

The Kafka sink supports TLS encryption to secure data in transit between worker and Kafka brokers.

TLS configuration enables:

SASL authentication is supported for secure access to Kafka clusters.

Common scenarios include:

Advanced authentication mechanisms can be configured using native Kafka client options.

The Kafka sink allows control over request behavior to maintain stability under load:

These controls help protect Kafka clusters from overload and ensure predictable producer behavior.

For advanced use cases, the sink supports passing low-level Kafka client options directly to the underlying Kafka library.

This enables fine-grained tuning for:

The Kafka sink is commonly used in the following scenarios:

The Kafka sink provides a robust and scalable mechanism for publishing logs and metrics to Apache Kafka. With support for batching, buffering, compression, flexible encoding, and enterprise-grade security, it serves as a foundational component for modern observability and data streaming architectures.