--- title: Kafka slug: telemetry-pipeline/6_Wn-kafka docTags: createdAt: 2025-04-21T01:01:14.141Z --- # Kafka Sink ## Overview The **Kafka sink** enables worker to publish observability data to **Apache Kafka topics**. It supports both **log** and **metric** data and is designed for scalable, distributed data pipelines where Kafka acts as a central event backbone. This sink integrates seamlessly with existing Kafka ecosystems, enabling downstream processing by stream processors, analytics platforms, and long-term storage systems. ## Supported Input Types - Logs - Metrics ## Prerequisites Before configuring the Kafka sink, ensure the following requirements are met: - A reachable Apache Kafka cluster - At least one existing Kafka topic - Network connectivity from worker to all Kafka bootstrap servers - Appropriate Kafka permissions for producing messages to the target topic ## Core Configuration Parameters ### Bootstrap Servers Bootstrap servers define the initial Kafka brokers worker uses to connect to the cluster. These servers allow properties.content\_encoding Sets the content encoding metadata. Use it when your consumers expect a specific encoding marker. This is especially relevant if you apply compression or special serialization conventions at the application level. properties.expiration\_ms Sets a per-message TTL in milliseconds. When used, messages may be dropped by the broker after the expiration window. This is useful for volatile telemetry where late data is worthless, but dangerous for audit/compliance logs. properties.priority Sets message priority. It can be fixed or templated per event, and must fit within broker constraints. Message priority only has effect if the target queues are configured to support priorities, and it may increase broker overhead. Buffering and Backpressure The sink supports memory or disk buffering. Memory buffering provides higher performance but loses buffered data on crash or restart. Disk buffering persists buffered data and survives restarts but requires sufficient disk capacity and meets minimum size requirements. Disk buffering is recommended when you cannot tolerate losing in-flight logs during agent restarts or node failures. when\_full behavior controls whether worker blocks upstream components (preserving data but applying backpressure) or drops new events (preserving system responsiveness but losing data under pressure). For security or compliance logs, blocking is usually the safer default. Acknowledgements Although the sink includes an acknowledgements configuration section, this sink is documented as not supporting acknowledgements. In practice, do not rely on end-to-end acknowledgements behavior for delivery correctness with this sink. If you need strong end-to-end acknowledgement semantics, you typically need a sink that explicitly supports them, or you need an architecture where the broker publish is confirmed and integrated into your ingestion guarantees. TLS and Secure Transport TLS configuration is used when connecting over amqps or when you need to customize certificate validation behavior. ca\_file is used when the broker certificate is signed by a private CA or when your environment injects TLS interception certificates. crt\_file and key\_file are used for client certificate authentication when the broker requires mutual TLS. verify\_certificate and verify\_hostname should remain enabled for production. Disabling them removes key protections against impersonation and man-in-the-middle attacks. server\_name can be set when the broker is behind a TLS endpoint that requires SNI to route correctly. Operational Notes and Common Pitfalls Single point of failure versus broker durability AMQP is often used as a durability layer, but delivery guarantees depend on the broker configuration. If your consumers require persistent messages, you must ensure the broker exchange/queues and consumers are configured accordingly; otherwise, data may be lost despite worker retrying. Dynamic exchange/routing\_key risks Templated exchange or routing key values make routing flexible, but increase the chance of silent drops if bindings don’t exist. Treat this like a contract between producers and broker topology. At-least-once duplicates If the broker or network is unstable, retries can result in duplicates. Downstream consumers should be idempotent where possible, or you should include stable identifiers in events to enable deduplication. Credential handling Because the connection string embeds user/password, avoid storing it in plaintext config repos. Use secret injection mechanisms.to discover all brokers and partitions within the Kafka cluster. Multiple bootstrap servers should be provided to ensure resiliency and high availability. ### Topic The topic defines the Kafka destination where events are published. The topic value can be static or dynamically derived from event fields, enabling: - Multi-tenant routing - Environment-based topic separation - Event-type-based fan-out patterns ## Event Batching The Kafka sink supports batching to improve throughput and reduce network overhead. Batching can be controlled using: - Maximum batch size in bytes - Maximum number of events per batch - Maximum batch lifetime before flushing Batching is strongly recommended for high-volume workloads and metric streams. ## Buffering Behavior The sink supports both **memory-based** and **disk-based** buffering. ### Memory Buffering - Optimized for performance - Buffered events are lost if the process restarts unexpectedly ### Disk Buffering - Provides durability across restarts - Recommended for production environments with strict data-loss requirements Buffer limits and overflow behavior can be tuned to apply backpressure or drop events when capacity is exceeded. ## Message Encoding The Kafka sink supports multiple encoding formats, allowing events to be serialized in a way that aligns with downstream consumers. Common use cases include: - Structured formats for analytics and stream processing - Text-based formats for debugging or compatibility - Schema-based formats for strong typing and validation Encoding configuration determines how logs and metrics are transformed into Kafka messages. ## Compression Message compression can be enabled to reduce network usage and improve throughput. Supported compression options allow you to balance: - CPU utilization - Network efficiency - Consumer compatibility Compression is particularly effective for high-volume log pipelines. ## Message Keys and Headers ### Message Keys Kafka message keys can be derived from event fields. Keys influence partition selection and ordering guarantees within Kafka. Typical use cases include: - Ensuring ordering per host, service, or tenant - Improving consumer locality - Enabling deterministic partitioning ### Headers Event fields can be mapped to Kafka headers. Headers provide lightweight metadata that can be consumed without parsing the message payload. This is useful for: - Routing decisions - Lightweight filtering - Metadata enrichment ## Security and Authentication ### TLS Encryption The Kafka sink supports TLS encryption to secure data in transit between worker and Kafka brokers. TLS configuration enables: - Server certificate verification - Mutual TLS authentication - Protection against man-in-the-middle attacks ### SASL Authentication SASL authentication is supported for secure access to Kafka clusters. Common scenarios include: - Username/password authentication - SCRAM-based mechanisms - Enterprise Kafka deployments with centralized authentication Advanced authentication mechanisms can be configured using native Kafka client options. ## Rate Limiting and Timeouts The Kafka sink allows control over request behavior to maintain stability under load: - Request rate limiting - Network timeout configuration - Message delivery timeouts These controls help protect Kafka clusters from overload and ensure predictable producer behavior. ## Advanced Kafka Client Options For advanced use cases, the sink supports passing low-level Kafka client options directly to the underlying Kafka library. This enables fine-grained tuning for: - Broker compatibility - Performance optimization - Custom security or networking requirements ## Typical Use Cases The Kafka sink is commonly used in the following scenarios: - Centralized log ingestion pipelines - Metric streaming into analytics platforms - Event-driven architectures - Multi-tenant observability platforms - Integration with stream processors such as Flink or Spark ## Summary The Kafka sink provides a robust and scalable mechanism for publishing logs and metrics to Apache Kafka. With support for batching, buffering, compression, flexible encoding, and enterprise-grade security, it serves as a foundational component for modern observability and data streaming architectures.