--- title: Pulser slug: telemetry-pipeline/pulser docTags: createdAt: 2025-04-21T01:02:59.508Z --- # Pulsar Sink The **Pulsar sink** publishes observability data (logs and metrics) to topics in **Apache Pulsar**. This section explains each configuration parameter, its role, and its operational impact. ## ## Authentication Defines how the Pulsar client authenticates to the cluster. ### auth Root authentication configuration. auth.name - Used as a username for basic authentication or as a token identifier for JWT authentication. auth.token - Password for basic authentication or the signed JWT token when using token-based auth. ### auth.oauth2 OAuth2-based authentication configuration. auth.oauth2.audience - Identifies the intended recipient of the OAuth2 token. auth.oauth2.credentials\_url - Location of OAuth2 credentials. - Can be a remote URL or an embedded data URL. auth.oauth2.issuer\_url - OAuth2 authorization server that issues tokens. auth.oauth2.scope - Limits the access scope granted to the token. ## Batching Controls how events are grouped before being sent. ### batch Improves throughput by sending events in batches. batch.max\_bytes - Flushes the batch when total size reaches this limit. - Helps control memory usage and network payload size. batch.max\_events - Flushes the batch when the number of events reaches this threshold. - Useful for latency-sensitive pipelines. ## Buffering Controls temporary storage when downstream delivery is delayed. ### buffer Defines buffering strategy and limits. buffer.type - **memory:** Faster, but data is lost on crash. - **disk:** Slower, but durable across restarts. buffer.max\_events - Maximum number of events allowed in memory buffer. - Applies only when buffer type is memory. buffer.max\_size - Maximum memory or disk space used by the buffer. - Disk buffers require a minimum size to operate safely. buffer.when\_full - **block:** Applies backpressure to upstream components. - **drop\_newest:** Drops incoming events when buffer is full. - Choose based on whether data loss or backpressure is acceptable. ## Compression Controls payload compression before sending to Pulsar. ### compression - Reduces network bandwidth usage. - Supported options include LZ4, Snappy, Zlib, Zstandard, or no compression. - Compression trades CPU usage for network efficiency. ## Connection Retry Options Defines how the client behaves when connectivity issues occur. ### connection\_retry\_options connection\_timeout\_secs - Maximum time allowed to establish a connection. keep\_alive\_secs - Interval for sending keep-alive signals to brokers. min\_backoff\_ms - Initial delay before retrying a failed connection. max\_backoff\_secs - Maximum delay between retries. max\_retries - Maximum number of reconnection attempts before giving up. ## Encoding Defines how events are serialized before being sent. ### encoding Mandatory configuration that determines: - Event format - Supported input types - Interoperability with downstream consumers ### encoding.codec Specifies the serialization format, such as: - JSON - Avro - CSV - CEF - Protobuf - OTLP - Text - Raw message Each codec enables a corresponding encoding subsection. ### Encoding: Avro encoding.avro.schema - Defines the Avro schema used to serialize events. - Ensures strong typing and schema validation downstream. ### Encoding: CEF (Common Event Format) Used for security and SIEM integrations. Key parameters define: - Vendor identity - Product name - Version - Event classification - Severity mapping - Extension fields extracted from logs CEF enforces strict field length and formatting rules. ### Encoding: CSV Defines how structured fields are flattened into CSV. Key aspects include: - Field order - Delimiter and quoting rules - Escaping behavior - Handling of missing or unsupported field types ### Encoding: GELF Used for Graylog-compatible ingestion. encoding.gelf.max\_chunk\_size - Controls chunking behavior for large messages. - Must align with receiver capabilities. ### Encoding: JSON encoding.json.pretty - Enables human-readable formatting. - Should be disabled for production due to size overhead. ### Encoding: Metrics Tag Values encoding.metric\_tag\_values - **single:** Only last value is preserved. - **full:** All tag values are preserved as arrays. - Impacts cardinality and payload size. ### Encoding: Protobuf Used for strongly typed binary serialization. Key parameters define: - Descriptor file location - Message type - Field naming conventions (JSON vs Protobuf style) ### Encoding: Timestamp Format Controls how timestamps are represented: - RFC 3339 - Unix (seconds, milliseconds, microseconds, nanoseconds) - Floating-point Unix time ## Endpoint ### endpoint - Defines the Pulsar broker connection address. - Must include protocol and port. - Used by the client to establish the initial connection. ## Healthcheck ### healthcheck.enabled - Enables startup-time validation of sink availability. - Prevents silent failures at boot time. ## Inputs ### inputs - List of upstream sources or transforms feeding this sink. - Supports wildcard matching for flexible topology design. ## Partitioning and Routing ### partition\_key\_field - Determines how events are distributed across topic partitions. - Pulsar hashes this value to select a partition. - Missing fields result in round-robin distribution. ## Producer Identification ### producer\_name - Sets a human-readable producer name in Pulsar. - Useful for debugging and monitoring producer behavior. ## Message Properties ### properties\_key - Specifies which log field is used to populate Pulsar message properties. - Enables metadata-based routing and filtering downstream. ## TLS Configuration ### tls Defines secure transport settings. tls.ca\_file - Trusted Certificate Authority list. tls.verify\_certificate - Enables certificate validation. - Disabling weakens security and is strongly discouraged. tls.verify\_hostname - Verifies broker hostname during TLS handshake. - Should remain enabled for secure deployments. ## Topic ### topic - Target Pulsar topic where events are published. - Supports dynamic templates for per-event routing. - Required for all Pulsar sink configurations. ## Supported Input Types - Logs - Metrics Both are encoded according to the selected codec.