Pulser
Pulsar Sink
The Pulsar sink publishes observability data (logs and metrics) to topics in Apache Pulsar. This section explains each configuration parameter, its role, and its operational impact.
Authentication
Defines how the Pulsar client authenticates to the cluster.
auth
Root authentication configuration.
auth.name
- Used as a username for basic authentication or as a token identifier for JWT authentication.
auth.token
- Password for basic authentication or the signed JWT token when using token-based auth.
auth.oauth2
OAuth2-based authentication configuration.
auth.oauth2.audience
- Identifies the intended recipient of the OAuth2 token.
auth.oauth2.credentials_url
- Location of OAuth2 credentials.
- Can be a remote URL or an embedded data URL.
auth.oauth2.issuer_url
- OAuth2 authorization server that issues tokens.
auth.oauth2.scope
- Limits the access scope granted to the token.
Batching
Controls how events are grouped before being sent.
batch
Improves throughput by sending events in batches.
batch.max_bytes
- Flushes the batch when total size reaches this limit.
- Helps control memory usage and network payload size.
batch.max_events
- Flushes the batch when the number of events reaches this threshold.
- Useful for latency-sensitive pipelines.
Buffering
Controls temporary storage when downstream delivery is delayed.
buffer
Defines buffering strategy and limits.
buffer.type
- memory: Faster, but data is lost on crash.
- disk: Slower, but durable across restarts.
buffer.max_events
- Maximum number of events allowed in memory buffer.
- Applies only when buffer type is memory.
buffer.max_size
- Maximum memory or disk space used by the buffer.
- Disk buffers require a minimum size to operate safely.
buffer.when_full
- block: Applies backpressure to upstream components.
- drop_newest: Drops incoming events when buffer is full.
- Choose based on whether data loss or backpressure is acceptable.
Compression
Controls payload compression before sending to Pulsar.
compression
- Reduces network bandwidth usage.
- Supported options include LZ4, Snappy, Zlib, Zstandard, or no compression.
- Compression trades CPU usage for network efficiency.
Connection Retry Options
Defines how the client behaves when connectivity issues occur.
connection_retry_options
connection_timeout_secs
- Maximum time allowed to establish a connection.
keep_alive_secs
- Interval for sending keep-alive signals to brokers.
min_backoff_ms
- Initial delay before retrying a failed connection.
max_backoff_secs
- Maximum delay between retries.
max_retries
- Maximum number of reconnection attempts before giving up.
Encoding
Defines how events are serialized before being sent.
encoding
Mandatory configuration that determines:
- Event format
- Supported input types
- Interoperability with downstream consumers
encoding.codec
Specifies the serialization format, such as:
- JSON
- Avro
- CSV
- CEF
- Protobuf
- OTLP
- Text
- Raw message
Each codec enables a corresponding encoding subsection.
Encoding: Avro
encoding.avro.schema
- Defines the Avro schema used to serialize events.
- Ensures strong typing and schema validation downstream.
Encoding: CEF (Common Event Format)
Used for security and SIEM integrations.
Key parameters define:
- Vendor identity
- Product name
- Version
- Event classification
- Severity mapping
- Extension fields extracted from logs
CEF enforces strict field length and formatting rules.
Encoding: CSV
Defines how structured fields are flattened into CSV.
Key aspects include:
- Field order
- Delimiter and quoting rules
- Escaping behavior
- Handling of missing or unsupported field types
Encoding: GELF
Used for Graylog-compatible ingestion.
encoding.gelf.max_chunk_size
- Controls chunking behavior for large messages.
- Must align with receiver capabilities.
Encoding: JSON
encoding.json.pretty
- Enables human-readable formatting.
- Should be disabled for production due to size overhead.
Encoding: Metrics Tag Values
encoding.metric_tag_values
- single: Only last value is preserved.
- full: All tag values are preserved as arrays.
- Impacts cardinality and payload size.
Encoding: Protobuf
Used for strongly typed binary serialization.
Key parameters define:
- Descriptor file location
- Message type
- Field naming conventions (JSON vs Protobuf style)
Encoding: Timestamp Format
Controls how timestamps are represented:
- RFC 3339
- Unix (seconds, milliseconds, microseconds, nanoseconds)
- Floating-point Unix time
Endpoint
endpoint
- Defines the Pulsar broker connection address.
- Must include protocol and port.
- Used by the client to establish the initial connection.
Healthcheck
healthcheck.enabled
- Enables startup-time validation of sink availability.
- Prevents silent failures at boot time.
Inputs
inputs
- List of upstream sources or transforms feeding this sink.
- Supports wildcard matching for flexible topology design.
Partitioning and Routing
partition_key_field
- Determines how events are distributed across topic partitions.
- Pulsar hashes this value to select a partition.
- Missing fields result in round-robin distribution.
Producer Identification
producer_name
- Sets a human-readable producer name in Pulsar.
- Useful for debugging and monitoring producer behavior.
Message Properties
properties_key
- Specifies which log field is used to populate Pulsar message properties.
- Enables metadata-based routing and filtering downstream.
TLS Configuration
tls
Defines secure transport settings.
tls.ca_file
- Trusted Certificate Authority list.
tls.verify_certificate
- Enables certificate validation.
- Disabling weakens security and is strongly discouraged.
tls.verify_hostname
- Verifies broker hostname during TLS handshake.
- Should remain enabled for secure deployments.
Topic
topic
- Target Pulsar topic where events are published.
- Supports dynamic templates for per-event routing.
- Required for all Pulsar sink configurations.
Supported Input Types
- Logs
- Metrics
Both are encoded according to the selected codec.