MQTT
MQTT Sink
The MQTT sink delivers log events to an MQTT broker by publishing messages to a configured topic. This guide explains what each parameter does and the operational trade-offs.
Core Connectivity
host (required)
The MQTT broker address (DNS name or IP).
Operational impact:
- Incorrect host or network policy issues cause publish failures and buffer growth.
- For HA brokers (LB / cluster), make sure host matches the broker access method used in your environment.
port (optional, default: 1883)
TCP port used to connect to the broker.
Operational impact:
- 1883 is the common plaintext MQTT port.
- TLS deployments commonly use different ports (often 8883) depending on broker configuration.
keep_alive (optional, default: 60)
MQTT keep-alive interval in seconds.
What it controls:
- Heartbeat/ping behavior to keep the connection active and detect dead connections.
Operational impact:
- Lower values detect failures sooner but add more keep-alive traffic.
- Higher values reduce overhead but can increase time-to-detect broken connections.
Publishing Target (Routing)
topic (required)
The MQTT topic to publish to (templating supported).
What it controls:
- How downstream subscribers receive messages (topic filters and hierarchies are central to MQTT routing).
Operational impact:
- Highly dynamic topics (per-host/per-container/per-user) increase topic cardinality and can complicate subscriptions and broker performance.
- A stable topic taxonomy is easier to operate and secure (ACLs are usually topic-based).
Delivery Semantics (MQTT QoS and Retained Messages)
quality_of_service (optional, default: atleastonce)
MQTT Quality of Service (QoS) level used for publishes.
Options:
- atmostonce (QoS 0): “fire and forget” – lowest latency, highest loss risk.
- atleastonce (QoS 1): broker acknowledges receipt – duplicates possible.
- exactlyonce (QoS 2): strongest delivery guarantee – highest overhead/latency.
Operational impact:
- QoS 1 is typically the default trade-off for telemetry: better than QoS 0, but duplicates can happen (important for downstream idempotency).
- QoS 2 can significantly reduce throughput and increase broker/client CPU due to the extra handshake.
retain (optional, default: false)
Whether published messages should be retained by the broker.
What it does:
- If enabled, the broker stores the last retained message for a topic and delivers it immediately to new subscribers.
Operational impact:
- Useful for “state-like” topics (e.g., latest status/health), not usually desirable for high-volume logs.
- Can create misleading behavior for logs (new subscribers receive an old retained log message and interpret it as current).
Client Session Behavior
client_id (optional)
MQTT client identifier.
Operational impact:
- Many brokers require client IDs to be unique per connection.
- Stable client_id can be important when you rely on persistent sessions (especially with clean_session=false).
- Collisions (two clients using the same ID) can cause frequent disconnects.
clean_session (optional, default: false)
Controls whether the broker should “clean” session state at login.
What it means:
- true: start fresh each time; broker forgets subscriptions/queued messages tied to that client session.
- false: session may persist across reconnects (depending on broker and MQTT version behavior).
Operational impact:
- Persistent sessions can help continuity in intermittent networks, but require stable client_id and careful broker resource planning.
- Clean sessions are simpler operationally and reduce broker-side session state.
Authentication
user (optional)
MQTT username.
password (optional)
MQTT password.
Operational impact:
- Use only over TLS; otherwise credentials travel without transport encryption.
- Treat as sensitive secret material and rotate according to policy.
Payload Format (Encoding)
encoding (required)
Controls how each log event is serialized into bytes before publishing.
encoding.codec (required)
Selects the wire format (commonly json/text/logfmt/raw_message, plus other structured codecs).
Operational considerations:
- json: best for structured downstream parsing, typically larger payloads.
- text/raw_message: simplest and smallest, but less queryable downstream.
- If you later want to parse logs centrally, selecting a structured encoding early saves effort.
Field filtering and timestamps
- encoding.only_fields / encoding.except_fields: keep payloads small and avoid leaking sensitive fields.
- encoding.timestamp_format: ensures consistent timestamp representation for downstream parsing and correlation.
Buffering and Backpressure
buffer (optional)
Controls how worker buffers log events when the broker is slow or unreachable.
buffer.type (optional, default: memory)
- memory: fast, but buffered data is lost on crash/restart
- disk: durable across restarts, slower; requires disk sizing and IO planning
buffer.max_size (required)
Hard cap on buffer usage.
Operational impact:
- Too small → pipeline applies backpressure early or drops (depending on when_full)
- Too large → long drain time and potential burst load on broker after recovery
buffer.max_events (optional; memory only, default: 500)
Event-count limit for memory buffers.
buffer.when_full (optional, default: block)
- block: apply backpressure upstream (prefer when you want completeness)
- drop_newest: drop new events under sustained pressure (prefer when uptime/latency matters more than completeness)
Safety Limits
max_packet_size (optional, default: 10240)
Maximum packet size for MQTT messages.
Operational impact:
- Protects broker/client from very large events.
- If too low, large log events may fail to publish (and may trigger retries/backlog).
TLS (Transport Security)
tls (optional)
TLS settings for connecting to the broker securely.
Key knobs:
- tls.enabled: enable TLS
- tls.ca_file: trust roots
- tls.crt_file / tls.key_file: client certificate identity (mTLS), if required
- tls.verify_certificate / tls.verify_hostname: validation controls
Operational guidance:
- Don’t disable verification in production unless you explicitly accept the risk.