MQTT

mqtt sink the mqtt sink delivers log events to an mqtt broker by publishing messages to a configured topic this guide explains what each parameter does and the operational trade offs core connectivity host (required) the mqtt broker address (dns name or ip) operational impact incorrect host or network policy issues cause publish failures and buffer growth for ha brokers (lb / cluster), make sure host matches the broker access method used in your environment port (optional, default 1883) tcp port used to connect to the broker operational impact 1883 is the common plaintext mqtt port tls deployments commonly use different ports (often 8883) depending on broker configuration keep alive (optional, default 60) mqtt keep alive interval in seconds what it controls heartbeat/ping behavior to keep the connection active and detect dead connections operational impact lower values detect failures sooner but add more keep alive traffic higher values reduce overhead but can increase time to detect broken connections publishing target (routing) topic (required) the mqtt topic to publish to (templating supported) what it controls how downstream subscribers receive messages (topic filters and hierarchies are central to mqtt routing) operational impact highly dynamic topics (per host/per container/per user) increase topic cardinality and can complicate subscriptions and broker performance a stable topic taxonomy is easier to operate and secure (acls are usually topic based) delivery semantics (mqtt qos and retained messages) quality of service (optional, default atleastonce) mqtt quality of service (qos) level used for publishes options atmostonce (qos 0) “fire and forget” – lowest latency, highest loss risk atleastonce (qos 1) broker acknowledges receipt – duplicates possible exactlyonce (qos 2) strongest delivery guarantee – highest overhead/latency operational impact qos 1 is typically the default trade off for telemetry better than qos 0, but duplicates can happen (important for downstream idempotency) qos 2 can significantly reduce throughput and increase broker/client cpu due to the extra handshake retain (optional, default false) whether published messages should be retained by the broker what it does if enabled, the broker stores the last retained message for a topic and delivers it immediately to new subscribers operational impact useful for “state like” topics (e g , latest status/health), not usually desirable for high volume logs can create misleading behavior for logs (new subscribers receive an old retained log message and interpret it as current) client session behavior client id (optional) mqtt client identifier operational impact many brokers require client ids to be unique per connection stable client id can be important when you rely on persistent sessions (especially with clean session=false) collisions (two clients using the same id) can cause frequent disconnects clean session (optional, default false) controls whether the broker should “clean” session state at login what it means true start fresh each time; broker forgets subscriptions/queued messages tied to that client session false session may persist across reconnects (depending on broker and mqtt version behavior) operational impact persistent sessions can help continuity in intermittent networks, but require stable client id and careful broker resource planning clean sessions are simpler operationally and reduce broker side session state authentication user (optional) mqtt username password (optional) mqtt password operational impact use only over tls; otherwise credentials travel without transport encryption treat as sensitive secret material and rotate according to policy payload format (encoding) encoding (required) controls how each log event is serialized into bytes before publishing encoding codec (required) selects the wire format (commonly json/text/logfmt/raw message, plus other structured codecs) operational considerations json best for structured downstream parsing, typically larger payloads text/raw message simplest and smallest, but less queryable downstream if you later want to parse logs centrally, selecting a structured encoding early saves effort field filtering and timestamps encoding only fields / encoding except fields keep payloads small and avoid leaking sensitive fields encoding timestamp format ensures consistent timestamp representation for downstream parsing and correlation buffering and backpressure buffer (optional) controls how worker buffers log events when the broker is slow or unreachable buffer type (optional, default memory) memory fast, but buffered data is lost on crash/restart disk durable across restarts, slower; requires disk sizing and io planning buffer max size (required) hard cap on buffer usage operational impact too small → pipeline applies backpressure early or drops (depending on when full) too large → long drain time and potential burst load on broker after recovery buffer max events (optional; memory only, default 500) event count limit for memory buffers buffer when full (optional, default block) block apply backpressure upstream (prefer when you want completeness) drop newest drop new events under sustained pressure (prefer when uptime/latency matters more than completeness) safety limits max packet size (optional, default 10240) maximum packet size for mqtt messages operational impact protects broker/client from very large events if too low, large log events may fail to publish (and may trigger retries/backlog) tls (transport security) tls (optional) tls settings for connecting to the broker securely key knobs tls enabled enable tls tls ca file trust roots tls crt file / tls key file client certificate identity (mtls), if required tls verify certificate / tls verify hostname validation controls operational guidance don’t disable verification in production unless you explicitly accept the risk