---
title: MQTT
slug: telemetry-pipeline/mqtt
docTags: 
createdAt: 2025-04-21T01:01:36.421Z
---

# MQTT Sink

The **MQTT** sink delivers **log events** to an **MQTT broker** by publishing messages to a configured **topic**. This guide explains what each parameter does and the operational trade-offs.

## Core Connectivity

### host (required)

The MQTT broker address (DNS name or IP).

Operational impact:

- Incorrect host or network policy issues cause publish failures and buffer growth. 
- For HA brokers (LB / cluster), make sure host matches the broker access method used in your environment. 

### port (optional, default: 1883)

TCP port used to connect to the broker.

Operational impact:

- **1883** is the common plaintext MQTT port. 
- TLS deployments commonly use different ports (often 8883) depending on broker configuration. 

### keep\_alive (optional, default: 60)

MQTT keep-alive interval in seconds.

What it controls:

- Heartbeat/ping behavior to keep the connection active and detect dead connections. 

Operational impact:

- Lower values detect failures sooner but add more keep-alive traffic. 
- Higher values reduce overhead but can increase time-to-detect broken connections. 

## Publishing Target (Routing)

### topic (required)

The MQTT topic to publish to (templating supported).

What it controls:

- How downstream subscribers receive messages (topic filters and hierarchies are central to MQTT routing). 

Operational impact:

- Highly dynamic topics (per-host/per-container/per-user) increase topic cardinality and can complicate subscriptions and broker performance. 
- A stable topic taxonomy is easier to operate and secure (ACLs are usually topic-based). 

## Delivery Semantics (MQTT QoS and Retained Messages)

### quality\_of\_service (optional, default: atleastonce)

MQTT Quality of Service (QoS) level used for publishes.

Options:

- **atmostonce (QoS 0):** “fire and forget” – lowest latency, highest loss risk. 
- **atleastonce (QoS 1):** broker acknowledges receipt – duplicates possible. 
- **exactlyonce (QoS 2):** strongest delivery guarantee – highest overhead/latency. 

Operational impact:

- QoS 1 is typically the default trade-off for telemetry: better than QoS 0, but duplicates can happen (important for downstream idempotency). 
- QoS 2 can significantly reduce throughput and increase broker/client CPU due to the extra handshake. 

### retain (optional, default: false)

Whether published messages should be retained by the broker.

What it does:

- If enabled, the broker stores the last retained message for a topic and delivers it immediately to new subscribers. 

Operational impact:

- Useful for “state-like” topics (e.g., latest status/health), not usually desirable for high-volume logs. 
- Can create misleading behavior for logs (new subscribers receive an old retained log message and interpret it as current). 

## Client Session Behavior

### client\_id (optional)

MQTT client identifier.

Operational impact:

- Many brokers require client IDs to be unique per connection. 
- Stable client\_id can be important when you rely on persistent sessions (especially with clean\_session=false). 
- Collisions (two clients using the same ID) can cause frequent disconnects. 

### clean\_session (optional, default: false)

Controls whether the broker should “clean” session state at login.

What it means:

- **true:** start fresh each time; broker forgets subscriptions/queued messages tied to that client session. 
- **false:** session may persist across reconnects (depending on broker and MQTT version behavior). 

Operational impact:

- Persistent sessions can help continuity in intermittent networks, but require stable client\_id and careful broker resource planning. 
- Clean sessions are simpler operationally and reduce broker-side session state. 

## Authentication

### user (optional)

MQTT username.

### password (optional)

MQTT password.

Operational impact:

- Use only over TLS; otherwise credentials travel without transport encryption. 
- Treat as sensitive secret material and rotate according to policy. 

## Payload Format (Encoding)

### encoding (required)

Controls how each log event is serialized into bytes before publishing.

### encoding.codec (required)

Selects the wire format (commonly json/text/logfmt/raw\_message, plus other structured codecs).

Operational considerations:

- **json:** best for structured downstream parsing, typically larger payloads. 
- **text/raw\_message:** simplest and smallest, but less queryable downstream. 
- If you later want to parse logs centrally, selecting a structured encoding early saves effort. 

### Field filtering and timestamps

- **encoding.only\_fields / encoding.except\_fields:** keep payloads small and avoid leaking sensitive fields. 
- **encoding.timestamp\_format:** ensures consistent timestamp representation for downstream parsing and correlation. 

## Buffering and Backpressure

### buffer (optional)

Controls how worker buffers log events when the broker is slow or unreachable.

buffer.type (optional, default: memory)

- **memory:** fast, but buffered data is lost on crash/restart 
- **disk:** durable across restarts, slower; requires disk sizing and IO planning 

buffer.max\_size (required)

Hard cap on buffer usage.

Operational impact:

- Too small → pipeline applies backpressure early or drops (depending on when\_full) 
- Too large → long drain time and potential burst load on broker after recovery 

buffer.max\_events (optional; memory only, default: 500)

Event-count limit for memory buffers.

buffer.when\_full (optional, default: block)

- **block:** apply backpressure upstream (prefer when you want completeness) 
- **drop\_newest:** drop new events under sustained pressure (prefer when uptime/latency matters more than completeness) 

## Safety Limits

### max\_packet\_size (optional, default: 10240)

Maximum packet size for MQTT messages.

Operational impact:

- Protects broker/client from very large events. 
- If too low, large log events may fail to publish (and may trigger retries/backlog). 

## TLS (Transport Security)

### tls (optional)

TLS settings for connecting to the broker securely.

Key knobs:

- **tls.enabled:** enable TLS 
- **tls.ca\_file:** trust roots 
- **tls.crt\_file / tls.key\_file:** client certificate identity (mTLS), if required 
- **tls.verify\_certificate / tls.verify\_hostname:** validation controls 

Operational guidance:

- Don’t disable verification in production unless you explicitly accept the risk.