---
title: Loki
slug: telemetry-pipeline/loki
docTags: 
createdAt: 2025-04-21T01:01:27.191Z
---

# Loki Sink 

The **Loki** sink pushes **log events** from worker into a **Grafana Loki** cluster using Loki’s /loki/api/v1/push endpoint. This is a **batching** sink with **at-least-once** delivery semantics (duplicates are possible).

##

## Destination & Routing

### endpoint (required)

Base URL of the Loki instance (the path is appended to this).

Operational impact:

- Wrong endpoint → connection errors + retries + buffer growth. 
- In k8s, this is usually the Loki *gateway* or distributor service. 

### path (optional, default: /loki/api/v1/push)

HTTP path appended to endpoint.

Why it matters:

- Leave default for standard Loki. 
- Change only if you’re behind a reverse proxy / gateway with a custom route. 

### tenant\_id (optional, template)

Sets the Loki tenant for multi-tenant Loki (sent as tenant header).

Operational impact:

- Required when Loki is running in multi-tenant mode (common in “shared Loki” setups). 
- Template support is powerful, but **dangerous**: tenant explosion = operational pain + potential security boundary issues. 

## Labels & Metadata (The Most Important Part)

In Loki, **labels define your index**, and everything else is the log payload. Getting labels right is *the* core admin task.

### labels (optional, object; keys & values templateable)

A set of labels attached to each batch of events. Supports label expansion (e.g., kubernetes.\*-style expansion rules per docs).

Operational impact:

- **Cardinality killer**: high-cardinality labels (request\_id, session\_id, user\_id, IP, full URL, etc.) will wreck Loki performance and cost. 
- Keep labels to stable dimensions: cluster, namespace, app, pod (sometimes), node, env, job, instance (careful). 

Practical guidance:

- If you can only pick 3–6 labels, pick the ones you use to “narrow down” searches quickly. 
- Put everything high-cardinality into the **log body** or **structured metadata** (see below), not labels. 

### structured\_metadata (optional, object; keys & values templateable)

Additional metadata attached to events, using Loki’s structured metadata support.

Operational impact:

- Designed to carry extra searchable context **without** exploding label cardinality. 
- Still don’t treat it like a dumping ground for per-event unique IDs, but it’s much safer than labels for richer context. 

### remove\_label\_fields (optional, default: false)

If true, worker deletes fields from the event once they’ve been used as labels.

Why you’d enable it:

- Reduce payload size and avoid duplicating the same info both as label and in JSON/body.
   Why you might not: 
- If downstream consumers expect the field in the log body (for display, parsing, or alerting). 

### remove\_structured\_metadata\_fields (optional, default: false)

Same idea as above, but for structured metadata fields.

## Timestamp Handling (Common Loki Pain Point)

### remove\_timestamp (optional, default: true)

Whether to remove the timestamp from the event payload (timestamp is still sent as Loki event metadata).

Operational impact:

- Default true typically keeps payload cleaner while Loki still indexes by timestamp. 
- Set false if you explicitly need the timestamp field inside the stored log line (e.g., you rely on it in parsing/regex or for downstream exports). 

### out\_of\_order\_action (optional, default: accept)

What to do if events arrive with timestamps that are *older* than the latest timestamps already pushed for the same stream.

Options:

- **accept**: send as-is and let Loki handle it (**requires Loki ≥ 2.4.0**) 
- **drop**: drop out-of-order events (data loss, but avoids ingestion errors) 
- **rewrite\_timestamp**: force timestamp to “latest seen” (no drop, but time accuracy is compromised) 

Operational decision:

- If your sources can produce out-of-order timestamps (multi-node forwarding, buffers, replays, batch merges), **accept** is best *if* your Loki version supports it. 
- Use **rewrite\_timestamp** if you can’t upgrade Loki and you’d rather keep the message than keep exact time. 
- Use **drop** only when correctness and stability matter more than completeness. 

## Compression & Encoding

### compression (optional, default: snappy)

Compression used for push requests. Note: **snappy implies Protobuf** push format.

Operational impact:

- **snappy** is usually best for Loki (good CPU/network tradeoff). 
- If you have proxy/load balancer quirks or strict intermediaries, sometimes switching compression can help troubleshooting—but usually you keep default. 

### encoding (required)

How events are turned into bytes before shipping.

Important reality:

- For Loki, you normally keep log payload as JSON or text; your *labels* and *metadata* carry key routing fields. 
- If you choose codecs like raw\_message or text, ensure the message field is present; transforms that remove/rename it can lead to empty outputs. 

## Batching (Latency vs Throughput)

### batch.max\_bytes (optional, default: 1,000,000 bytes)

Max uncompressed size before flushing.

Trade-off:

- Larger → fewer requests, better throughput efficiency, larger retry “blast radius”. 
- Smaller → lower latency, smaller retry impact, more overhead. 

### batch.max\_events (optional, default: 100000)

Max events per batch before flushing.

Operational note:

- The default is huge; in high-volume environments this can create very large batches for single streams. 
- Consider lowering if you see memory pressure or large “retry storms”. 

### batch.timeout\_secs (optional, default: 1s)

Max wait time before flushing a batch.

Trade-off:

- Lower timeout = near-real-time logs, more requests. 
- Higher timeout = fewer requests, more delay. 

## Buffering (Outage Tolerance & Backpressure)

### buffer.type (optional, default: memory)

- **memory**: fastest, data lost on crash/restart 
- **disk**: durable across restarts (slower; requires disk sizing) 

### buffer.max\_size (required)

Hard cap on memory/disk usage for buffering.

Operational impact:

- Too small → you’ll hit backpressure/drops quickly during Loki downtime. 
- Too large → long drain times after Loki recovers; can cause bursty ingestion. 

### buffer.when\_full (optional, default: block)

- **block**: backpressure upstream (best when “don’t lose logs” is the priority) 
- **drop\_newest**: sacrifice newest logs to keep pipeline alive (best when uptime/latency matters most) 

## Authentication & Transport Security

### auth (optional)

HTTP auth strategies supported:

- basic (user/password) 
- bearer (token/JWT) 
- aws (SigV4 signing) 
- custom (raw Authorization header value) 

Operational guidance:

- Use auth **only with HTTPS** (credentials are headers). 
- For Grafana Cloud Loki or gateways, bearer tokens are common. 

### tls (optional)

TLS settings for secure transport (CA, client certs, verification toggles).

Operational guidance:

- Keep verify\_certificate and verify\_hostname enabled unless you’re debugging in a lab. 
- If you’re using mTLS, crt\_file + key\_file become mandatory. 

## Proxies (Enterprise Egress)

### proxy.enabled (optional, default: true)

Enable proxy support.

### proxy.http / proxy.https / proxy.no\_proxy

Standard outbound proxy controls.

Operational impact:

- Misconfigured proxies commonly look like timeouts + infinite retries + buffer growth. 
- Use no\_proxy to bypass proxy for in-cluster Loki endpoints. 

##