Loki
Loki Sink
The Loki sink pushes log events from worker into a Grafana Loki cluster using Loki’s /loki/api/v1/push endpoint. This is a batching sink with at-least-once delivery semantics (duplicates are possible).
Destination & Routing
endpoint (required)
Base URL of the Loki instance (the path is appended to this).
Operational impact:
- Wrong endpoint → connection errors + retries + buffer growth.
- In k8s, this is usually the Loki gateway or distributor service.
path (optional, default: /loki/api/v1/push)
HTTP path appended to endpoint.
Why it matters:
- Leave default for standard Loki.
- Change only if you’re behind a reverse proxy / gateway with a custom route.
tenant_id (optional, template)
Sets the Loki tenant for multi-tenant Loki (sent as tenant header).
Operational impact:
- Required when Loki is running in multi-tenant mode (common in “shared Loki” setups).
- Template support is powerful, but dangerous: tenant explosion = operational pain + potential security boundary issues.
Labels & Metadata (The Most Important Part)
In Loki, labels define your index, and everything else is the log payload. Getting labels right is the core admin task.
labels (optional, object; keys & values templateable)
A set of labels attached to each batch of events. Supports label expansion (e.g., kubernetes.*-style expansion rules per docs).
Operational impact:
- Cardinality killer: high-cardinality labels (request_id, session_id, user_id, IP, full URL, etc.) will wreck Loki performance and cost.
- Keep labels to stable dimensions: cluster, namespace, app, pod (sometimes), node, env, job, instance (careful).
Practical guidance:
- If you can only pick 3–6 labels, pick the ones you use to “narrow down” searches quickly.
- Put everything high-cardinality into the log body or structured metadata (see below), not labels.
structured_metadata (optional, object; keys & values templateable)
Additional metadata attached to events, using Loki’s structured metadata support.
Operational impact:
- Designed to carry extra searchable context without exploding label cardinality.
- Still don’t treat it like a dumping ground for per-event unique IDs, but it’s much safer than labels for richer context.
remove_label_fields (optional, default: false)
If true, worker deletes fields from the event once they’ve been used as labels.
Why you’d enable it:
- Reduce payload size and avoid duplicating the same info both as label and in JSON/body. Why you might not:
- If downstream consumers expect the field in the log body (for display, parsing, or alerting).
remove_structured_metadata_fields (optional, default: false)
Same idea as above, but for structured metadata fields.
Timestamp Handling (Common Loki Pain Point)
remove_timestamp (optional, default: true)
Whether to remove the timestamp from the event payload (timestamp is still sent as Loki event metadata).
Operational impact:
- Default true typically keeps payload cleaner while Loki still indexes by timestamp.
- Set false if you explicitly need the timestamp field inside the stored log line (e.g., you rely on it in parsing/regex or for downstream exports).
out_of_order_action (optional, default: accept)
What to do if events arrive with timestamps that are older than the latest timestamps already pushed for the same stream.
Options:
- accept: send as-is and let Loki handle it (requires Loki ≥ 2.4.0)
- drop: drop out-of-order events (data loss, but avoids ingestion errors)
- rewrite_timestamp: force timestamp to “latest seen” (no drop, but time accuracy is compromised)
Operational decision:
- If your sources can produce out-of-order timestamps (multi-node forwarding, buffers, replays, batch merges), accept is best if your Loki version supports it.
- Use rewrite_timestamp if you can’t upgrade Loki and you’d rather keep the message than keep exact time.
- Use drop only when correctness and stability matter more than completeness.
Compression & Encoding
compression (optional, default: snappy)
Compression used for push requests. Note: snappy implies Protobuf push format.
Operational impact:
- snappy is usually best for Loki (good CPU/network tradeoff).
- If you have proxy/load balancer quirks or strict intermediaries, sometimes switching compression can help troubleshooting—but usually you keep default.
encoding (required)
How events are turned into bytes before shipping.
Important reality:
- For Loki, you normally keep log payload as JSON or text; your labels and metadata carry key routing fields.
- If you choose codecs like raw_message or text, ensure the message field is present; transforms that remove/rename it can lead to empty outputs.
Batching (Latency vs Throughput)
batch.max_bytes (optional, default: 1,000,000 bytes)
Max uncompressed size before flushing.
Trade-off:
- Larger → fewer requests, better throughput efficiency, larger retry “blast radius”.
- Smaller → lower latency, smaller retry impact, more overhead.
batch.max_events (optional, default: 100000)
Max events per batch before flushing.
Operational note:
- The default is huge; in high-volume environments this can create very large batches for single streams.
- Consider lowering if you see memory pressure or large “retry storms”.
batch.timeout_secs (optional, default: 1s)
Max wait time before flushing a batch.
Trade-off:
- Lower timeout = near-real-time logs, more requests.
- Higher timeout = fewer requests, more delay.
Buffering (Outage Tolerance & Backpressure)
buffer.type (optional, default: memory)
- memory: fastest, data lost on crash/restart
- disk: durable across restarts (slower; requires disk sizing)
buffer.max_size (required)
Hard cap on memory/disk usage for buffering.
Operational impact:
- Too small → you’ll hit backpressure/drops quickly during Loki downtime.
- Too large → long drain times after Loki recovers; can cause bursty ingestion.
buffer.when_full (optional, default: block)
- block: backpressure upstream (best when “don’t lose logs” is the priority)
- drop_newest: sacrifice newest logs to keep pipeline alive (best when uptime/latency matters most)
Authentication & Transport Security
auth (optional)
HTTP auth strategies supported:
- basic (user/password)
- bearer (token/JWT)
- aws (SigV4 signing)
- custom (raw Authorization header value)
Operational guidance:
- Use auth only with HTTPS (credentials are headers).
- For Grafana Cloud Loki or gateways, bearer tokens are common.
tls (optional)
TLS settings for secure transport (CA, client certs, verification toggles).
Operational guidance:
- Keep verify_certificate and verify_hostname enabled unless you’re debugging in a lab.
- If you’re using mTLS, crt_file + key_file become mandatory.
Proxies (Enterprise Egress)
proxy.enabled (optional, default: true)
Enable proxy support.
proxy.http / proxy.https / proxy.no_proxy
Standard outbound proxy controls.
Operational impact:
- Misconfigured proxies commonly look like timeouts + infinite retries + buffer growth.
- Use no_proxy to bypass proxy for in-cluster Loki endpoints.