Mezmo
Mezmo Sink (formerly LogDNA)
The Mezmo sink delivers log events to Mezmo over HTTP, typically in batches. This guide explains what each parameter does, with an emphasis on operational behavior and why you would tune it.
Required Identification & Auth
api_key (required)
The Mezmo Ingestion API key used to authenticate requests.
Operational impact:
- If invalid/expired, the sink will fail to deliver and buffering will grow (until limits are hit).
- Treat as a secret (rotate, scope, and store securely).
hostname (required)
The hostname attached to each batch of events (templating supported).
Why it matters:
- This is a primary dimension Mezmo uses for host-level views and filtering.
- If you run worker as a shared/centralized collector, you may want this to reflect the origin host (not the worker host), otherwise all logs can appear to come from the same machine.
Destination
endpoint (optional, default: https://logs.mezmo.com/)
The HTTP endpoint to send logs to (IP or hostname accepted).
Operational impact:
- Use this to target a specific Mezmo region/ingestion endpoint if your account requires it.
- Incorrect endpoint leads to connection failures, retries, and buffer buildup.
Event Attribution (Optional Metadata)
These fields are attached to each batch and help enrich/search logs in Mezmo.
ip (optional)
IP address attached to the batch.
When useful:
- For edge collectors, NAT gateways, or when “hostname” is not sufficient for identification.
- Helpful for correlation when hostnames are not unique or not stable.
mac (optional)
MAC address attached to the batch.
When useful:
- More common in on-prem / appliance-style deployments where MAC is a stable identifier.
- Less useful in cloud environments where MAC visibility is abstracted or changes.
tags (optional)
Tags attached to each batch (templating supported).
Operational impact:
- Tags are a powerful way to partition environments (e.g., prod, staging), teams, clusters, or tenant identifiers.
- Overusing dynamic tags can increase cardinality and make operations/search noisier.
default_app (optional, default: vector)
Default “app” assigned when an event lacks a file or app field.
What it does:
- Provides a fallback application grouping so logs don’t end up unlabeled.
Operational note:
- If your pipeline doesn’t populate app/file fields, this value becomes very important for usability in Mezmo.
default_env (optional, default: production)
Default environment assigned when an event lacks an env field.
Operational note:
- If you forget to set env per source, everything may look like production—dangerous for triage and dashboards.
Payload Shaping (Encoding)
encoding (optional)
Controls which fields are sent and how timestamps are represented.
encoding.only_fields (optional)
Allow-list of fields to include.
Why you’d use it:
- Reduce payload size and cost
- Prevent sensitive fields from leaving the environment
- Standardize a minimal schema
encoding.except_fields (optional)
Block-list of fields to exclude.
Why you’d use it:
- Quick redaction (tokens, passwords, PII fields, raw payloads)
encoding.timestamp_format (optional)
How timestamps are serialized (rfc3339, unix, unix_ms, etc.).
Operational impact:
- Choose a consistent format to avoid “time drift” confusion or parsing inconsistencies downstream.
Batching (Throughput vs Latency)
batch (optional)
Controls when worker flushes a batch to Mezmo.
batch.max_bytes (optional, default: 10,000,000 bytes)
Maximum uncompressed batch size before flushing.
Trade-off:
- Larger batches → higher throughput efficiency, but increased memory usage and more impact if a request fails.
- Smaller batches → lower latency and smaller retry “blast radius,” but more HTTP overhead.
batch.max_events (optional)
Maximum events per batch before flushing.
Trade-off:
- Useful when events are small but extremely high rate (controls per-request payload “shape”).
- Also helps constrain duplicate volume under retry (at-least-once behavior).
batch.timeout_secs (optional, default: 1s)
Maximum time a batch can wait before being flushed.
Trade-off:
- Lower timeout → fresher delivery (near-real-time), more requests.
- Higher timeout → fewer requests, more delay, potentially larger bursts.
Buffering (Durability vs Performance)
buffer (optional)
Controls what happens when Mezmo is slow/unreachable.
buffer.type (optional, default: memory)
- memory: fastest, loses buffered data on crash/restart
- disk: slower, but durable across restarts (requires disk sizing)
buffer.max_size (required)
Hard cap on total buffer memory/disk usage.
Operational impact:
- Too small → backpressure/drops sooner during outages.
- Too large → longer drain time after recovery and potentially large outbound bursts.
buffer.max_events (optional; memory only, default: 500)
Event-count cap for memory buffering.
buffer.when_full (optional, default: block)
- block: apply backpressure upstream (prefer when completeness matters)
- drop_newest: intentionally drop new events under sustained pressure (prefer when uptime/latency matters more than completeness)
Request Behavior (Reliability & Load Control)
request (optional)
Controls concurrency, timeouts, rate limits, and retry behavior.
Key controls to understand:
- request.concurrency
- adaptive (default): worker adjusts concurrency automatically based on observed latency.
- none or a fixed number: deterministic behavior, sometimes better for strict network policies.
- request.timeout_secs (default: 60s)
- Max time for a request before aborting.
- Too low can cause premature aborts → more retries → duplicates.
- request.retry_ settings*
- Retries follow a Fibonacci backoff (helps avoid synchronized retry storms).
- High retry limits can increase duplicate risk (since delivery is at-least-once).
- request.rate_limit_ settings*
- Caps request volume; useful to protect Mezmo endpoints and your own egress.