Prometheus Exporter
Prometheus Exporter Sink
The Prometheus Exporter sink exposes metric events as a local HTTP endpoint so that Prometheus can scrape them (typically on /metrics). This is the “pull model” (Prometheus pulls), unlike Remote Write which is “push model”.
Important Warning: Cardinality
High-cardinality metric names and label sets are risky in Prometheus because they increase:
- Ingestion pressure and scrape payload size
- Memory usage in both worker (export cache) and Prometheus
- Query cost and operational instability
Operational best practice: treat labels like a schema. Avoid identifiers that explode series count (user IDs, session IDs, IPs, IMSI/IMEI, per-request IDs, etc.).
Rename / Compatibility Note
This sink used to be called prometheus and is now prometheus_exporter. Operationally, the rename matters for configuration compatibility and upgrade readiness (you must update the sink type during migration).
Exposure / Listener
address (optional, default: 0.0.0.0:9598)
- Network address where worker exposes the scrape endpoint.
- Affects:
- Which interfaces are reachable (localhost-only vs all interfaces)
- Firewall/security group requirements
- Multi-tenant host considerations (port collisions)
Metrics are exposed on the standard Prometheus path: /metrics.
Authentication (HTTP)
auth (optional)
Defines authentication for HTTP requests to the exporter endpoint.
Important operational note: Use HTTP authentication only with HTTPS/TLS, because credentials are transmitted via headers and rely on transport encryption.
auth.strategy (required when auth is configured)
Supported strategies:
- aws: AWS request signing style authentication (for specific environments)
- basic: HTTP Basic Auth (username + password)
- bearer: Bearer token passed as-is
- custom: Custom Authorization header value
Basic Authentication
auth.user (required when strategy = basic)
- Username used for Basic Auth.
auth.password (required when strategy = basic)
- Password used for Basic Auth.
Bearer Authentication
auth.token (required when strategy = bearer)
- Token provided in the Authorization header as a bearer token.
- Useful behind gateways or service meshes enforcing token auth.
Custom Authorization Header
auth.value (required when strategy = custom)
- Sets the Authorization header value directly.
- Useful when integration expects a non-standard Authorization scheme.
AWS Authentication
auth.service (required when strategy = aws)
- The AWS service name used for signing.
- Must match the expected service identifier of the verifying endpoint/gateway.
auth.auth.* (required when strategy = aws)
This nested block controls how credentials are obtained and how roles are assumed. Key parameters include:
- access_key_id / secret_access_key: static credentials
- session_token: temporary credentials support
- credentials_file / profile: file-based credential selection
- assume_role / external_id / session_name: STS role assumption behavior
- region: STS/service region targeting
- load_timeout_secs: time limit to obtain usable credentials
- imds: metadata-based credential fetching behavior (timeouts, retries)
These settings matter most in cloud deployments where the agent relies on instance roles or rotates credentials.
Distribution Metrics Rendering
Prometheus cannot consume raw “distribution sample sets” directly; they must be represented as either histograms or summaries.
distributions_as_summaries (optional, default: false)
- When false, distributions are exported as aggregated histograms (typical recommendation for Prometheus).
- When true, distributions are exported as aggregated summaries (quantile-focused view).
- Operational impact:
- Histograms are better for aggregation across instances and are generally preferred for SLO-style queries.
- Summaries can be useful locally but are less ideal to aggregate fleet-wide.
buckets (optional)
- Defines default histogram bucket boundaries used for distribution-to-histogram conversion.
- Impacts:
- Accuracy of percentile approximation
- Number of time series created (more buckets → more series → higher cost)
quantiles (optional)
- Defines quantiles used for distribution-to-summary conversion (only relevant when summaries are used).
- Impacts:
- How many quantile series are emitted
- CPU/memory overhead during aggregation
Namespace Management
default_namespace (optional)
- Applied only when a metric has no existing namespace.
- When applied, it prefixes the metric name using underscore separation.
- Helps enforce naming hygiene and reduce collisions across multiple upstream metric sources.
Flush / Expiration Behavior
Because this sink is stateful, it caches metrics and periodically flushes/expunges entries.
flush_period_secs (optional, default: 60)
- Interval at which worker refreshes the exported metric set and removes stale metrics.
- If a metric is not seen since the previous flush window, it is treated as expired and removed.
- Critical operational rule: Set this higher than the Prometheus scrape interval, otherwise metrics can disappear between scrapes and create “gaps” or unstable dashboards.
Timestamp Handling
suppress_timestamp (optional, default: false)
- When enabled, worker omits timestamps in the Prometheus output.
- Useful when:
- Metrics have timestamps too old for Prometheus ingestion rules
- You are replaying metrics from disk buffers
- You are aggregating over long periods and timestamps drift into the past
- Operational effect: Prometheus will use the scrape time as the sample timestamp, which can improve ingest acceptance in problematic timestamp scenarios.
Buffering
Even though Prometheus scrapes (“pull”), worker still maintains internal buffering/state for managing metric event handling and backpressure.
buffer (optional)
Controls internal buffer behavior.
buffer.type (optional, default: memory)
- memory: faster, but state can be lost on crash/restart
- disk: more durable, but requires disk sizing and incurs I/O overhead
buffer.max_size (required)
- Hard cap on memory/disk usage for buffering.
- Too small → increased backpressure or dropped data (depending on when_full).
buffer.max_events (optional; memory only)
- Limits number of buffered events in memory.
buffer.when_full (optional, default: block)
- block: backpressure upstream (preferable when losing metrics is unacceptable)
- drop_newest: drop newest events (preferable when maintaining system responsiveness is the priority)
Healthcheck
healthcheck.enabled (optional, default: true)
- Validates at startup that the sink can initialize properly.
- Helps catch bind failures (port in use), TLS misconfiguration, or auth issues early.
Inputs
inputs (required)
- Defines which upstream metric sources/transforms feed this exporter.
- Wildcards allow large-scale topologies without listing every component explicitly.
TLS (for Scrape Endpoint Security)
tls (optional)
Controls HTTPS and certificate behavior for the exporter endpoint.
tls.enabled (optional)
- Enables TLS requirement for incoming scrape connections.
- When enabling TLS for incoming connections, you generally need an identity certificate configured.
tls.crt_file / tls.key_file (optional)
- Server certificate and private key used to identify the exporter endpoint.
- Required for HTTPS operation in most setups.
tls.key_pass (optional)
- Passphrase for decrypting the private key file (if encrypted).
tls.ca_file (optional)
- Additional trusted CA certificates.
- Often used for mTLS setups or private PKI.
tls.verify_certificate (optional)
- Enables certificate verification.
- For server components, this is relevant when validating client certificates (mTLS).
tls.verify_hostname (optional)
- Hostname verification behavior (mostly relevant to outgoing connections, but included for consistency).
tls.alpn_protocols (optional)
- Controls ALPN protocol negotiation preferences (advanced/edge cases).