--- title: Prometheus Exporter slug: telemetry-pipeline/prometheus-exporter docTags: createdAt: 2025-04-21T01:02:36.877Z --- # Prometheus Exporter Sink The **Prometheus Exporter** sink exposes **metric events** as a local HTTP endpoint so that Prometheus can **scrape** them (typically on /metrics). This is the “pull model” (Prometheus pulls), unlike Remote Write which is “push model”. ## Important Warning: Cardinality High-cardinality metric names and label sets are risky in Prometheus because they increase: - Ingestion pressure and scrape payload size - Memory usage in both worker (export cache) and Prometheus - Query cost and operational instability Operational best practice: treat labels like a schema. Avoid identifiers that explode series count (user IDs, session IDs, IPs, IMSI/IMEI, per-request IDs, etc.). ## Rename / Compatibility Note This sink used to be called **prometheus** and is now **prometheus\_exporter**. Operationally, the rename matters for configuration compatibility and upgrade readiness (you must update the sink type during migration). ## Exposure / Listener ### address (optional, default: 0.0.0.0:9598) - Network address where worker exposes the scrape endpoint. - Affects: - Which interfaces are reachable (localhost-only vs all interfaces) - Firewall/security group requirements - Multi-tenant host considerations (port collisions) Metrics are exposed on the standard Prometheus path: **/metrics**. ## Authentication (HTTP) ### auth (optional) Defines authentication for HTTP requests to the exporter endpoint. Important operational note: **Use HTTP authentication only with HTTPS/TLS**, because credentials are transmitted via headers and rely on transport encryption. auth.strategy (required when auth is configured) Supported strategies: - **aws:** AWS request signing style authentication (for specific environments) - **basic:** HTTP Basic Auth (username + password) - **bearer:** Bearer token passed as-is - **custom:** Custom Authorization header value ### Basic Authentication auth.user (required when strategy = basic) - Username used for Basic Auth. auth.password (required when strategy = basic) - Password used for Basic Auth. ### Bearer Authentication auth.token (required when strategy = bearer) - Token provided in the Authorization header as a bearer token. - Useful behind gateways or service meshes enforcing token auth. ### Custom Authorization Header auth.value (required when strategy = custom) - Sets the Authorization header value directly. - Useful when integration expects a non-standard Authorization scheme. ### AWS Authentication auth.service (required when strategy = aws) - The AWS service name used for signing. - Must match the expected service identifier of the verifying endpoint/gateway. auth.auth.\* (required when strategy = aws) This nested block controls how credentials are obtained and how roles are assumed. Key parameters include: - **access\_key\_id / secret\_access\_key:** static credentials - **session\_token:** temporary credentials support - **credentials\_file / profile:** file-based credential selection - **assume\_role / external\_id / session\_name:** STS role assumption behavior - **region:** STS/service region targeting - **load\_timeout\_secs:** time limit to obtain usable credentials - **imds:** metadata-based credential fetching behavior (timeouts, retries) These settings matter most in cloud deployments where the agent relies on instance roles or rotates credentials. ## Distribution Metrics Rendering Prometheus cannot consume raw “distribution sample sets” directly; they must be represented as either histograms or summaries. ### distributions\_as\_summaries (optional, default: false) - When **false**, distributions are exported as aggregated **histograms** (typical recommendation for Prometheus). - When **true**, distributions are exported as aggregated **summaries** (quantile-focused view). - Operational impact: - Histograms are better for aggregation across instances and are generally preferred for SLO-style queries. - Summaries can be useful locally but are less ideal to aggregate fleet-wide. ### buckets (optional) - Defines default histogram bucket boundaries used for distribution-to-histogram conversion. - Impacts: - Accuracy of percentile approximation - Number of time series created (more buckets → more series → higher cost) ### quantiles (optional) - Defines quantiles used for distribution-to-summary conversion (only relevant when summaries are used). - Impacts: - How many quantile series are emitted - CPU/memory overhead during aggregation ## Namespace Management ### default\_namespace (optional) - Applied only when a metric has no existing namespace. - When applied, it prefixes the metric name using underscore separation. - Helps enforce naming hygiene and reduce collisions across multiple upstream metric sources. ## Flush / Expiration Behavior Because this sink is stateful, it caches metrics and periodically flushes/expunges entries. ### flush\_period\_secs (optional, default: 60) - Interval at which worker refreshes the exported metric set and removes stale metrics. - If a metric is not seen since the previous flush window, it is treated as expired and removed. - Critical operational rule: **Set this higher than the Prometheus scrape interval**, otherwise metrics can disappear between scrapes and create “gaps” or unstable dashboards. ## Timestamp Handling ### suppress\_timestamp (optional, default: false) - When enabled, worker omits timestamps in the Prometheus output. - Useful when: - Metrics have timestamps too old for Prometheus ingestion rules - You are replaying metrics from disk buffers - You are aggregating over long periods and timestamps drift into the past - Operational effect: Prometheus will use the scrape time as the sample timestamp, which can improve ingest acceptance in problematic timestamp scenarios. ## Buffering Even though Prometheus scrapes (“pull”), worker still maintains internal buffering/state for managing metric event handling and backpressure. ### buffer (optional) Controls internal buffer behavior. buffer.type (optional, default: memory) - **memory:** faster, but state can be lost on crash/restart - **disk:** more durable, but requires disk sizing and incurs I/O overhead buffer.max\_size (required) - Hard cap on memory/disk usage for buffering. - Too small → increased backpressure or dropped data (depending on when\_full). buffer.max\_events (optional; memory only) - Limits number of buffered events in memory. buffer.when\_full (optional, default: block) - **block:** backpressure upstream (preferable when losing metrics is unacceptable) - **drop\_newest:** drop newest events (preferable when maintaining system responsiveness is the priority) ## Healthcheck ### healthcheck.enabled (optional, default: true) - Validates at startup that the sink can initialize properly. - Helps catch bind failures (port in use), TLS misconfiguration, or auth issues early. ## Inputs ### inputs (required) - Defines which upstream metric sources/transforms feed this exporter. - Wildcards allow large-scale topologies without listing every component explicitly. ## TLS (for Scrape Endpoint Security) ### tls (optional) Controls HTTPS and certificate behavior for the exporter endpoint. tls.enabled (optional) - Enables TLS requirement for incoming scrape connections. - When enabling TLS for incoming connections, you generally need an identity certificate configured. tls.crt\_file / tls.key\_file (optional) - Server certificate and private key used to identify the exporter endpoint. - Required for HTTPS operation in most setups. tls.key\_pass (optional) - Passphrase for decrypting the private key file (if encrypted). tls.ca\_file (optional) - Additional trusted CA certificates. - Often used for mTLS setups or private PKI. tls.verify\_certificate (optional) - Enables certificate verification. - For server components, this is relevant when validating client certificates (mTLS). tls.verify\_hostname (optional) - Hostname verification behavior (mostly relevant to outgoing connections, but included for consistency). tls.alpn\_protocols (optional) - Controls ALPN protocol negotiation preferences (advanced/edge cases).