Prometheus Remote Write
Prometheus Remote Write Source
Collect metrics pushed by Prometheus using the remote write protocol.
This source allows worker to act as a Prometheus Remote Write receiver, accepting metric data sent by Prometheus servers and integrating it into worker-based telemetry pipelines.
Ingestion Model
- Prometheus sends metric samples using the Remote Write protocol
- worker exposes an HTTP endpoint to receive write requests
- Incoming requests are parsed and validated
- Metric samples are converted into worker metric events
- Successfully processed samples participate in end-to-end acknowledgements
This model enables worker to replace or complement long-term Prometheus storage backends.
Reliability and Acknowledgements
- End-to-end acknowledgements are supported
- Acknowledgement behavior is controlled globally or at the sink level
- Source-level acknowledgement configuration is deprecated
This ensures compatibility with Prometheus retry semantics and reliable delivery guarantees.
Source Configuration Parameters
sources.<id>.type
required string
Specifies the source type.
- Must be set to prometheus_remote_write
sources.<id>.address
required string
The socket address on which worker listens for incoming Prometheus remote write requests.
- Must include a port number
- Supports binding to all interfaces or specific addresses
Example format:
Authentication Configuration
Controls access to the remote write endpoint.
Authentication should always be used together with HTTPS.
sources.<id>.auth
optional object
Defines the authentication strategy for incoming HTTP requests.
sources.<id>.auth.strategy
required string enum
Selects the authentication mechanism.
Supported values:
- basic – HTTP Basic Authentication
- custom – Custom authentication using VRL logic
sources.<id>.auth.username
required string
The username used for Basic Authentication.
Relevant when: strategy = "basic"
sources.<id>.auth.password
required string
The password used for Basic Authentication.
Relevant when: strategy = "basic"
sources.<id>.auth.source
required string
A VRL boolean expression used to validate incoming requests.
- Receives the HTTP request context
- Must evaluate to true for the request to be accepted
Relevant when: strategy = "custom"
HTTP Keepalive Configuration
Controls connection lifecycle behavior for incoming HTTP clients.
sources.<id>.keepalive
optional object
Configures HTTP keepalive behavior.
sources.<id>.keepalive.max_connection_age_secs
optional uint
Maximum lifetime of an HTTP connection before worker closes it.
- A Connection: close header is sent when the limit is reached
- Applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 only
- Can be set to a very large value to effectively disable rotation
Default
sources.<id>.keepalive.max_connection_age_jitter_factor
optional float
Adds random jitter to the maximum connection age.
- Helps avoid synchronized connection termination
- Prevents connection storms
Example:
- 0.1 → actual lifetime varies between 90% and 110%
Default
Metadata Conflict Handling
sources.<id>.metadata_conflict_strategy
optional string enum
Defines how conflicting metric metadata is handled.
Supported values:
- ignore Silently keeps the first metadata entry Matches Prometheus and Thanos behavior
- reject Rejects the request with HTTP 400 Preserves strict metadata correctness
Default
Request Path Configuration
sources.<id>.path
optional string
Specifies the HTTP path used to receive remote write requests.
- Useful when exposing multiple services on the same address
- Must match Prometheus remote_write configuration
Default
Sample Validation
sources.<id>.skip_nan_values
optional bool
Controls handling of samples with NaN values.
- When enabled, samples with NaN values are discarded
- Prevents invalid metrics from entering downstream systems
Default
TLS Configuration
Secures communication between Prometheus and worker.
sources.<id>.tls
optional object
Defines TLS behavior for incoming connections.
sources.<id>.tls.enabled
optional bool
Enables TLS encryption.
- When enabled, a server certificate is required
sources.<id>.tls.crt_file
optional string
Path to the server certificate file.
- PEM, DER, or PKCS#12 formats supported
- Required when TLS is enabled
sources.<id>.tls.key_file
optional string
Path to the private key file.
- PEM or DER format
- Required unless using PKCS#12
sources.<id>.tls.key_pass
optional string
Passphrase for encrypted private keys.
sources.<id>.tls.ca_file
optional string
Additional CA certificate bundle.
- Used for client certificate verification
sources.<id>.tls.verify_certificate
optional bool
Enables certificate chain validation.
- Ensures certificates are trusted and not expired
- Strongly recommended for production
sources.<id>.tls.verify_hostname
optional bool
Enables hostname verification.
- Ensures the certificate matches the expected hostname
sources.<id>.tls.alpn_protocols
optional array[string]
Defines supported ALPN protocols in priority order.
Reliability Characteristics
- At-least-once delivery semantics
- Batch-oriented processing
- Backpressure-aware ingestion
- Prometheus-compatible retry behavior
- Stateless processing model
Common Use Cases
- Replacing Prometheus long-term storage
- Centralizing metrics ingestion
- Multi-cluster Prometheus federation
- Secure metric ingestion endpoints
- Unified logs–metrics–traces pipelines