--- title: Prometheus Remote Write slug: telemetry-pipeline/prometheus-remote-write docTags: createdAt: 2025-04-21T00:45:22.212Z --- # Prometheus Remote Write Source Collect metrics pushed by Prometheus using the remote write protocol. This source allows worker to act as a Prometheus Remote Write receiver, accepting metric data sent by Prometheus servers and integrating it into worker-based telemetry pipelines. ## Ingestion Model - Prometheus sends metric samples using the Remote Write protocol - worker exposes an HTTP endpoint to receive write requests - Incoming requests are parsed and validated - Metric samples are converted into worker metric events - Successfully processed samples participate in end-to-end acknowledgements This model enables worker to replace or complement long-term Prometheus storage backends. ## Reliability and Acknowledgements - End-to-end acknowledgements are supported - Acknowledgement behavior is controlled globally or at the sink level - Source-level acknowledgement configuration is deprecated This ensures compatibility with Prometheus retry semantics and reliable delivery guarantees. ## Source Configuration Parameters ## sources.\.type **required** **string** Specifies the source type. - Must be set to prometheus\_remote\_write ## sources.\.address **required** **string** The socket address on which worker listens for incoming Prometheus remote write requests. - Must include a port number - Supports binding to all interfaces or specific addresses Example format: :::BlockQuote 0.0.0.0:9090 ::: ## Authentication Configuration Controls access to the remote write endpoint. :::BlockQuote Authentication should always be used together with HTTPS. ::: ## sources.\.auth **optional** **object** Defines the authentication strategy for incoming HTTP requests. ### sources.\.auth.strategy **required** **string** **enum** Selects the authentication mechanism. Supported values: - basic – HTTP Basic Authentication - custom – Custom authentication using VRL logic ### sources.\.auth.username **required** **string** The username used for Basic Authentication. **Relevant when:** strategy = "basic" ### sources.\.auth.password **required** **string** The password used for Basic Authentication. **Relevant when:** strategy = "basic" ### sources.\.auth.source **required** **string** A VRL boolean expression used to validate incoming requests. - Receives the HTTP request context - Must evaluate to true for the request to be accepted **Relevant when:** strategy = "custom" ## HTTP Keepalive Configuration Controls connection lifecycle behavior for incoming HTTP clients. ## sources.\.keepalive **optional** **object** Configures HTTP keepalive behavior. ### sources.\.keepalive.max\_connection\_age\_secs **optional** **uint** Maximum lifetime of an HTTP connection before worker closes it. - A Connection: close header is sent when the limit is reached - Applies to HTTP/0.9, HTTP/1.0, and HTTP/1.1 only - Can be set to a very large value to effectively disable rotation **Default** :::BlockQuote 300 ::: ### sources.\.keepalive.max\_connection\_age\_jitter\_factor **optional** **float** Adds random jitter to the maximum connection age. - Helps avoid synchronized connection termination - Prevents connection storms Example: - 0.1 → actual lifetime varies between 90% and 110% **Default** :::BlockQuote 0.1 ::: ## Metadata Conflict Handling ## sources.\.metadata\_conflict\_strategy **optional** **string** **enum** Defines how conflicting metric metadata is handled. Supported values: - ignore Silently keeps the first metadata entry Matches Prometheus and Thanos behavior - reject Rejects the request with HTTP 400 Preserves strict metadata correctness **Default** :::BlockQuote reject ::: ## Request Path Configuration ## sources.\.path **optional** **string** Specifies the HTTP path used to receive remote write requests. - Useful when exposing multiple services on the same address - Must match Prometheus remote\_write configuration **Default** :::BlockQuote / ::: ## Sample Validation ## sources.\.skip\_nan\_values **optional** **bool** Controls handling of samples with NaN values. - When enabled, samples with NaN values are discarded - Prevents invalid metrics from entering downstream systems **Default** :::BlockQuote false ::: ## TLS Configuration Secures communication between Prometheus and worker. ## sources.\.tls **optional** **object** Defines TLS behavior for incoming connections. ### sources.\.tls.enabled **optional** **bool** Enables TLS encryption. - When enabled, a server certificate is required ### sources.\.tls.crt\_file **optional** **string** Path to the server certificate file. - PEM, DER, or PKCS#12 formats supported - Required when TLS is enabled ### sources.\.tls.key\_file **optional** **string** Path to the private key file. - PEM or DER format - Required unless using PKCS#12 ### sources.\.tls.key\_pass **optional** **string** Passphrase for encrypted private keys. ### sources.\.tls.ca\_file **optional** **string** Additional CA certificate bundle. - Used for client certificate verification ### sources.\.tls.verify\_certificate **optional** **bool** Enables certificate chain validation. - Ensures certificates are trusted and not expired - Strongly recommended for production ### sources.\.tls.verify\_hostname **optional** **bool** Enables hostname verification. - Ensures the certificate matches the expected hostname ### sources.\.tls.alpn\_protocols **optional** **array\[string]** Defines supported ALPN protocols in priority order. ## Reliability Characteristics - At-least-once delivery semantics - Batch-oriented processing - Backpressure-aware ingestion - Prometheus-compatible retry behavior - Stateless processing model ## Common Use Cases - Replacing Prometheus long-term storage - Centralizing metrics ingestion - Multi-cluster Prometheus federation - Secure metric ingestion endpoints - Unified logs–metrics–traces pipelines