GCP Cloud Monitoring (formerly Stackdriver)
GCP Cloud Monitoring Sink
Overview
The GCP Cloud Monitoring (formerly Stackdriver) metrics sink delivers metric events to Google Cloud Monitoring (Metrics API). Use it when you want your worker-emitted metrics to appear as Cloud Monitoring time series, so you can build dashboards, alerts, SLOs, and integrate with the broader Cloud Operations suite.
Supported Input Types
- Metrics
Authentication
You can authenticate in one of these ways:
- Service account JSON file via credentials_path
- API key via api_key
- Application Default Credentials (ADC) (implicit auth):
- uses GOOGLE_APPLICATION_CREDENTIALS if set
- otherwise, tries instance metadata credentials when running on GCE/GKE with a service account
If you don’t explicitly set api_key or credentials_path, ADC is the fallback.
Required Configuration
project_id (required)
The GCP project ID that will own the published metrics (time series are written into this project).
resource (required)
Defines the monitored resource that the time series will be associated with.
- resource.type (required): the monitored resource type (e.g., gce_instance, k8s_container, global, etc.)
- resource.* (required): label values required by that resource type (e.g., for gce_instance: projectId, instanceId, zone)
This matters because Cloud Monitoring organizes and queries metrics heavily via resource metadata. If your resource labels don’t match the type’s descriptor, writes will fail.
Namespacing
default_namespace (optional)
Cloud Monitoring can differentiate metrics by namespace. If a metric event does not already include a namespace, worker will apply:
- default_namespace (default: namespace)
This prevents collisions when different metric producers share the same metric name.
Delivery and Reliability Controls
acknowledgements (optional)
Controls end-to-end acknowledgements at the sink level.
- acknowledgements.enabled: when enabled, sources that support E2E acks wait until this sink confirms delivery before acknowledging upstream.
This is important when you want “don’t ack unless it’s safely delivered” semantics across the pipeline.
batch (optional)
Controls how metric events are grouped before sending.
- batch.max_events (default: 1 event)
- batch.timeout_secs (default: 1 second)
Because the default is effectively “send immediately”, you’ll get low latency but potentially higher request volume. Increasing batch sizes can improve throughput and reduce API calls, at the expense of latency.
buffer (optional)
Controls buffering when the sink cannot keep up temporarily.
- buffer.type: memory (default) or disk
- buffer.max_size: required (and if disk, must be >= ~256MB)
- buffer.when_full: block (default) or drop_newest
Guidance:
- memory + block is typical for “don’t drop data, apply backpressure”
- disk helps survive restarts or temporary outages without losing buffered metrics
Request / Networking Controls
proxy (optional)
HTTP(S) proxy support (including no_proxy) for outbound requests to Google APIs.
request (optional)
Advanced outbound behavior:
- request.concurrency: adaptive (default) or fixed / none
- rate limiting (rate_limit_num, rate_limit_duration_secs)
- retries (retry_attempts, backoff, jitter)
- timeouts (timeout_secs)
This is where you tune for:
- quota limits,
- high-volume emission,
- resilience against transient API failures.
TLS
tls (optional)
Standard TLS controls:
- custom CA (ca_file)
- client certs (crt_file, key_file)
- verification toggles (verify_certificate, verify_hostname)
Keep verification enabled unless you have a controlled internal interception setup and fully understand the risk.
Operational Notes
- This sink is beta, so expect occasional behavior changes and be ready to pin/test worker's versions in production.
- Cloud Monitoring enforces constraints on time series writes (resource typing/labels, descriptor compatibility, quotas). If you see rejects, it’s usually resource label mismatch, quota, or metric descriptor conflicts (namespacing helps here).