GCP Stack Driver
GCP Operations Logs Sink
Overview
The GCP Operations Logs sink delivers log events to Google Cloud Operations (formerly Stackdriver Logging). It integrates worker with Google Cloud’s centralized logging platform, enabling native ingestion into Cloud Logging for analysis, retention, and correlation with other GCP observability signals.
This sink is typically used when:
- workloads run on Google Cloud infrastructure,
- logs must be stored and analyzed inside GCP for compliance or operational reasons,
- Cloud Logging is used as the primary observability backend.
Supported Input Types
- Logs
Authentication
Authentication Methods
The sink supports multiple authentication mechanisms. One of the following must be available:
- Service account credentials file
- API key
- Application Default Credentials (ADC)
If no explicit credentials are configured:
- the GOOGLE_APPLICATION_CREDENTIALS environment variable is checked,
- otherwise, instance-level service accounts are used when running on Google Compute Engine.
This allows seamless authentication in managed GCP environments without embedding credentials in configuration.
Target Scope Configuration
Exactly one of the following target scopes must be defined. This determines where logs are published within Google Cloud.
Project
Publishes logs to a specific GCP project.
This is the most common configuration and is recommended for:
- application-level logging,
- team-owned workloads,
- isolated billing and access control.
Organization
Publishes logs at the organization level.
Typically used for:
- centralized logging across multiple projects,
- organization-wide compliance and audit pipelines.
Folder
Publishes logs to a specific GCP folder.
Useful for:
- grouping related projects,
- enforcing logging policies at an intermediate hierarchy level.
Billing Account
Publishes logs associated with a billing account.
This mode is generally used for:
- billing-related telemetry,
- organization-wide financial observability.
Log Identification
Log ID (required)
Defines the logical log stream name inside Cloud Logging.
Key characteristics:
- Used to group related log entries.
- Appears as the log name in the Cloud Logging UI and APIs.
- Supports dynamic values to route different event types into separate log streams.
Resource Mapping
Monitored Resource (required)
Specifies the GCP monitored resource associated with log entries.
Examples include:
- Compute Engine instances,
- Kubernetes workloads,
- managed services.
Each resource:
- has a required type,
- includes type-specific labels such as instance ID, zone, or cluster name.
Correct resource mapping is critical for:
- log-to-metric correlation,
- native GCP dashboards,
- infrastructure-aware filtering.
Metadata Enrichment
Labels
Custom key–value labels can be attached to log entries.
Labels are used for:
- indexing and filtering,
- cost attribution,
- environment or tenant tagging.
Labels may be:
- statically defined,
- dynamically extracted from event fields.
Severity Mapping
Defines how log severity is derived from incoming events.
Behavior:
- a specified event field is mapped to GCP’s severity levels,
- numeric or textual severity values are supported,
- if unset, logs default to DEFAULT severity.
This enables consistent severity handling across heterogeneous log sources.
Delivery Shaping
Batch Behavior
Controls how log events are grouped before being sent to GCP.
Batching affects:
- API request rate,
- ingestion latency,
- quota utilization.
Default settings balance throughput and responsiveness for most workloads.
Buffering Behavior
Defines how logs are buffered locally when delivery is temporarily constrained.
Supported buffering strategies:
- Memory buffering for higher performance,
- Disk buffering for improved durability across restarts.
When buffers are full:
- backpressure can be applied upstream,
- or newer events can be dropped if continuity is prioritized.
Payload Control
Encoding Controls
Field-level transformations can be applied prior to ingestion:
- include only selected fields,
- exclude noisy or unnecessary fields,
- normalize timestamps to supported formats.
These controls help:
- reduce ingestion volume,
- improve query efficiency,
- align logs with Cloud Logging schema expectations.
Network and Transport Controls
Proxy Support
Supports HTTP and HTTPS proxies with optional exclusion rules.
Used in environments with:
- restricted outbound access,
- centralized egress gateways,
- regulated network paths.
Request Behavior
Advanced request controls allow fine-tuning delivery behavior:
- adaptive or fixed concurrency,
- rate limiting aligned with GCP quotas,
- retry strategies with backoff,
- request timeouts.
These settings are typically adjusted in:
- high-throughput logging environments,
- multi-sink deployments,
- quota-constrained projects.
TLS Configuration
Supports secure communication using TLS, including:
- custom certificate authorities,
- certificate and key configuration,
- strict hostname and certificate verification.
TLS should always be enabled when logs traverse untrusted networks.