GCP PubSub
GCP Pub/Sub Source
The GCP Pub/Sub source retrieves observability events from Google Cloud Pub/Sub subscriptions. It is designed to consume log, metric, and trace data published to Pub/Sub topics and forward them into centralized observability pipelines.
This source operates in an aggregator role and provides at-least-once delivery guarantees by leveraging Pub/Sub acknowledgement semantics.
Requirements
This source requires:
- An existing GCP project
- A Pub/Sub subscription associated with a topic that publishes observability data
- Appropriate IAM permissions to pull and acknowledge messages
The subscription must be reachable using either:
- A service account
- An API key
- Or workload / instance identity when running on GCP infrastructure
Authentication and Credentials
Authentication to GCP Pub/Sub can be configured using one of the following methods:
- Service account credentials file
- API key
- Implicit credentials via the GOOGLE_APPLICATION_CREDENTIALS environment variable
- Compute instance service account when running on GCE, GKE, or other supported GCP environments
If multiple authentication mechanisms are present, explicit configuration takes precedence over environment-based discovery.
Acknowledgement Handling
Acknowledgement Deadline
The acknowledgement deadline defines how long Pub/Sub waits for a message to be acknowledged before considering it eligible for redelivery.
- ack_deadline_secs Specifies the acknowledgement deadline in seconds. Messages not acknowledged within this period may be retransmitted.
The default value is 600 seconds, which is suitable for most log ingestion pipelines where downstream processing may involve buffering or batching.
The legacy field ack_deadline_seconds is deprecated and should not be used.
Source-Level Acknowledgements (Deprecated)
The following settings are deprecated and no longer affect runtime behavior:
- acknowledgements
- acknowledgements.enabled
Acknowledgement control is handled globally or at the sink level. Enabling or disabling acknowledgements at the source level has no effect.
Decoding Configuration (decoding)
Controls how raw Pub/Sub message payloads are converted into events. Some decoders are capable of emitting logs, metrics, or traces, depending on the payload format.
Supported Codecs
- bytes
- json
- avro
- gelf
- influxdb
- syslog
- protobuf
- otlp
- native
- native_json
- vrl
If no codec is specified, the payload is treated as raw bytes.
Multi-Signal Decoding
Certain codecs (such as otlp, native, and native_json) can automatically detect and emit different signal types:
- Logs
- Metrics
- Traces
When applicable, signal parsing order can be controlled to optimize performance and avoid unnecessary decoding attempts.
Codec-Specific Considerations
- Lossy decoding options allow invalid UTF-8 sequences to be replaced instead of causing failures.
- Avro decoding requires a schema definition and does not support all Apache Avro logical types.
- Protobuf decoding may require a descriptor set and explicit message type selection.
- VRL decoding executes a Vector Remap Language program to produce the final event.
Framing Configuration (framing)
Framing determines how individual events are separated within a byte stream before decoding.
This is primarily relevant when message payloads contain multiple logical events.
Supported Framing Methods
- bytes
- newline_delimited
- character_delimited
- length_delimited
- varint_length_delimited
- octet_counting
- chunked_gelf
Each framing strategy includes optional safety limits to prevent unbounded memory usage when processing malformed or untrusted input.
Pub/Sub Streaming Behavior
Concurrency and Throughput
The source maintains multiple concurrent streaming pull connections to Pub/Sub to maximize throughput.
Key behaviors include:
- Dynamically opening new streams when existing streams are busy
- Periodically polling stream utilization
- Backing off and retrying on transient errors
Flow Control and Keepalive
- keepalive settings ensure long-lived streaming connections remain active
- full response size thresholds are used to detect subscription pressure and scale stream concurrency accordingly
These mechanisms allow the source to efficiently handle high-volume subscriptions without overwhelming the downstream pipeline.
Network and Endpoint Configuration
By default, the source connects to the standard GCP Pub/Sub API endpoint. Custom endpoints may be specified for:
- Private Service Connect
- Regional endpoints
- Testing or proxy-based deployments
Proxy Support (proxy)
Outbound connections to GCP APIs can be routed through HTTP or HTTPS proxies.
Proxy configuration supports:
- Separate HTTP and HTTPS proxy endpoints
- Exclusion lists (no_proxy) for specific hosts, IPs, or CIDR ranges
This is particularly useful in restricted enterprise networks or on-prem deployments with controlled egress.
TLS Configuration (tls)
TLS settings apply to outbound connections to the Pub/Sub API.
Supported options include:
- Custom CA certificates
- Client certificates and private keys
- ALPN protocol configuration
- Certificate and hostname verification controls
Disabling certificate or hostname verification is strongly discouraged unless the security risks are fully understood.