---
title: GCP PubSub
slug: telemetry-pipeline/gcp-pubsub
docTags: 
createdAt: 2025-04-21T00:51:54.515Z
---

## GCP Pub/Sub Source

The GCP Pub/Sub source retrieves observability events from **Google Cloud Pub/Sub** subscriptions. It is designed to consume log, metric, and trace data published to Pub/Sub topics and forward them into centralized observability pipelines.

This source operates in an **aggregator** role and provides **at-least-once delivery guarantees** by leveraging Pub/Sub acknowledgement semantics.

## Requirements

This source requires:

- An existing **GCP project** 
- A **Pub/Sub subscription** associated with a topic that publishes observability data 
- Appropriate IAM permissions to pull and acknowledge messages 

The subscription must be reachable using either:

- A service account 
- An API key 
- Or workload / instance identity when running on GCP infrastructure 

## Authentication and Credentials

Authentication to GCP Pub/Sub can be configured using one of the following methods:

- **Service account credentials file** 
- **API key** 
- **Implicit credentials** via the GOOGLE\_APPLICATION\_CREDENTIALS environment variable 
- **Compute instance service account** when running on GCE, GKE, or other supported GCP environments 

If multiple authentication mechanisms are present, explicit configuration takes precedence over environment-based discovery.

## Acknowledgement Handling

### Acknowledgement Deadline

The acknowledgement deadline defines how long Pub/Sub waits for a message to be acknowledged before considering it eligible for redelivery.

- **ack\_deadline\_secs**
   Specifies the acknowledgement deadline in seconds.
   Messages not acknowledged within this period may be retransmitted. 

The default value is **600 seconds**, which is suitable for most log ingestion pipelines where downstream processing may involve buffering or batching.

:::BlockQuote
The legacy field ack\_deadline\_seconds is deprecated and should not be used.
:::

### Source-Level Acknowledgements (Deprecated)

The following settings are deprecated and no longer affect runtime behavior:

- **acknowledgements** 
- **acknowledgements.enabled** 

Acknowledgement control is handled globally or at the sink level.
 Enabling or disabling acknowledgements at the source level has **no effect**.

## Decoding Configuration (decoding)

Controls how raw Pub/Sub message payloads are converted into events.
 Some decoders are capable of emitting **logs, metrics, or traces**, depending on the payload format.

### Supported Codecs

- bytes 
- json 
- avro 
- gelf 
- influxdb 
- syslog 
- protobuf 
- otlp 
- native 
- native\_json 
- vrl 

If no codec is specified, the payload is treated as raw bytes.

### Multi-Signal Decoding

Certain codecs (such as otlp, native, and native\_json) can automatically detect and emit different signal types:

- Logs 
- Metrics 
- Traces 

When applicable, signal parsing order can be controlled to optimize performance and avoid unnecessary decoding attempts.

### Codec-Specific Considerations

- **Lossy decoding** options allow invalid UTF-8 sequences to be replaced instead of causing failures. 
- **Avro decoding** requires a schema definition and does not support all Apache Avro logical types. 
- **Protobuf decoding** may require a descriptor set and explicit message type selection. 
- **VRL decoding** executes a Vector Remap Language program to produce the final event. 

## Framing Configuration (framing)

Framing determines how individual events are separated within a byte stream before decoding.

This is primarily relevant when message payloads contain multiple logical events.

### Supported Framing Methods

- bytes 
- newline\_delimited 
- character\_delimited 
- length\_delimited 
- varint\_length\_delimited 
- octet\_counting 
- chunked\_gelf 

Each framing strategy includes optional safety limits to prevent unbounded memory usage when processing malformed or untrusted input.

## Pub/Sub Streaming Behavior

### Concurrency and Throughput

The source maintains multiple concurrent streaming pull connections to Pub/Sub to maximize throughput.

Key behaviors include:

- Dynamically opening new streams when existing streams are busy 
- Periodically polling stream utilization 
- Backing off and retrying on transient errors 

### Flow Control and Keepalive

- **keepalive** settings ensure long-lived streaming connections remain active 
- **full response size thresholds** are used to detect subscription pressure and scale stream concurrency accordingly 

These mechanisms allow the source to efficiently handle high-volume subscriptions without overwhelming the downstream pipeline.

## Network and Endpoint Configuration

By default, the source connects to the standard GCP Pub/Sub API endpoint.
 Custom endpoints may be specified for:

- Private Service Connect 
- Regional endpoints 
- Testing or proxy-based deployments 

## Proxy Support (proxy)

Outbound connections to GCP APIs can be routed through HTTP or HTTPS proxies.

Proxy configuration supports:

- Separate HTTP and HTTPS proxy endpoints 
- Exclusion lists (no\_proxy) for specific hosts, IPs, or CIDR ranges 

This is particularly useful in **restricted enterprise networks** or **on-prem deployments** with controlled egress.

## TLS Configuration (tls)

TLS settings apply to outbound connections to the Pub/Sub API.

Supported options include:

- Custom CA certificates 
- Client certificates and private keys 
- ALPN protocol configuration 
- Certificate and hostname verification controls 

:::BlockQuote
Disabling certificate or hostname verification is strongly discouraged unless the security risks are fully understood.
:::