Internal Metrics
Internal Metrics Source
Access and process metrics produced by worker itself within the same worker pipeline.
This source exposes worker's internal operational metrics, enabling you to collect, transform, and route them like any other metrics signal. It is commonly used for self-observability, health monitoring, and pipeline performance analysis.
Key Characteristics
Status: Stable Roles: Aggregator, Daemon, Sidecar Delivery: At-least-once Acknowledgements: Not supported Egress Mode: Batch State: Stateless Outputs: Metrics
Ingestion Model
- Worker periodically samples its own internal state
- Metrics are generated in-memory
- Collected metrics are emitted into the pipeline as standard metric events
- Metrics can be filtered, enriched, aggregated, or exported using normal worker components
This source enables self-monitoring without requiring an external scraper.
Source Configuration Parameters
sources.<id>.type
required string
Specifies the source type.
- Must be set to internal_metrics
sources.<id>.namespace
optional string
Overrides the default namespace applied to emitted metrics.
- Useful for avoiding naming collisions
- Enables alignment with organizational metric naming standards
sources.<id>.scrape_interval_secs
optional float
Interval, in seconds, at which worker collects and emits its internal metrics.
- Lower values increase metric resolution
- Higher values reduce overhead
This setting directly impacts:
- Metric freshness
- CPU and memory overhead
Default
Tag Configuration
Controls automatic tag enrichment for emitted metrics.
sources.<id>.tags
optional object
Defines tag-related behavior for the internal metrics source.
sources.<id>.tags.host_key
optional string
Overrides the tag name used to attach the peer host address to each metric.
- The value contains the peer address including port (e.g. 1.2.3.4:9000)
- By default, the global log_schema.host_key is used
Special behavior:
- Set to an empty string ("") to suppress this tag entirely
sources.<id>.tags.pid_key
optional string
Defines the tag name used to attach the current process ID (PID) to each metric.
- Useful for debugging multi-process or sidecar deployments
- Not enabled by default
If unset, the PID tag is not added.
Metric Categories
The internal metrics source emits metrics covering:
- Pipeline throughput and latency
- Component-level event counts
- Error and drop counters
- Buffer usage
- Backpressure indicators
- Resource utilization related to worker internals
These metrics reflect worker real-time operational health.
Reliability Characteristics
- Metrics are generated locally
- No external dependencies
- Stateless emission model
- Batch-oriented export
- No acknowledgement semantics
This design prioritizes low overhead and continuous visibility.
Deployment Patterns
The internal metrics source can be used in:
- Central aggregators
- Per-node daemons
- Sidecar deployments
- Edge collectors
It adapts automatically to the deployment role.
Common Use Cases
- Monitoring worker pipeline health
- Detecting backpressure and bottlenecks
- Observing ingestion and delivery rates
- Exporting worker metrics to Prometheus, OTLP, or other backends
- Building dashboards for telemetry infrastructure itself