ElasticSearch

The Elasticsearch sink enables worker to index observability events directly into Elasticsearch or compatible services such as OpenSearch. It supports both logs and metrics, and delivers events using the Elasticsearch Bulk API to achieve high throughput and efficient indexing.

The sink operates in a stateless, batch-oriented mode and is suitable for production-scale ingestion pipelines. It supports traditional index-based ingestion as well as Elasticsearch Data Streams, allowing alignment with modern Elastic Common Schema (ECS) and lifecycle-managed storage patterns.

This sink can authenticate using basic authentication or AWS-based authentication mechanisms, making it usable with self-managed Elasticsearch clusters, Elastic Cloud, Amazon OpenSearch Managed domains, and OpenSearch Serverless collections.

This sink supports logs and metrics as input types.

When using Elasticsearch Data Streams, worker must be configured to use the create bulk action. This behavior is required by Elasticsearch and is not enabled by default. If data stream mode is selected, worker automatically adapts timestamp handling to comply with ECS expectations.

Endpoints (optional) Defines one or more Elasticsearch endpoints where events are sent. Each endpoint must include an HTTP or HTTPS scheme and may optionally include credentials. When multiple endpoints are provided, worker distributes requests and manages endpoint health automatically.

Authentication (optional) Controls how worker authenticates with Elasticsearch. Basic authentication is commonly used for self-managed or Elastic Cloud deployments. AWS authentication is required when interacting with Amazon OpenSearch services, including both managed and serverless offerings. AWS authentication supports IAM roles, credential files, instance metadata, and role assumption.

API Version (optional) Specifies the Elasticsearch API version to use. When set to auto, worker attempts to detect the cluster version dynamically. Explicit version selection can be used to ensure compatibility with older clusters or to avoid ambiguity in mixed environments.

Mode (optional) Determines whether events are ingested using traditional index-based bulk ingestion or Elasticsearch Data Streams. In bulk mode, events are written using the index action. In data stream mode, events are written using the create action and routed into data streams that follow the <type>-<dataset>-<namespace> naming convention.

Bulk Configuration (optional) Controls how events are indexed using the Elasticsearch Bulk API. This includes the bulk action type, index naming templates, document versioning behavior, and fallback index selection when templates cannot be resolved. Index names can be dynamically derived using time-based or event-based templates.

Data Stream Configuration (optional) Controls dataset, namespace, and type values used when constructing data stream names. Automatic routing and field synchronization can be enabled to ensure that incoming events match the target data stream structure.

Encoding (optional) Controls how events are transformed before being serialized and sent to Elasticsearch. Encoding options allow selecting which fields are included or excluded, as well as controlling timestamp formats. This is commonly used to reduce payload size, normalize field naming, or align events with ECS or custom schemas.

Document ID Mapping (optional) Allows mapping a specific event field to the Elasticsearch _id field. By default, Elasticsearch generates document IDs automatically. Custom IDs can be useful for idempotency or deduplication, but may reduce indexing performance at high scale.

Buffer (optional) Controls how events are buffered prior to indexing. Memory buffers provide higher throughput but lose data on crash. Disk buffers provide higher durability and can survive restarts after data is flushed to disk. Buffer configuration also determines whether worker blocks upstream components or drops new events when buffers are full.

Batch (optional) Controls how events are grouped into bulk requests. Batch size can be limited by event count, total byte size, or time-based flushing. Proper batch sizing is critical for achieving optimal indexing throughput while avoiding oversized bulk requests that may be rejected by Elasticsearch.

Request Configuration (optional) Controls outbound HTTP behavior, including concurrency, adaptive concurrency control, retries, rate limiting, timeouts, and custom headers. Adaptive concurrency dynamically adjusts request parallelism based on observed latency to maximize throughput while avoiding overload.

Partial Retry Handling (optional) Determines whether bulk requests that partially succeed should be retried. When enabled, this can increase delivery guarantees but may introduce duplicates unless document IDs are explicitly controlled.

Proxy (optional) Allows routing Elasticsearch traffic through HTTP or HTTPS proxies. Proxy configuration supports separate proxies for different protocols and includes bypass rules for specific hosts or address ranges.

TLS (optional) Controls secure transport to Elasticsearch endpoints. TLS configuration supports custom certificate authorities, client certificates for mutual TLS, and hostname verification controls. TLS should be enabled for all production deployments, especially when communicating with remote or cloud-hosted clusters.

The Elasticsearch sink is commonly used as the primary backend for centralized log and metric storage. Index-based ingestion is typically used for legacy clusters or custom index management, while data stream mode is preferred for ECS-aligned workloads with automated lifecycle management. AWS-authenticated configurations are frequently used with Amazon OpenSearch services, while basic authentication is common in Elastic Cloud and self-managed environments.