Dataend

The Databend sink enables worker to deliver log event data into a Databend database table over Databend’s HTTP/DSN-style endpoint. Worker sends events in batches and provides at-least-once delivery behavior, making it suitable for pipelines where Databend is used as a centralized analytic store for logs.

This sink is a good fit when you want to land logs into Databend for SQL-driven analytics, downstream warehousing, or long-term retention while keeping worker as the collection/normalization layer.

This sink supports logs as an input type.

Databend version >= 1.2.216 is required.

Endpoint (required) Defines the Databend DSN that worker uses to connect and write data. This DSN includes the host, port, and default database context, and it can also embed transport/security options such as sslmode.

If you provide credentials inside the DSN, worker can use them directly unless overridden by the auth configuration.

Table (required) Specifies the target Databend table that receives inserted log events. The table must exist and be compatible with the event shape produced by your encoding configuration.

Database (optional) Overrides the database portion of the DSN and selects which database contains the target table. Use this when you want to keep DSN generic but route to a specific database via config.

Auth (optional) Controls how worker authenticates to Databend and overrides any username/password embedded in the DSN.

Databend sink authentication supports multiple strategies:

Basic authentication uses a username and password.

Bearer authentication sends a bearer token (for example OAuth2/JWT) as-is.

Custom authentication lets you provide a complete Authorization header value to be inserted into outbound requests.

AWS authentication signs requests using AWS credentials and is useful in AWS-integrated deployments where Databend sits behind infrastructure requiring AWS-style request signing.

Encoding (optional) Controls how worker serializes each log event before sending it to Databend.

The sink supports JSON and CSV encoding, with JSON as the default. Encoding is where you control:

which fields are included or excluded,

timestamp formatting,

and whether output should be compact or pretty.

If you choose CSV, you must define the field list and ordering, and you can tune quoting, delimiters, and escaping behavior to match your ingestion expectations.

Missing Field Behavior (optional) Controls how missing fields are handled when sending newline-delimited JSON style payloads.

This setting determines whether missing fields are treated as NULL, replaced with defaults, or treated as an error condition. Use stricter modes when you want ingestion to fail fast on schema drift, and more permissive modes when you want ingestion to continue even if some fields are absent.

Compression (optional) Controls whether outbound payloads are compressed. Supported options include gzip and none.

Use compression when bandwidth is constrained or when sending high-volume logs over WAN links. If CPU is more constrained than bandwidth, disabling compression may increase throughput.

Buffer (optional) Configures how events are buffered before being sent to Databend.

Memory buffering is higher performance but loses buffered events on crash. Disk buffering is more durable and can survive restarts after flush-to-disk. You can also control what happens when the buffer is full, choosing between blocking upstream (backpressure) or dropping newest events.

Batch (optional) Controls how events are grouped into batches before being flushed to Databend.

Batch tuning knobs include maximum bytes per batch, maximum events per batch, and maximum time before a batch is flushed. Larger batches generally improve throughput by reducing request overhead, while smaller batches reduce latency and can reduce worst-case duplication during retries.

Proxy (optional) Allows routing Databend traffic through HTTP/HTTPS proxies, including separate proxy endpoints per protocol and no-proxy bypass rules for specific hosts, IPs, or CIDR ranges.

This is useful in enterprise environments where outbound egress must traverse controlled proxy layers.

Request (optional) Controls outbound request behavior such as concurrency, rate limiting, timeouts, and retry strategy.

Adaptive concurrency can be used to automatically tune throughput based on observed latency. Retry behavior follows a backoff policy designed to reduce overload during downstream instability. These settings are where you tune for “fast recovery” vs “minimal duplication” depending on your pipeline priorities.

TLS (deprecated) TLS configuration is deprecated for this sink and should be configured via DSN arguments instead. Use DSN-based TLS controls for certificate verification and transport security in production deployments.

Databend sink is commonly used for SQL-first log analytics and warehousing, where worker performs ingestion, normalization, filtering, and enrichment, and Databend provides the analytical query layer.

Typical patterns include:

Landing normalized JSON logs into a single wide table for ad-hoc SQL exploration.

Using CSV encoding when you want strict column ordering and predictable ingestion mapping.

Enforcing stricter missing-field handling to detect schema drift early in controlled environments.

Using disk buffers when durability through restarts is required.