---
title: File
slug: telemetry-pipeline/file
docTags: 
createdAt: 2025-04-20T16:43:32.649Z
---

## 🗂️ Directory & File Handling

### Data\_dir (optional, string)

Directory for storing checkpoint positions.

If not set, the global data\_dir is used. Worker will attempt to create this directory if needed.

Ensure write permissions for the running user.



### Include (required, \[string])

List of file patterns to include. Supports globbing.



### exclude (optional, \[string])

List of file patterns to exclude, applied after include.

Helps filter out inaccessible files after include globbing.



### file\_key (optional, string)

Overrides the event field name containing the file path.

Set to "" to disable. Default: "file".



### Offset\_key (optional, string)

Adds the byte offset of each line to the event under the given key.

Only included if explicitly set.



### Remove\_after\_secs (optional, uint)

Time to wait after EOF before removing the file.

Requires permission to delete files.



### Rotate\_wait\_secs (optional, uint)

Time to keep a handle on a rotated log file.

 Default: unlimited (\~292 billion years).



### oldest\_first (optional, bool)

Prioritize reading older files first instead of balancing evenly.

 Default: false.



### Ignore\_checkpoints (optional, bool)

Start reading from scratch but still writes checkpoints.



### Ignore\_older\_secs (optional, uint)

Skip files modified longer than this number of seconds ago.



### Ignore\_not\_found (optional, bool)

Ignore missing files (e.g., broken symlinks).

Default: false.



### read\_from (optional, enum)

Starting position for new files.

Options: beginning (default), end.



## 🔎 File Identification (Fingerprinting)

- fingerprint.strategy (optional, string)

Strategy for uniquely identifying files.

Options:

- checksum (default): Read lines and compute checksum.

device\_and\_inode: Use device & inode.

- fingerprint.lines (optional, uint)

Number of lines to use for checksum (if checksum strategy).

 Default: 1.

- fingerprint.ignored\_header\_bytes (optional, uint)

Bytes to skip when generating checksum.

Useful for skipping common file headers. Only works with gzip.



## 🧠 Encoding & Decoding

encoding.charset (required inside encoding, string)

Character set of source messages (e.g., utf-16le, iso-8859-1).

Transcodes to UTF-8. Malformed sequences are replaced with U+FFFD.



## 📊 Performance & Metrics

- glob\_minimum\_cooldown\_ms (optional, uint)

Minimum interval between file discovery checks.

 Default: 1000 (ms).

- max\_read\_bytes (optional, uint)

Max bytes to read from a file before rotating to the next.

 Not used if oldest\_first is true. 

Default: 2048.

- max\_line\_bytes (optional, uint)

Discards lines exceeding this size.

Default: 102400.

- line\_delimiter (optional, string)

Custom delimiter for line separation.



## 🧩 Multiline Log Aggregation

multiline (optional, object)

Enables aggregation of multiline logs. Includes:

- condition\_pattern (required, string): Regex to match continuation lines.
- start\_pattern (required, string): Regex to match the start of new messages.
- mode (required, enum): Aggregation behavior. Options:
- continue\_past: Include lines matching pattern + one after.
- continue\_through: Include all matching lines after the start.
- halt\_before: Include until line matches (excludes match).
- halt\_with: Include up to and including matching line.
- timeout\_ms (required, uint): Max wait time for continuation lines before flushing.





![](https://api.archbee.com/api/optimize/g9cApVza9NIhSh0pjZYtA/UvT2QLTkpXYWDBhWYU-fR_image.png)