File
🗂️ Directory & File Handling
Data_dir (optional, string)
Directory for storing checkpoint positions.
If not set, the global data_dir is used. Worker will attempt to create this directory if needed.
Ensure write permissions for the running user.
Include (required, [string])
List of file patterns to include. Supports globbing.
exclude (optional, [string])
List of file patterns to exclude, applied after include.
Helps filter out inaccessible files after include globbing.
file_key (optional, string)
Overrides the event field name containing the file path.
Set to "" to disable. Default: "file".
Offset_key (optional, string)
Adds the byte offset of each line to the event under the given key.
Only included if explicitly set.
Remove_after_secs (optional, uint)
Time to wait after EOF before removing the file.
Requires permission to delete files.
Rotate_wait_secs (optional, uint)
Time to keep a handle on a rotated log file.
Default: unlimited (~292 billion years).
oldest_first (optional, bool)
Prioritize reading older files first instead of balancing evenly.
Default: false.
Ignore_checkpoints (optional, bool)
Start reading from scratch but still writes checkpoints.
Ignore_older_secs (optional, uint)
Skip files modified longer than this number of seconds ago.
Ignore_not_found (optional, bool)
Ignore missing files (e.g., broken symlinks).
Default: false.
read_from (optional, enum)
Starting position for new files.
Options: beginning (default), end.
🔎 File Identification (Fingerprinting)
- fingerprint.strategy (optional, string)
Strategy for uniquely identifying files.
Options:
- checksum (default): Read lines and compute checksum.
device_and_inode: Use device & inode.
- fingerprint.lines (optional, uint)
Number of lines to use for checksum (if checksum strategy).
Default: 1.
- fingerprint.ignored_header_bytes (optional, uint)
Bytes to skip when generating checksum.
Useful for skipping common file headers. Only works with gzip.
🧠 Encoding & Decoding
encoding.charset (required inside encoding, string)
Character set of source messages (e.g., utf-16le, iso-8859-1).
Transcodes to UTF-8. Malformed sequences are replaced with U+FFFD.
📊 Performance & Metrics
- glob_minimum_cooldown_ms (optional, uint)
Minimum interval between file discovery checks.
Default: 1000 (ms).
- max_read_bytes (optional, uint)
Max bytes to read from a file before rotating to the next.
Not used if oldest_first is true.
Default: 2048.
- max_line_bytes (optional, uint)
Discards lines exceeding this size.
Default: 102400.
- line_delimiter (optional, string)
Custom delimiter for line separation.
🧩 Multiline Log Aggregation
multiline (optional, object)
Enables aggregation of multiline logs. Includes:
- condition_pattern (required, string): Regex to match continuation lines.
- start_pattern (required, string): Regex to match the start of new messages.
- mode (required, enum): Aggregation behavior. Options:
- continue_past: Include lines matching pattern + one after.
- continue_through: Include all matching lines after the start.
- halt_before: Include until line matches (excludes match).
- halt_with: Include up to and including matching line.
- timeout_ms (required, uint): Max wait time for continuation lines before flushing.
