Logparser

• Introduction
• Requirements
• Compatibility Matrix
• Configuration Parameters
• Plugin configuration
• Usage examples
• How it works

Introduction

This document describes the configuration and usage of the Advanced Log Parser plugin for log monitoring in Pandora FMS. The plugin scans a directory for log files, applies regular expression filters on file names and line content, and generates log-type modules with the results encoded in base64.

Unlike other log monitoring plugins, Advanced Log Parser uses an incremental indexing system: it only processes new lines that appear in files since the last execution, avoiding re-reads and duplicates.

Requirements

• Read access to the log files in the specified directory.
• Read and write access to the index directory. Default path: /tmp.
• The user running the plugin must have permissions to read the log files and write to the index directory.

Compatibility Matrix

Tested on	Rocky 9
Expected to work on	Any Linux system supported by Pandora FMS

Configuration Parameters

Configuration Parameters

The plugin receives all parameters directly via the command line:

advlogparser [--idx-dir <dir>] <log_dir> [<name_regex>] [<content_regex>] [<source_type>]

General Parameters

Parameter	Description
--idx-dir	Directory where index files are stored. The plugin must be able to read and write in this directory. Default: /tmp.

Log-Specific Parameters

Parameter	Description
log_dir *	Directory containing the log files to process. All files in the directory are listed and filtered using name_regex.
name_regex	Regular expression to filter file names within the directory. Default: .* (all files).
content_regex	Regular expression to filter lines within the log content. Only matching lines are included in the output. Default: .* (all lines).
source_type	Value of the <source> field in the generated log module. Identifies the data source in Pandora FMS. Required if you expect to generate modules.

* Required parameter.

Regular Expression Examples

File name filter (name_regex):

.*\.log              → only files with .log extension
access.*\.log        → files starting with "access" and ending in ".log"
(app|sys)\.log       → files named "app.log" or "sys.log"

Content filter (content_regex):

(?i)error                → lines containing "error" (case-insensitive)
(?i)error|critical|fail  → lines containing error, critical, or fail
^ERROR                   → lines starting with ERROR
[0-9]{3}\s               → lines containing a 3-digit code followed by a space

Plugin configuration

Copy the binary to the Pandora FMS agent plugins directory and configure it as a module_plugin in the agent configuration file.

Configuration example on Linux:

module_begin
module_name LogParser_AppErrors
module_plugin /var/opt/PandoraFMS/etc/pandora/plugins/advlogparser --idx-dir /var/spool/pandora /var/log/myapp '.*\.log' '(?i)error|critical' 'app_source'
module_interval 300
module_end

The agent will run the plugin every module_interval seconds. The first execution will not generate data: the plugin will create indexes pointing to the end of each file. On subsequent executions, the plugin will only process new lines that match the configured filters.

Multiple Modules on the Same Directory

You can define several modules with different filters on the same log directory:

# Module for errors
module_begin
module_name Log_AppErrors
module_plugin /opt/pandora/plugins/advlogparser /var/log/myapp '.*\.log' '(?i)error' 'app_errors'
module_interval 300
module_end

# Module for warnings
module_begin
module_name Log_AppWarnings
module_plugin /opt/pandora/plugins/advlogparser /var/log/myapp '.*\.log' '(?i)warn' 'app_warnings'
module_interval 300
module_end

Usage examples

Capture all new lines from .log files

advlogparser /var/log/myapp '.*\.log'

Filter only lines containing ERROR

advlogparser /var/log/myapp '.*\.log' '(?i)error' 'myapp_errors'

Filter by file name and content, with a custom index directory

advlogparser --idx-dir /var/spool/pandora /opt/app/logs 'access.*\.log' '^50[023]' 'http_errors'

Typical Execution Flow

# First execution: creates indexes, no output
advlogparser /tmp/logs '.*\.log' 'ERROR' 'my_source'

# New lines are written to the log
printf 'INFO: ok\nERROR: disk full\n' >> /tmp/logs/app.log

# Second execution: only captures new lines containing ERROR
advlogparser /tmp/logs '.*\.log' 'ERROR' 'my_source'

Generated output:

<log_module>
  <source><![CDATA[my_source]]></source>
  <data encoding="base64">RVJST1I6IGRpc2NvIGxsZW5v</data>
</log_module>

# Third execution with no changes: no output
advlogparser /tmp/logs '.*\.log' 'ERROR' 'my_source'

How it works

Incremental Reading

The plugin maintains an index file for each processed log file. The index stores the exact byte position where reading stopped in the previous execution. On each new execution, the plugin reads only from that position to the end of the file, processing only new content.

First Execution

When the plugin encounters a log file for the first time, it creates an index pointing to the end of the file. This way, historical log content is not processed, avoiding massive dumps of old data.

Log Rotation Detection

The plugin automatically detects when a log file has been rotated:

• If the file has been renamed and a new one created with the same name, the plugin detects it and starts reading from the beginning of the new file.
• If the file has been truncated (its size is smaller than the saved position), the plugin restarts reading from the beginning.

Orphan Index Cleanup

On each execution, the plugin checks that all stored indexes correspond to log files that still exist. If a log file has been deleted, its associated index is automatically removed, preventing the accumulation of unnecessary indexes.

Output Encoding

Captured lines are concatenated and encoded in base64 before being included in the output XML. This guarantees valid XML regardless of the log content, including special characters, multilingual logs (Japanese, Russian, etc.), or binary content.

Index File Format

Each .idx file contains:

/absolute/path/to/log/file
<byte_position> <inode_number>

• First line: absolute path to the log file (used for orphan cleanup).
• Second line: byte position where reading stopped and internal file identifier (used for rotation detection).