Advanced Log Monitoring

• Introduction
• Requirements
• Compatibility matrix
• Configuration parameters
• Plugin configuration

Introduction

This document describes generic log monitoring based on the advanced log monitoring plugin,
which is different from the OpenSource plugin. 

Requirements

The plugin has the requirements to function correctly:

• Have access to the file containing the required configuration requirements.
• Access to the temporary directory; the plugin may create temporary files. Default: /tmp in Linux or %TEMP% / C:\Windows\Temp in Windows.
• You can read the files to be processed with the user running Pandora, or call a script that in turn calls the plugin with all the parameters, so that it can read the log completely. If you use an external script, it will need to have permissions for the plugin to generate its index files.

Compatibility matrix

Linux

Systems where it has been tested	Rocky 9
Systems where it should work	Any Linux system supported by PandoraFMS

Windows

Systems where it has been tested	Windows Server 2022
Systems where it should work	Any Windows system supported by PandoraFMS

Configuration parameters

The plugin is configured using an external configuration file. This configuration file has a series of “general” parameters, a series of parameters specific to each record, and a set of parameters specific to each regular expression block.

General parameters

• include
  Calls another configuration file. It can be nested without limit, and its loading order is sequential.
  It is important to call files with absolute paths.
• index_dir
  Use this directory to store index files. The plugin must be able to write to and read from the directory.
  
• logfile
  Plugin log file.
  

Specific log parameters

• log_begin y log_end
  Set start and end marks for the definition of a logparser.log file
  
• log_module_name
  Name of the module generated by the plugin.
  
• log_type
  There are three types of log module:
  - log_module: Returns the log lines in a log-type module.
  - return_occurrences: Returns a numerical value with the number of occurrences.
  - return_lines: Returns the lines of the log that match.
  - return_message: Returns a message specified by the configuration file.
  
• log_rotate_mode
  It can be of the inode or md5 type. This is the type of detection performed to determine whether a record is rotated or not.
  
• log_force_readall
  When this token is present, the log analyzer processes the entire log from the beginning if it has not already done so
  (it is the first time it has been opened or a rotation has been detected). NOTE: This can generate large volumes of data.
  
• log_location_exec
  Execute the specified command to obtain the name (absolute path) of the file to be processed.
  
• log_location_filename
  Specifies the name of the log file (absolute path) to be processed.
  
• log_location_multiple
  Allows you to specify one or more files using a wildcard, for example: /tmp/apache/file_log* or /tmp/system*/mail*.
  Used in combination with log_create_module_for_each_log, it will create a different module for each log file
  or put all the information from different files into the same module.
  
• log_create_module_for_each_log
  If used with “log_location_multiple,” it will force the plugin to generate a different module for each log file.
  

Definition of regular expressions

• log_regexp_begin y log_regexp_end
  Mark the beginning and end of the definition of a regular expression for the definition of the log file
  in which they are found.
  
• log_regexp_rule
  Define the regular expression. NOTE: Do not use / / markers directly in the extended regular expression
  (Perl-like). Examples:
  

File\sdoes\snot\sexist → Find “File does not exist”
[0-9]*\serrores → Find strings “043 errores”

• log_regexp_severity
  Send a severity in the XML, which can be WARNING, CRITICAL, or NORMAL (in uppercase). This is optional.
  
• log_return_message
  Text that was sent to find at least one occurrence (if several were found, only one message will be sent). You
  can use the switches $1 .. $2 for fields previously identified with a regular expression to perform the search
  field → syntax ()
  
• log_regexp_action
  Command that runs to find at least one occurrence (if it finds several, it runs only once).
  When defining a record, you can define several regular expression blocks. Each regular expression block
  will contain only one regexp rule. In case of multiple matches, it will count each occurrence,
  but will only send one message or execute one action.
  

To understand each element, an example configuration file is shown below.

Linux

# Include, to load extenal/aditional configuration files
# include /tmp/my_other_configuration.conf

# Directory where temporal indexes will be stored (/tmp by default)
#index_dir /tmp

# Log problems with the logparser, (/tmp/pandora_logparser.log by default)
#logfile /tmp/pandora_logparser.log

# Sample of creating a log-type module
log_begin
log_module_name errors file
log_force_readall
log_location_file /var/log/errors.out
log_description get errors
log_type log_module
log_regexp_begin
log_regexp_rule (?i)error
log_regexp_severity CRITICAL
log_return_message Error
log_regexp_end
log_end

# Sample of a single log match
log_begin
log_module_name Weekly
log_location_file /var/log/weekly.out
log_description Errors cannot find
log_type return_lines
log_regexp_begin
log_regexp_rule output
log_regexp_severity WARNING
log_return_message Cannot find process to run
log_regexp_end
log_end

# Sample of wildcard matching of several logfiles within the same module
log_begin
log_rotate_mode md5
log_module_name system_log
log_force_readall
log_location_multiple /var/log/system.log*
log_description Errors cannot find
log_type return_lines
log_regexp_begin
log_regexp_rule Cannot
log_regexp_severity WARNING
log_return_message Cannot find process to run
log_regexp_end
log_end

# Sample of several wildcard matching on the same file
log_begin
log_module_name hits_apache
log_location_file /var/log/apache2
log_description Access log from Apache, we will get the integria access
log_type return_ocurrences
log_regexp_begin
log_regexp_rule Error -($1)\-($2) [0-9a-zA-Z]*
log_regexp_severity WARNING
log_return_message Otro bonito texto de error
log_regexp_end
log_regexp_begin
log_regexp_rule File\sdoes\snot\sexist
log_regexp_severity WARNING
log_regexp_end
log_regexp_begin
log_regexp_rule pandora_backend\.html
log_regexp_severity WARNING
log_return_message Something possible harmful happen
log_regexp_end
log_end

# Sample of wildcard matching of several logfiles with diferent dynamic modules
log_begin
log_rotate_mode inode
log_module_name test_log
log_force_readall
# If enabled, this token will create a different module using the module_name
# provided plus the full logfilename replacing / with " ".
log_create_module_for_each_log
log_location_multiple /tmp/log*/hola*
log_description Errors cannot find
log_type return_lines
log_regexp_begin
log_regexp_rule adios
log_regexp_severity WARNING
log_return_message Cannot find process to run
log_regexp_end
log_end

Windows

# Include external/additional configuration files
# include C:\PandoraFMS\conf\extra.conf

# Temporary index directory
index_dir C:\PandoraFMS\tmp\logparser

# Log file for logparser errors
logfile C:\PandoraFMS\logs\pandora_logparser.log


# Example: log module creation
log_begin
log_module_name errors_file
log_force_readall
log_location_file C:\Logs\errors.log
log_description get_errors
log_type log_module
log_regexp_begin
log_regexp_rule (?i)error
log_regexp_severity CRITICAL
log_return_message Error_detected
log_regexp_end
log_end


# Example: single log file matching
log_begin
log_module_name weekly_log
log_location_file C:\Logs\weekly.log
log_description weekly_errors
log_type return_lines
log_regexp_begin
log_regexp_rule output
log_regexp_severity WARNING
log_return_message Cannot_find_process_to_run
log_regexp_end
log_end


# Example: multiple log files in same module
log_begin
log_rotate_mode md5
log_module_name system_log
log_force_readall
log_location_multiple C:\Logs\system.log*
log_description system_errors
log_type return_lines
log_regexp_begin
log_regexp_rule Cannot
log_regexp_severity WARNING
log_return_message Cannot_find_process_to_run
log_regexp_end
log_end


# Example: multiple patterns in one log file
log_begin
log_module_name apache_access
log_location_file C:\Logs\apache\access.log
log_description apache_access_log
log_type return_occurrences

log_regexp_begin
log_regexp_rule Error\-([0-9]+)
log_regexp_severity WARNING
log_return_message Apache_error_detected
log_regexp_end

log_regexp_begin
log_regexp_rule File\sdoes\snot\sexist
log_regexp_severity WARNING
log_regexp_end

log_regexp_begin
log_regexp_rule pandora_backend\.html
log_regexp_severity WARNING
log_return_message Potential_issue_detected
log_regexp_end

log_end


# Example: dynamic modules for multiple files
log_begin
log_rotate_mode inode
log_module_name test_log_dynamic
log_force_readall

log_create_module_for_each_log
log_location_multiple C:\Logs\app*.log
log_description application_logs
log_type return_lines

log_regexp_begin
log_regexp_rule adios
log_regexp_severity WARNING
log_return_message Match_found
log_regexp_end

log_end

Plugin configuration

Copy the plugins to the agent's plugins directory, distribute them through file collections, or copy them to the Pandora agent folder. Do the same with any additional files that are needed. The plugin execution command within the agent configuration will be similar to this, but using the paths where the plugin and configuration file would be located.

Linux Execution example:

module_plugin /var/opt/PandoraFMS/etc/pandora/plugins/pandora_logparser /var/opt/PandoraFMS/etc/pandora/collections/fc_23/log_example.conf

In Windows it would be the same, but adapted to the paths on your machine:

module_plugin C:\Program Files\PandoraFMS\Pandora Agent\plugins\pandora_logparser.exe C:\Program Files\PandoraFMS\Pandora Agent\collections\fc_23\log_example.conf