Log collection and monitoring

Log monitoring in Pandora FMS allows the user to view in a single console all the logs from multiple sources they wish to capture, organizing the information sequentially, using the timestamp when the logs were processed.

This information does not contain structures or formats; it is saved in text format along with a timestamp (from the moment it was received), in addition to the original timestamps these files might have.

These logs can be used to generate security events (SIEM) and/or for troubleshooting, legal compliance or forensic analysis purposes. The log processing capacity is only limited by the capacity of the device used for their storage.

• The logs analyzed by the EndPoints (eventlog or text files), are forwarded to the Pandora FMS server, in RAW form within the agent report XML.
• The Pandora FMS Data Server receives the XML from the EndPoint, which contains both monitoring and log information.
• When the Data Server processes the XML data, it identifies the log information, saving in the main database the references of the reporting EndPoint and the log origin, and then automatically sending the information to OpenSearch.
• Pandora FMS stores the data in OpenSearch indexes, generating a unique index daily for each Pandora FMS instance.
• The Pandora FMS server has a maintenance task that deletes the indexes in the interval defined by the system administrator (by default, 30 days, modifiable).
• Logs travel across the network encoded to avoid format issues.
• If logs are required to travel encrypted across the network, a secure transport (encrypted Tentacle) can be used for this purpose.
• Logs can be sent via Syslog to the Pandora FMS Syslog Server, which directly processes the logs from a local Syslog server, making processing much faster.
• The load can be distributed using different agents (with their EndPoints) and remote Syslog servers to adopt the best distribution and adaptation to the network topology.

The following values must be configured in the OpenSearch options section:

1. OpenSearch IP: IP address of the OpenSearch server to use with Pandora FMS.
2. Use https: Must be activated if the installed OpenSearch environment has HTTPS enabled for its connection.
3. OpenSearch Port: TCP port number.
4. Days to purge old information: Number of days before deleting the collected data.
5. Basic authentication: (optional) if basic authentication has been installed in OpenSearch (recommended) you must enter the set user (User field) and password (Password field).

• Indexing size: This parameter value will be used as the ignore_above configuration in OpenSearch.
• Number of shards: The number of primary shards an index should have. The default value is 1. This setting can only be established at index creation time. It cannot be changed on a closed index.
• Number of replicas: The number of replicas each primary shard has. The default value is 1. WARNING: Setting it to 0 can cause a temporary loss of availability during node restarts or a permanent data loss in case of data corruption.
• Auto expand replicas: Automatically expands the number of replicas based on the number of data nodes in the cluster. Sets a lower and upper limit bounded by dashes (for example, 0-5) or uses “all” for the upper limit (for example, 0-all). Keep in mind that the number of automatically expanded replicas only considers allocation filtering rules, but ignores other allocation rules, such as total shards per node, which can cause the cluster status to turn YELLOW if applicable rules prevent all replicas from being allocated.

Log collection is actually done through the EndPoints, for both Microsoft Windows® and Unix® (Linux®, MacOS X®, Solaris®, HPUX®, AIX®, BSD®, etc.).

Agents are used to host the local information of each monitored EndPoint.

You can collect events or plain texts (conventional logs).

Similar to Linux syntax, the log module type is used along with the module_regexp token.

MS Windows does not support module_pattern_exclude.

# Logs extraction
module_begin
module_name Apache_log
module_description Logs extraction module
module_type log
module_regexp C:\server\logs\apache.log
module_source_type apache
module_pattern .*
module_end

For this, the special module type logchannel is used, which works only for Microsoft Windows® environments. It allows obtaining information from the MS Windows® event log based on certain filters, depending on the source and event type.

General format of this module:

module_begin
module_name MyEvent 
module_type log
module_logchannel 
module_source <logName>
module_eventtype <event_type/level (optional)>
module_eventcode <event_id (optional)>
module_application <source (optional)>
module_pattern <text substring to match (optional)>
module_source_type <siem log type (optional)>
module_end

To avoid displaying repeated information, only MS Windows events that have occurred since the last time the EndPoint was executed are taken into account. Therefore, in the first execution, it will not collect all existing events. It will start sending the collected events from the first execution onwards.

All the parameters indicated below require the correct introduction of uppercase and lowercase letters:

• module_source.
• module_source_type.
• module_eventtype.
• module_eventcode.
• module_application.

Source that originates the event registered in the log. It is essential to distinguish it well from module_application, which indicates the name of the application (another way to obtain events). It is an optional parameter if module_application is used, but one of the two must be used.

This is an old categorization that comes from the MS Windows NT® days. The most well-known are System, Application, Security and hundreds more. They can be seen in any event, in this case in Spanish under the name “Registro” or “Nombre de registro”.

That is, in this screenshot it is seen as “Sistema”, but the keys to be able to filter must be expressed in English, “System” in this case. “Application” instead of “Aplicación” and “Security” instead of “Seguridad”. Other long log names must be used as is, System, Application, Security and they must always be placed in English to be able to obtain and view events.

Example of module_source with a long name, without translation:

Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Administración

SIEM monitoring relies heavily on the type of collected log, so it will be necessary to specify module_source_type in the log collection modules to indicate this type. The type is used by decoders and rules, so you should consult the active decoders and rules to know the type that must be indicated in each log.

The most used log types are:

• syslog.
• ids.
• web-log.
• squid.
• host-information.
• ossec.

The ossec type must be used to process Windows events by the SIEM. For more information, consult the SIEM monitoring documentation.

This would be a concrete example to collect all security events in Windows, which allows the specific security events to be processed by the SIEM:

# Security Events
module_begin
module_logchannel 
module_name Security_Events 
module_type log
module_source Security
module_source_type ossec
module_end

It is an optional attribute used to filter the event type (depending on the MS Windows® version: Error, Warning, Information, Audit success, Audit failure …). It is expressed as a value that can be seen in the Windows event viewer.

This example would take level 3 (warning) security events:

module_begin
module_logchannel 
module_name Security_Events_Warning 
module_type log
module_source Security
module_event_type 3
module_source_type ossec
module_end

Optional parameter. It allows using a specific event ID to filter only events with that ID. It is perhaps the most accurate filter.

As you can see in this screenshot, reference is made to the eventID in the event viewer:

This example would take level 3 (warning) security events:

module_begin
module_logchannel 
module_name Security_Events_4798
module_type log
module_source Security
module_eventcodee 4798
module_source_type ossec
module_end

Event origin. It is an optional field.

To obtain the event provider name, it will be necessary to specify its exact name as it appears in the name field of the detailed event list. To do this, in the MS Windows event viewer, after right-clicking on it, select Properties and copy the Full name parameter, necessary for module_application, as shown in the following screenshot:

These are some common examples of provider names that appear in the MS Windows Event Viewer; there could be many more since each application will use a specific name.

Operating system providers:

• Microsoft-Windows-Winlogon
• Microsoft-Windows-Security-Auditing
• Microsoft-Windows-User Profiles Service
• Microsoft-Windows-WindowsUpdateClient
• Microsoft-Windows-DNS-Client
• Microsoft-Windows-GroupPolicy
• Microsoft-Windows-TaskScheduler
• Microsoft-Windows-TerminalServices-LocalSessionManager
• Microsoft-Windows-Eventlog
• Microsoft-Windows-WMI
• Microsoft-Windows-Application-Experience
• Microsoft-Windows-PrintService
• Microsoft-Windows-Time-Service

Common application or service providers:

• MSSQLSERVER
• Microsoft Outlook
• Microsoft Office Alerts
• VSS
• Microsoft Exchange Server
• Application Error
• Windows Defender
• Windows Firewall

Specific examples of relevant events for advanced diagnostics:

• Microsoft-Windows-Kernel-General
• Microsoft-Windows-DistributedCOM
• Microsoft-Windows-Power-Troubleshooter
• Microsoft-Windows-Kernel-Boot
• Microsoft-Windows-Resource-Exhaustion-Detector
• Microsoft-Windows-Ntfs
• Microsoft-Windows-Kernel-Processor-Power

These are some examples of common providers (non-Microsoft):

• Google Chrome
• Mozilla Firefox
• VMware Tools
• Citrix Broker Service
• Oracle Database
• Symantec Endpoint Protection
• McAfee Security
• ESET Security
• Apache Service
• TeamViewer
• Adobe Acrobat
• Backup Exec
• Dropbox Update

These providers serve as filtering criteria to collect or exclude MS Windows events via the Pandora FMS EndPoint. If it has spaces, quotes must not be used; for example, in the case of “Pandora FMS” it would be:

module_begin
module_name Pandora_Events_Windows
module_type log
module_logchannel
module_application Pandora FMS
module_source Application
module_end

• What is Sysmon?

Sysmon (System Monitor) is a free tool developed by Microsoft® that is part of the Sysinternals Tools, designed to significantly improve the monitoring and logging of detailed events on Windows operating systems.

Its main objective is to log activities related to system and application security, which are not usually logged in sufficient detail in the standard event viewer.

• What is Sysmon for?

Sysmon is primarily focused on monitoring and early detection of security incidents through comprehensive logging of key system events, such as:

1. Process creation and execution.
2. Network connections (incoming and outgoing).
3. Windows registry modifications.
4. DLL library loading.
5. File access and modifications.
6. Suspicious memory and process usage.
7. Code injections into legitimate processes.

These activities are typically exploited by malware or attackers during intrusion attempts or lateral movements within a corporate network.

• How does Sysmon work?

Sysmon works by installing a driver in kernel mode, collecting real-time information about operating system activities. Later, this data is sent to the Windows Event Viewer (Event Viewer).

• How does Sysmon improve monitoring with Pandora FMS?

Pandora FMS can use the MS Windows EndPoint to collect these detailed events directly from the Sysmon channel and send them to the corresponding PFMS Server.

This combined approach (Sysmon + Pandora FMS + Pandora SIEM) provides great advantages in the monitoring and security of a Windows-based host:

1. Deep security visibility: Provides a level of detail far superior to traditional event logging. It allows comprehensive analysis of suspicious behaviors, identifying anomalies and potential threats in real time.
2. Early detection of advanced attacks: Facilitates the detection of sophisticated attacks such as lateral movements, execution of malicious scripts, or connection to suspicious servers.
3. Immediate response capability: Pandora FMS can generate automatic alerts based on defined rules on Sysmon events (for example, execution of suspicious processes, registry accesses, or unusual connection attempts).
4. History and correlation: Pandora FMS allows storing, querying, and correlating historical events generated by Sysmon over long periods, which is essential in forensic investigations or security audits.
5. Advanced reporting: Automated extraction allows the simple creation of reports oriented towards cybersecurity audits, regulatory compliance, or post-incident analysis.

• Practical integration example.

Suppose a Sysmon alert sent through the agent in Pandora FMS indicating that:

1. An unknown process has created an outgoing connection to a suspicious IP address.
2. An unknown DLL has been loaded into memory.
3. An attempt has been made to modify the Windows registry by an unexpected process.
4. Pandora FMS would receive these detailed events, generate an immediate alert, and allow acting quickly on the incident before significant damage occurs.

• Sysmon installation and use.

Sysmon comes included in the official Microsoft “Sysinternals Suite” package and is available for Intel 32 and 64-bit systems as well as the new 64-bit ARM architecture (Windows 11).

Sysmon has a fairly extensive configuration file to indicate what it should “monitor”. Monitoring “everything” would affect both the performance of the host and the system collecting all that information. The Pandora FMS EndPoints incorporate a remote base configuration file that can be improved by the user, since by default it will generate a lot of security information.

For more information on how to use Sysmon, check the SIEM documentation.

module_begin
module_name WinEvtLog
module_type log
module_logchannel
module_source Microsoft-Windows-Sysmon/Operational
module_source_type windows
module_end

Unlike other events, it uses module_source_type windows instead of module_source_type ossec. In addition, it uses the name WinEvtLog, as defined in the SIEM rules provided with PFMS SIEM. If other names and/or types are used, it will not work as expected.

Log example:

module_begin
module_name Syslog
module_description Sample log collection of syslog messages file
module_type log
module_regexp /var/log/messages
module_pattern .*
module_pattern_exclude DEBUG
module_end

For more information about the description of log type modules, you can consult the following section regarding Specific directives.

By defining this type of tag, module_type log, it is indicated that it is not to be stored in the database, but sent to the log collector. Any module with this data type will be sent to the log collector, as long as it is enabled. Otherwise, the information will be discarded.

More complete examples:

module_begin
module_name apache_access
module_type log
module_regexp /var/log/httpd/access_log
module_pattern .*
module_pattern_exclude \s200\s|\s301\s
module_source_type web-log
module_end

module_begin
module_name Audit denied
module_type log
module_regexp /var/log/audit/audit.log
module_pattern denied
module_end

module_begin
module_name Syslog
module_type log
module_regexp /var/log/messages
module_pattern error|fail|panic|segfault|denied|timeout|critical|alert|emergency|memory|core_dumped|failure|attack|bad|illegal|refused|unauthorized|fatal|failed|Segmentation|Corrupted
module_end

module_begin
module_name Secure
module_type log
module_regexp /var/log/secure
module_pattern Failed|failure|invalid|denied|accepted|root
module_end

This component allows Pandora FMS to analyze the syslog of the machine where it is located, analyzing its content and storing the references in the corresponding OpenSearch server.

The main advantage of the Syslog Server is that it complements the unification of logs. Supported by the Syslog Server export features of Linux® and Unix® environments, Syslog Server allows querying logs regardless of the origin, searching in a single common point (log viewer of the Pandora FMS console).

For each host from which logs are received, if a corresponding agent does not already exist in Pandora FMS, this agent will be automatically created to store the logs of that host.

The installation of Syslog Server 8.2102 must be performed on both client and server:

dnf install rsyslog

Access the configuration file /etc/rsyslog.conf to enable the TCP and UDP input.

(...)

# Provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# Provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

(...)

Restart the rsyslog service. Once the service is available, check that port 514 is accessible with:

netstat -ltnp

On the client, configure it so that it can send logs to the Syslog Server; access the rsyslog /etc/rsyslog.conf. Locate and enable the line that allows configuring the remote host (change remote-host to the server's IP address):

action(type="omfwd" Target="remote-host" Port="514" Protocol="tcp")

The size of the logs received by rsyslog is 8 kilobytes by default. If logs with a larger size are received, new entries are added with the remaining content until the complete log is received. These new entries do not contain the name of the host that sent the log, so this behavior can cause both unwanted new log origins and new agents to be created in the web Console. To avoid this, it is recommended to increase the size of the received logs by adding the following line:

$MaxMessageSize 512k

Save the file and exit the text editor.

To activate this functionality in the Pandora FMS Server, enable the following content in the pandora_server.conf file:

# Enable (1) or disable (0) the Pandora FMS Syslog Server
syslogserver 1

# Absolute path to the input file read by the SyslogServer.
syslog_file /var/log/messages

# Number of threads for the Syslog Server
syslog_threads 2

# Maximum number of lines queued by the Syslog Server's
syslog_max 65535

In the Pandora FMS server, through the syslog_whitelist token, you can only allow records that match a regular expression or regexp, which is case-sensitive (for example, windows is not equal to Windows) and discard everything else .

With the syslog_blacklist token, you can deny records that match the established regexp (and let everything else in ).

Both tokens are disabled by default.

• syslog_whitelist: Activating this token will only let in logs that meet the regexp, and the rest are discarded.
  • If this token is activated and it has the default filter .*, everything will be allowed.
  • Important: If this token is activated WITHOUT a regexp, NOTHING will be allowed.
• The filtering of allowed keywords is done first; this reduces the work for the next step.
• syslog_blacklist: By placing a regexp, everything that complies with it will be discarded (if this token is activated but left WITHOUT a regexp, NOTHING will be blocked).
• Filtering by syslog_blacklist is done last.

In a log collection tool, two features are mainly of interest: being able to search for information - filtering by date, data sources and/or keywords, etc. - and being able to display this information (Operation → Logs → Log viewer menu) drawn in occurrences per unit of time.

The most important - and useful - field will be the search string to be entered in the Search text box in combination with the three available search types (Search mode):

• Exact match: Literal string search, the log contains an exact match.
• All words: Search that contains all the indicated words, regardless of the order in the same log line.
• Any word: Search that contains any of the indicated words, regardless of the order.
• If you check the option to see the context of the filtered content, you will get a general view of the situation with information from other log lines related to the search.

With this feature, you can graphically display log entries, classifying the information based on data capture models.

These data capture models are basically regular expressions and identifiers that allow data sources to be analyzed and displayed as a graph.

To access the advanced options click on Advanced options. A form will be displayed where you can choose the type of result view:

• Show log entries (plain text).
• Show log graph.
• Through the show log graph option (Display mode) you can select the capture model (Use capture model).
• The default model, Apache log model, offers the possibility to process or parse Apache logs in standard format (access_log), being able to extract comparative graphs of response time, grouping by visited page and response code:
• You can either click the edit button or the create button to make a new capture model.

Through this option, you can save frequently used filtering preferences, thus creating a list of filters. Once all the filter values have been configured, by clicking the Save filter button and assigning a name you will be able to press the Save button and save the preferences or changes (can be saved in an existing filter).

At any other time, these preferences can be loaded using the Load filter button to drop down the list of saved filters. You must select one of them and click Load filter.

Using the pinned items system in PFMS, you can save a shortcut to the Log viewer with the filtering preferences if you click on the pushpin icon located in the section title.

When the filter is loaded again, the icon will appear and when clicked the filter will be removed from the pinned items.

As of Pandora FMS version 749, a box called Log sources status has been added to the Agent View, where the date of the last log update by that agent will appear. Clicking on the magnifying glass icon of Review redirects to the Log Viewer view filtered by that log.

Return to Pandora FMS documentation index