Log collection and monitoring

Log monitoring in Pandora FMS allows the user to visualize in a single console all the logs from multiple sources to be captured, organizing the information sequentially, using the time stamp in which the logs were processed.

This information does not contain structures or formats, it is stored in text format along with a time stamp (of the time of receipt), in addition to the original time stamps that the files may have had.

These logs can be used to generate security events (SIEM) and/or for troubleshooting, legal compliance or forensic analysis purposes. The log processing capacity is limited only by the capacity of the device used to store them.

• The logs analyzed by the Software Agents (eventlog or text files), are forwarded to the Pandora FMS server, in RAW form inside the XML agent report.
• The Pandora FMS Data Server receives the XML from the agent, which contains both monitoring and log information.
• When the Data Server processes the XML data it identifies the log information, storing in the main database the references of the reporting agent and the origin of the log and then automatically sending the information to OpenSearch.
• Pandora FMS stores the data in OpenSearch indexes generating daily a unique index for each Pandora FMS instance.
• The Pandora FMS server has a maintenance task that deletes the indexes in the interval defined by the system administrator (by default, 30 days, modifiable).
• The logs travel through the network encrypted to avoid formatting problems.
• If you want the logs to travel over the network encrypted, you can use a secure transport (Secure communication Tentacle) for this purpose.
• The logs can be sent by Syslog to Pandora FMS Syslog Server, which directly processes the logs from a local Syslog server, making the processing much faster.
• The load can be distributed using different agents and remote Syslog servers to adopt the best distribution and suitability to the network topology.

To activate the log display system you must activate it, go to Management → Settings → System Settings → Log collector. Click Activate Log Collector and click Update.

The following values must be configured in the OpenSearch options section:

1. OpenSearch IP: IP address of the OpenSearch server to use with Pandora FMS.
2. Use https: Must be enabled if the installed OpenSearch environment has HTTPS enabled for its connection.
3. OpenSearch Port: TCP port number.
4. Days to purge old information: Number of days before deleting the collected data.
5. Basic authentication: (optional) if basic authentication has been installed in OpenSearch (recommended) the user (User field) and password (Password field) must be entered.

Log collecting is carried out by agents, both in Microsoft Windows® agents and in Unix® agents (Linux®, MacOS X®, Solaris®, HPUX®, AIX®, BSD®, etc).

Agents are used to collect local information from each monitored EndPoint.

You can collect events or plain texts (conventional logs).

Similar to the Linux syntax, the module type log is used together with the token module_regexp.

MS Windows does not support module_pattern_exclude.

# Logs extraction
module_begin
module_name Apache_log
module_description Logs extraction module
module_type log
module_regexp C:\server\logs\apache.log
module_source_type apache
module_pattern .*
module_end

The special logchannel module type is used for this purpose, which works only for Microsoft Windows® environments. It allows you to obtain information from the MS Windows® event log based on certain filters, depending on the source and type of event.

General format of this module:

module_begin
module_name MyEvent 
module_type log
module_logchannel 
module_source <logName>
module_eventtype <event_type/level (optional)>
module_eventcode <event_id (optional)>
module_application <source (optional)>
module_pattern <text substring to match (optional)>
module_source_type <siem log type (optional)>
module_end

To avoid displaying repeated information, only those events that have taken place since the last time the Agent was executed are taken into account. In the first execution it will therefore not collect all the events that already exist. It will start sending the collected events from the first execution.

All of the following parameters require the correct input of upper and lower case letters:

• module_source.
• module_source_type.
• module_eventtype.
• module_eventcode.
• module_application.

Source originating the event recorded in the log. It is fundamental to distinguish well from module_application, which indicates the name of the application (another way to obtain events). It is an optional parameter if module_application is used but one of the two must be used.

This is an old categorization that comes from the times of MS Windows NT®. The most known are System, Application, Security and hundreds more. They can be seen in any event, in this case in Spanish under the name “Registro” or “Nombre de registro”.

That is to say, in this screenshot it looks like “Sistema”, but the keys to be able to filter must be expressed in English, “System” in this case. “Application” instead of “Aplicación” and “Security” instead of “Seguridad”. Other long log names should be used as is, System, Application, Security and should always be placed in English to be able to retrieve and display events.

Example of module_source with long name, without translation:

Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Administración

SIEM monitoring relies heavily on the type of log collected, so it will be necessary to specify module_source_type in the log collection modules to indicate that type. The type is used by decoders and rules, so the active decoders and rules should be consulted to know the type to be indicated in each log.

The most commonly used types of logs are:

• syslog.
• ids.
• web-log.
• squid.
• host-information.
• ossec.

The ossec type must be used to process Windows events by the SIEM. For more information, please refer to the SIEM monitoring documentation.

This would be a concrete example for collecting all security events in Windows, which allows processing of specific security events by the SIEM:

# Security Events
module_begin
module_logchannel 
module_name Security_Events 
module_type log
module_source Security
module_source_type ossec
module_end

It is an optional attribute used to filter the type of event (depending on the version of MS Windows®: Error, Warning, Information, Audit success, Audit failure …). It is expressed as a value that can be observed in the Windows event viewer.

This example would take level 3 (warning) safety events:

module_begin
module_logchannel 
module_name Security_Events_Warning 
module_type log
module_source Security
module_event_type 3
module_source_type ossec
module_end

Optional parameter. Allows to use a specific event ID to filter only events with that ID. It is perhaps the most accurate filter.

As you can see in this screenshot, the eventID is referenced in the event viewer:

This example would take level 3 (warning) safety events:

module_begin
module_logchannel 
module_name Security_Events_4798
module_type log
module_source Security
module_eventcodee 4798
module_source_type ossec
module_end

Origin of the event. This is an optional field.

To obtain the name of the event provider, it will be necessary to specify the exact name of the event provider that appears in the name field of the detailed event list. To do this, in the Windows event viewer, after right-clicking on it, select Properties and copy the parameter Full name, required for module_application, as shown in the screenshot below:

These are some common examples of vendor names that appear in the Windows Event Viewer, but there may be many more, as each application will use a specific name.

Operating system providers:

• Microsoft-Windows-Winlogon
• Microsoft-Windows-Security-Auditing
• Microsoft-Windows-User Profiles Service
• Microsoft-Windows-WindowsUpdateClient
• Microsoft-Windows-DNS-Client
• Microsoft-Windows-GroupPolicy
• Microsoft-Windows-TaskScheduler
• Microsoft-Windows-TerminalServices-LocalSessionManager
• Microsoft-Windows-Eventlog
• Microsoft-Windows-WMI
• Microsoft-Windows-Application-Experience
• Microsoft-Windows-PrintService
• Microsoft-Windows-Time-Service

Common application or service providers:

• MSSQLSERVER
• Microsoft Outlook
• Microsoft Office Alerts
• VSS
• Microsoft Exchange Server
• Application Error
• Windows Defender
• Windows Firewall

Specific examples of events relevant to advanced diagnostics:

• Microsoft-Windows-Kernel-General
• Microsoft-Windows-DistributedCOM
• Microsoft-Windows-Power-Troubleshooter
• Microsoft-Windows-Kernel-Boot
• Microsoft-Windows-Resource-Exhaustion-Detector
• Microsoft-Windows-Ntfs
• Microsoft-Windows-Kernel-Processor-Power

Here are some examples of common vendors (not Microsoft):

• Google Chrome
• Mozilla Firefox
• VMware Tools
• Citrix Broker Service
• Oracle Database
• Symantec Endpoint Protection
• McAfee Security
• ESET Security
• Apache Service
• TeamViewer
• Adobe Acrobat
• Backup Exec
• Dropbox Update

These providers serve as filtering criteria to collect or exclude events through the Pandora FMS agent. If it has spaces, quotes should not be used, for example, in the case of “Pandora FMS” it would be:

module_begin
module_name Pandora_Events_Windows
module_type log
module_logchannel
module_application Pandora FMS
module_source Application
module_end

• What is Sysmon?

Sysmon(System Monitor) is a free tool developed by Microsoft as part of the Sysinternals Tools, designed to significantly improve the monitoring and logging of detailed events on Windows operating systems.

Its main purpose is to record activities related to system and application security, which are usually not recorded in sufficient detail in the standard event viewer.

• What is Sysmon for?

Sysmon is primarily focused on monitoring and early detection of security incidents through comprehensive logging of key system events, such as:

1. Creation and execution of processes.
2. Network connections (incoming and outgoing).
3. Windows registry modifications.
4. Loading of DLL libraries.
5. File access and modifications.
6. Suspicious memory and process usage.
7. Code injections in legitimate processes.

These activities are typically exploited by malware or attackers during intrusion attempts or lateral movements within a corporate network.

• How does Sysmon work?

Sysmon works by installing a driver in kernel mode, collecting real-time information about operating system activities. This data is then sent to the Windows Event Viewer (Event Viewer).

• How does Sysmon improve monitoring with Pandora FMS?

Pandora FMS can use the Windows agent to collect these detailed events directly from the Sysmon channel and send them to the Pandora central server.

This combined approach (Sysmon + Pandora FMS + Pandora SIEM) provides great advantages in the monitoring and security of a Windows-based host:

1. Deep security visibility: Provides a level of detail far superior to traditional event logging. Enables comprehensive analysis of suspicious behavior, identifying anomalies and potential threats in real time.
2. Early detection of advanced attacks: Facilitates detection of sophisticated attacks such as lateral movement, execution of malicious scripts or connection to suspicious servers.
3. Immediate response capability: Pandora FMS can generate automatic alerts based on rules defined in Sysmon events (e.g., execution of suspicious processes, log accesses or unusual connection attempts).
4. Historical and correlation: Pandora FMS allows to store, consult and correlate historical events generated by Sysmon during long periods, which is essential in forensic investigations or security audits.
5. Advanced Reporting: Automated extraction enables easy creation of reports for cyber security audits, compliance or post-incident analysis.

• Practical example of integration.

Suppose a Sysmon alert sent through the Pandora agent indicating that:

1. An unknown process has created an outgoing connection to a suspicious IP address.
2. An unknown DLL has been loaded into memory.
3. An attempt was made to modify the Windows registry by an unexpected process.
4. Pandora FMS would receive these detailed events, generate an immediate alert, and allow to act quickly on the incident before significant damage is done.

• Installation and use of Sysmon.

Sysmon is included in the official Microsoft “Sysinternals Suite” and is available for Intel 32-bit and 64-bit systems as well as the new ARM 64-bit architecture (Windows 11).

Sysmon has a fairly extensive configuration file to indicate what it should “monitor”. Monitoring “everything” would affect both the host performance and the system that collects all this information. Pandora FMS agents incorporate a base configuration file that can be improved by the user, by default it will generate a lot of security information.

For more information on how to use Sysmon, check the documentation SIEM.

module_begin
module_name WinEvtLog
module_type log
module_logchannel
module_source Microsoft-Windows-Sysmon/Operational
module_source_type windows
module_end

Unlike other events, it uses module_source_type windows instead of module_source_type ossec. Also, it uses the name WinEvtLog, as defined in the SIEM rules provided with Pandora SIEM. If other names and/or types are used, it will not work as expected.

Log example:

module_begin
module_name Syslog
module_description Sample log collection of syslog messages file
module_type log
module_regexp /var/log/messages
module_pattern .*
module_pattern_exclude DEBUG
module_end

For more information about the description of log type modules, please refer to the following section about Specific Directives.

By defining this type of tag, module_type log, you indicate that not to be stored in the database, but to be sent to the log collector. Any module with this type of data will be sent to the logs collector, as long as it is enabled. Otherwise the information will be discarded.

More complete examples:

module_begin
module_name apache_access
module_type log
module_regexp /var/log/httpd/access_log
module_pattern .*
module_pattern_exclude \s200\s|\s301\s
module_source_type web-log
module_end

module_begin
module_name Audit denied
module_type log
module_regexp /var/log/audit/audit.log
module_pattern denied
module_end

module_begin
module_name Syslog
module_type log
module_regexp /var/log/messages
module_pattern error|fail|panic|segfault|denied|timeout|critical|alert|emergency|memory|core_dumped|failure|attack|bad|illegal|refused|unauthorized|fatal|failed|Segmentation|Corrupted
module_end

module_begin
module_name Secure
module_type log
module_regexp /var/log/secure
module_pattern Failed|failure|invalid|denied|accepted|root
module_end

This component allows Pandora FMS to analyze the syslog of the machine where it is located, analyzing its content and storing the references in the corresponding OpenSearch server.

The main advantage of Syslog Server is to complement log unification. Supported by Syslog Server's export features from Linux® and Unix® environments, Syslog Server allows querying logs regardless of their source, searching in a single common point (Pandora FMS console log viewer).

The installation of Syslog Server 8.2102 must be performed on both the client and the server:

dnf install rsyslog

Access the configuration file /etc/rsyslog.conf to enable TCP and UDP input.

(...)

# Provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# Provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

(...)

Restart the rsyslog service. Once the service is available, verify that port 514 is accessible with:

netstat -ltnp

On the client it is configured so that it can send logs to the Syslog Server, access rsyslog /etc/rsyslog.conf. Locate and enable the line that allows you to configure the remote host (change remote-host to the server's IP address):

action(type="omfwd" Target="remote-host" Port="514" Protocol="tcp")

The size of the logs received by rsyslog is 8 kilobytes by default. If larger logs are received, new entries are added with the remaining content until the complete log is received. These new entries do not contain the name of the host that sent the log, so this behavior can cause both new unwanted log sources and new agents to be created in the console. To avoid this, it is recommended to increase the size of the logs received by adding the following line:

$MaxMessageSize 512k

Save the file and exit the text editor.

To activate this feature in Pandora FMS Server, enable the following content in pandora_server.conf file:

# Enable (1) or disable (0) the Pandora FMS Syslog Server
syslogserver 1

# Full path to syslog's output file.
syslog_file /var/log/messages

# Number of threads for the Syslog Server
syslog_threads 2

# Maximum number of lines queued by the Syslog Server's
syslog_max 65535

On Pandora FMS server, using the token syslog_whitelist, you may admit only logs that match a regular expression or regexp, which is case-sensitive (e.g. windows is not the same as Windows) and discard everything else.

With the token syslog_blacklist you may deny logs that match the set regexp (and let everything else in).

Both tokens are disabled by default.

• syslog_whitelist: By activating this token only logs that comply with the regexp will be accepted and the rest will be discarded.
  • If this token is activated and you have the default filter .*, everything will be accepted.
  • Important: If said token is activated WITHOUT regexp, NOTHING will be admitted.
• Filtering of allowed keywords is done first, this reduces the work for the next step.
• syslog_blacklist: Placing a regexp will discard everything that complies with it (if this token is activated but left WITHOUT regexp, NOTHING will be blocked.).
• Filtering by syslog_blacklist is done last.

In a log collection tool, two features are mainly of interest: being able to search for information -filtering by date, data sources and/or keywords, et cetera- and being able to see that information (menu Operation → Logs → Log viewer) drawn in occurrences per time unit.

The most important -and useful- field will be the search string to be entered in the Search text box in combination with the three available search types (Search mode):

• Exact match: Literal string search, the log contains an exact match.
• All words: Search that contains all the indicated words, regardless of the order in the same log line.
• Any word: Search that contains any of the indicated words, regardless of the order.
• If you check the option to see the context of the filtered content, you will get an overview of the situation with information from other log lines related to the search.

With this feature you may display log entries graphically, classifying the information based on data capture models.

These data capture models are basically regular expressions and identifiers that allow you to parse data sources and display them as a graph.

To access advanced options click on Advanced options. A form will be displayed where you may choose the type of results view:

• Show log entries (plain text).
• Show log graph.
• Using the display log graph option (Display mode) you can select the capture model (Use capture model).
• The default model, Apache log model, offers the possibility of processing or parsing Apache logs in standard format (access_log), being able to retrieve comparative graphs of response time, grouping by page visited and response code:
• You may either click edit or create to make a new capture model.

This option allows you to save frequently used filtering preferences, thus creating a list of filters. Once all filter values have been set up, click Save filter and assign a name, which will allow you to click Save and thus save your preferences or changes (can be saved to an existing filter).

At any other time, these preferences may be loaded by means of Load filter to drop down the list of saved filters. Select one of them and click Load filter.

By using the favorites system in PFMS, a shortcut to the Log viewer with filtering preferences may be saved by clicking on the star icon in the section title.

Starting with version 749 of Pandora FMS, a box called Log sources status has been added to the Agent View, in which the date of the last update of the logs by that agent will appear. When you click on the Review magnifying glass icon, you will be redirected to the Log Viewer view filtered by that log.

Return to Pandora FMS documentation index