====== SIEM monitoring ======
{{indexmenu_n>21}}
===== Introduction =====
The acronym **SIEM** stands for Security Information and Event Management. SIEM describes security monitoring features by collecting, filtering, normalizing, correlating and visualizing security events. It combines security information management (**SIM**) with security event management (**SEM**).
SIEM manages security events, which may come from an ordinary log (e.g. Apache logs), from the log of a specific security tool (logs of a firewall, IDS, honeypot, EDR, et cetera) or enriched events from another tool (another SIEM).
===== How it works =====
**Pandora FMS SIEM** is responsible for processing log entries obtained through [[:en:documentation:pandorafms:monitoring:09_log_monitoring|your collection]], normalizing the data and generating security events based on these entries.
As with log collection, the **generated SIEM** events are stored in OpenSearch®, so the first requirement to make this monitoring work is to have an [[:en:documentation:pandorafms:technical_annexes:38_opensearch_installation|OpenSearch installation]].
Pandora FMS SIEM generates security events using two components in the server, so events are also generated in two steps:
* **Data standardization**: All log collection entries are decoded, generating new normalized log entries that are stored temporarily.
* **Event generation**: Normalized logs are checked for compliance with a set of rules, in which case a SIEM event is generated with all the rule information and the normalized log that generated the event.
Thus the complete flow for SIEM monitoring is:
* Agents, plugins and other sources of information monitor or generate logs that are sent to the ''dataserver'' or ''syslogserver''.
* ''dataserver'' and ''syslogserver'' take care of processing these log entries and store them on the OpenSearch server configured for log collection.
* ''siemserver'' decodes all logs obtained through log collection to generate normalized logs on the OpenSearch server configured for SIEM monitoring. These normalized logs are stored temporarily.
* ''siemevents'' processes each of the normalized logs and if they meet a set of predefined rules, SIEM events are generated and stored on the OpenSearch server configured for SIEM monitoring.
With the events generated, it is possible to check them for their operation from Pandora FMS Console.
===== Console configuration =====
To use SIEM event monitoring, it must first be activated from the main configuration.
Go to menu **Management → Settings → System Settings → SIEM → Activate SIEM**, fill in the form completely and click **Update** to save the changes.
This will create the necessary templates on the OpenSearch server specified for log normalization and SIEM event generation.
It will also activate this type of monitoring in Pandora FMS servers where ''siemserver'' and ''siemevents'' were started.
===== Server configuration =====
To perform SIEM event monitoring, activate ''siemserver'' and ''siemevents'' servers in [[:en:documentation:pandorafms:installation:04_configuration#siemserver|server configuration]].
In order to decode and normalize log collection entries, it will be necessary to have decoding XML files that establish the way to obtain the necessary information in each case (hereinafter, "decoders"). These XML files will be located in the path indicated by the ''siem_decoders'' parameter of the server configuration, by default in:
/usr/share/pandora_server/util/siem/decoders
In order to generate SIEM events, it will be necessary to have rules XML files that set the conditions to generate an event based on the information obtained in normalized logs (hereinafter, "rules"). These XML files will be located in the path indicated by the ''siem_events_rules'' parameter of the server configuration, by default in:
/usr/share/pandora_server/util/siem/rules
The XML files of Wazuh® decoders and rules are supported by Pandora FMS SIEM monitoring.
===== Agent configuration =====
Most of the log collection is done through Pandora FMS Software Agents. These agents, both in GNU/Linux® and MS Windows® systems, have specific types of modules to perform this task.
SIEM monitoring relies heavily on the type of log collected, so it will be necessary to specify ''module_source_type'' in log collection modules to indicate that type.
The type is used by decoders and rules, so the active decoders and rules must be checked to know the type to be indicated in each log.
The most commonly used types of logs are:
* syslog
* ids
* web-log
* squid
* windows
* host-information
* ossec
==== Linux agents ====
Log collection on Linux systems is mainly done by reading log files. This can be achieved by using module configurations with this minimal structure:
module_begin
module_name
module_type log
module_regexp
module_pattern
module_source_type
module_end
For example, to collect all access log entries from an Apache server:
module_begin
module_name apache
module_type log
module_regexp /var/log/httpd/access_log
module_pattern .*
module_source_type web-log
module_end
Entries collected from a log like the above would be normalized by decoders such as ''web-accesslog'', ''web-accesslog-ip'' or ''web-accesslog-domain'' among others.
The decoded logs of such a log could generate events such as ''Common\_web\_attack'', ''XSS\_(Cross\_SiteScripting)\_attempt'' or ''SQL\_injection\_attempt'' among others.
==== MS Windows® agents ====
Log collection in MS Windows® is mainly done by monitoring system events, although it can also be done by reading log files as in Linux systems.
Using the Windows event system, these logs may be collected by using module configurations with one of these two minimum configurations.
If the events are either **Application**, **System** or **Security**:
module_begin
module_name
module_type log
module_logevent
module_source
module_source_type
module_end
Or if they are events belonging to a different channel:
module_begin
module_name
module_type log
module_logchannel
module_source
module_source_type
module_end
For example, to collect all **Security** and **Windows Defender** event entries:
module_begin
module_name Windows_LogEvents_System
module_type log
module_logevent
module_source Security
module_source_type ossec
module_end
module_begin
module_name Windows_LogchannelEvents_WindowsDefender
module_type log
module_logchannel
module_source Microsoft-Windows-Windows Defender/Operational
module_source_type ossec
module_end
Inputs collected from events such as the above would be normalized by decoders as ''windows_eventchannel''.
The decoded logs of events such as the above could generate events such as ''Windows\_error\_event'', ''Short-time_multiple\_Windows\_Defender\_warning\_events'' or ''Multiple\_Windows\_Defender\_error\_events'' among others.
Using log file monitoring, the configuration is identical to that of Linux systems. A minimal configuration like this is required:
module_begin
module_name
module_type log
module_regexp
module_pattern
module_source_type
module_end
For example, to collect all log entries of an X server:
module_begin
module_name xserver
module_type log
module_regexp C:\server\logs\xserver.log
module_pattern .*
module_source_type xserver
module_end
===== SIEM events =====
With SIEM monitoring enabled and configured, a preview of the monitoring status may be accessed in the **Operation → SIEM → Dashboard** menu.
{{ :wiki:siem_dashboard.png }}
Generated SIEM events can be fuly displayed in menu **Operation → SIEM → Events**.
{{ :wiki:siem_events.png }}
Each SIEM event will have a window with the event details, showing information of the normalized log and the rule that generated the event.
{{ :wiki:siem_event_details.png }}
Depending on the decoders that normalized the log, the event will have **Dynamic fields** tab with useful log information.
{{ :wiki:siem_event_fields.png }}
Depending on the rule that generated the event, there will be **MITRE ATT&CKS®** and **SIEM groups** tabs with useful information on the impact of the event.
{{ :wiki:siem_mitre_event.png }}
{{ :wiki:siem_event_groups.png }}
The information shown both in the general Dashboard and in the events table may be included in [[:en:documentation:pandorafms:management_and_operation:09_dashboard|Pandora FMS Dashboards]] by means of the included SIEM widgets.
===== Decoders =====
Pandora FMS includes a series of decoders by default for SIEM monitoring, but any administrator may include their own.
==== Management ====
To include new decoders, first add or edit an XML file in the path indicated to **Pandora FMS server** in its configuration parameter ''siem_decoders''.
Decoders are loaded into the environment via XML files that **Pandora FMS master** server reads at each service startup.
Once decoders are loaded, their configuration is stored in the database, and each ''siemserver'' processes them for entries obtained through log collection.
From Pandora FMS Console, it will be possible to view the loaded decoders with their complete configuration from menu **Operation → SIEM → Decoders**.
It is also possible to disable decoders from this view, so that ''siemserver'' does not take them into account when normalizing log entries.
All decoders are fully loaded at each restart. This means that decoders that could not be read from XML files will not be available (even if they were at some point), and decoders that were disabled from the Console will be re-enabled again (if they exist).
==== Syntax ====
Decoders are configured and loaded in Pandora FMS with XML files. These files may have the following valid syntax:
VarValueDecoderNameREGEXPEventTypeREGEXPREGEXPREGEXPREGEXPREGEXPREGEXPField1, Field2.Sub1, Field2.Sub2string|discard
* ''var'': Used to indicate variables with their values, and use them later in XML (''$VarName'').
* ''decoder'': It is the decoder information, with its name. The XML file itself may contain several.
- ''name'': It is the name of the decoder. Several decoders may have the same name, and they are all evaluated.
- ''discard'': With a ''yes'' or ''no'' value, it allows to discard logs from the decoder evaluation if they match this one. Decoders with ''discard="yes"'' are evaluated before the rest (as long as they do not have a parent decoder ).
---
* ''parent'': It specifies the name of the parent decoder to generate a hierarchical structure.
* ''program_name'': The name of the program to be found in the log headers or in the ''source_id'' of the log.
- ''type'': It allows you to specify the type of regular expression. If not specified, **OS Regex** is used.
---
* ''type'': The type of logs for which the decoder will be compared. It must match the log ''type''.
* ''prematch'': If the contents of the log match this, it generates a normalized log.
- ''type'': It allows you to specify the type of regular expression. If not specified, **OS Regex** is used.
---
* ''regex'': Groups captured with this regular expression are normalized information for the log.
- ''type'': It alows you to specify the type of regular expression. If not specified, **OS Regex** is used.
---
* ''order'': Values captured by regex are stored in these field names, sorted by capture groups.
* ''json_null_field'': Null values of the capture groups will be stored as empty strings or discarded.
* ''offset'': It is used to indicate the order in which checks are performed and to discard from the log content for the following checks the text that matches the regular expression: ''after_parent'', ''after_regex'', ''after_prematch''. For example:
^\d\d\d\d/\d\d/\d\d \d\d:\d\d:\d\d (\w):(\d+)srcip,srcport
It will check that the text in the log matches the regular expression ''^\d\d\d\d/\d\d/\d\d \d\d:\d\d:\d\d '', will discard that content of the text and when evaluating the ''regex'', it will capture accordingly. In the example, it would remove a date from the text when evaluating to simplify the capture groups.
===== Rules =====
Pandora FMS includes a set of default rules for SIEM monitoring, but any administrator may include their own.
==== Management ====
To include new rules, first add or edit an XML file in the path indicated to Pandora FMS server in its configuration parameter ''siem_rules''.
Rules are loaded into the environment by means of XML files that **Pandora FMS master** server reads at each service start.
Once rules are loaded, their configuration is stored in the database, and each ''siemevents'' server processes them for normalized entries in SIEM monitoring.
From Pandora FMS console, it will be possible to see loaded rules with their full configuration fom menu **Operation → SIEM → Rules**.
It is also possible to disable rules from this view, so that ''siemevents'' does not take them into account when processing normalized logs.
All rules are fully loaded at each restart. This means that the rules that could not be read from the XML files will not be available (even if they were at some point). Unlike decoders, rules have an ID that allows to keep them disabled at each reload.
Rules that could not be read from the XML files will be marked in the Console as "not active" and will not be taken into account when generating SIEM events. Rules that were manually disabled by an administrator will not be taken into account either. Therefore, for rules to be evaluated, they must be active and enabled.
===== Sorting =====
//Rules// are classified in several levels, ranging from the lowest (''0'') to the highest (''15''). The following table describes each level, providing information on the severity of each event generated by SIEM monitoring.
^Level ^Title ^Description|
|0 |Ignored. |No action was taken. Used to avoid false positives. These rules are scanned before all other rules, including events with no security relevance, and do not appear in the security events panel.|
|2 |Notification of low system priority. |System notifications or status messages. They have no security relevance and do not appear in the security event panel.|
|3 |Successful/authorized events. |These include successful login attempts, events allowed by the firewall, and so on.|
|4 |Low system priority error. |Errors related to bad configurations or unused devices/applications. These have no security relevance and are usually caused by default installations or software testing.|
|5 |User-generated error. |These include forgotten passwords, denied actions, and so on. By themselves, they have no security relevance.|
|6 |Low relevance attack. |These indicate a worm or virus that has no effect on the system (such as code red for Apache servers, etc.). This also includes frequent Intrusion Detection System (**IDS**) events and frequent errors.|
|7 |Coincidence of "bad words". |These include words such as "bad", "error", etc. Most of these events are not classified and may have some relevance from a security point of view.|
|8 |First time seen. |Include events seen for the first time. The first time an **IDS** event is triggered or the first time a user logs in. It also includes security-relevant actions, such as the activation of a sniffer or similar activities.|
|9 |Invalid source error. |It includes attempts to log in as an unknown user or from an invalid source. It may have security relevance (especially if repeated). This also includes errors related to the "admin" account (root).|
|10 |Multiple user-generated errors. |These include multiple incorrect passwords, multiple failed logins, etc. These may indicate an attack or just signal that a user forgot their credentials.|
|11 |Integrity check warning. |These include messages about binary modification or the presence of rootkits (by Rootcheck). These may indicate a successful attack. They also include **IDS** events that will be ignored (large number of repetitions).|
|12 |Event of high importance. |These include error or warning messages from the system, kernel, and so on. These may indicate an attack against a specific application.|
|13 |Unusual error (high importance). |It matches a common attack pattern most of the time.|
|14 |Security event of high importance. |It is most often triggered by correlation and indicates an attack.|
|15 |Severe attack. |There is no possibility of false positives. Immediate attention is necessary.|
Based on these levels, events will have a specific severity that will be displayed on the console:
* **Informational**: Levels from 0 to 6.
* **Normal**: Levels from 7 to 8.
* **Warning**: Levels from 9 to 11.
* **Critical**: Levels from 12 to 15.
==== Syntax ====
VarValueNGROUPField1Field2.Sub1Field1Field2.Sub1TEXTRREGEXPRREGEXPRREGEXPRREGEXPDecoderNameEventTypeREGEXPREGEXPREGEXPDAYSPARENT1, PARENT2GROUPNTEXT|LINK|CVEGROUP1,GROUP2,MITRE_IDMITRE_ID
* ''var'': Used to indicate variables with their values, and use them later in XML (''$VarName'').
* ''group'': It allows rule grouping. It is also used for conditions of the same rules.
* ''rule'': It is the information in the rule:
- ''id'': Rule identifier. It must be unique (unless another rule is overwritten).
- ''level'': Level of the event when generated (''0'' to ''15''). Rules with level ''0'' do not generate an event.
- ''frequency'': Times that a concurrence must take place to generate an event. Rule frequency is checked for logs of the agent itself, not for any log.
- ''timeframe'': Time window in which concurrences must be given (seconds).
- ''ignore'': This rule restarts your ''frequency'' counters after these seconds.
- ''overwrite'': With ''yes'' or ''no'' values, it allows to overwrite the configuration of a rule with the same ID. If together with ''overwrite="yes"'' the rule has ''level="0"'', it is evaluated before any other and in case of a log match, it discards the evaluation of the rest of the rules for that log.
---
* ''if_matched_sid'': The rule is satisfied if another rule with the indicated ID triggered an alert the times indicated by ''frequency'' within the ''timeframe'' time.
* ''if_matched_group'': Like ''if_matched_sid'' but for groups.
* ''same_id'': Concurrences with the same ID must be given.
* ''different_id'': Concurrences with different IDs must be given.
* ''same_field'': Concurrences with the same values in the field must take place.
* ''different_field'': Concurrences with different values in the field must take place.
* ''description'': The text for the event to be generated.
* ''match'': If the contents of the log match this, it generates a SIEM event.
- ''type'': It allows you to specify the type of regular expression. If not specified, **OS Regex** is used.
- ''negate'': With a ''yes'' value you may deny the match.
---
* ''regex'': Same as ''match''.
- ''type'': It allows you to specify the type of regular expression. If not specified, **OS Regex** is used.
- ''negate'': With a ''yes'' value you may deny the regular expression.
---
* ''decoded_as'': The rule is satisfied if the log was decoded by the indicated decoder.
* ''category'': The rule is satisfied if the type of the decoder matches.
* ''field'': If the indicated field matches the value, it complies with the rule.
- ''negate'': With a ''yes'' value, it allows to deny the match with the field.
---
* ''program_name'': If the source of the log matches, the rule is met.
- ''negate'': With a ''yes'' value, it allows to deny matches with the source log.
---
* ''time'': If the event is generated in the indicated time range, the rule matches.
* ''weekday'': If the event is generated the days the rule complies with (''monday - sunday'', ''weekdays'', ''weekends'').
* ''if_sid'': The rule is met if any of the parent rules are met.
* ''if_group'': The rule is met if the log meets any other rule in the specified group.
* ''if_level'': The rule is met if another rule with the same level has been met.
* ''info'': Additional information for the generated event.
* ''group'': List of rule groups.
* ''mitre'': List of the MITRE IDs of the rule.
Variables may be used in the description and in ''info'' with the values of custom fields, for example: ''$(Field1)'', ''$(Field2.Sub1)'', ''$(Field2.Sub2)''.
===== Regular expressions =====
Regular expressions are sequences of characters that define a pattern.
There are two types of regular expressions valid for decoders and rules: [[#os_regex|OS Regex]] and [[#pcre2|PCRE2]].
==== OS Regex ====
They are simple regular expressions based on a library made in C language. It is designed to be simple and at the same time support the most common regular expressions.
=== Acceptable expressions ===
^Expression^Valid characters|
|''\w'' |A-Z, a-z, 0-9, '-', '@', '_'|
|''\d'' |0-9|
|''\s'' |Spaces " "|
|''\t'' |Tabulation|
|''\p'' |()*+,-.:;<=>?[]!"'#$%&|{}|
|''\W'' |Anything other than \w|
|''\D'' |Anything other than \d|
|''\S'' |Anything other than \s|
|''\.'' |Anything other|
=== Modifiers ===
^Expression^Behavior|
|''+'' |To match one or more times|
|''*'' |To match zero or more times|
=== Special characters ===
^Expression^Behavior|
|''^'' |To specify the beginning of the text|
|''$'' |To specify the end of the text|
|''|'' |To create a logical pattern "or" among multiple patterns|
=== Escaping characters ===
To use the following characters you must escape them with ''\'':
^ $ ^ ( ^ ) ^ \ ^ | ^ < |
|\$ |\( |\) | \| \| |\< |
=== Limitations ===
* The ''*'' and ''+'' modifiers may only be applied to backslash expressions, not to single characters (e.g. ''+'' is supported, ''0+'' is not supported).
* Alternation cannot be used in a group, e.g. ''(foo|bar)'' is not allowed.
* Complex backtracking is not supported, e.g., ''\p*\d*\s*\w*:'', it does not match only colon because ''\p*'' consumes the colon.
* ''.'' matches a literal period, while ''\.'' matches any character.
* ''\s'' matches only an ASCII space (32), not other blanks such as tabs.
* There is no syntax to match a caret, asterisk or more literal (although it will match an asterisk or more, along with some other characters).
==== PCRE2 ====
Perl Compatible Regular Expression (**PCRE2**) provides features such as recursive patterns, look-ahead and look-back assertions, non-capture groups, non-voracious quantifiers, extended syntax for characters and character classes, among others.
For more details, see the [[https://www.pcre.org/current/doc/html/pcre2syntax.html|PCRE2 syntax documentation]].
=== Acceptable expressions ===
^Expression^Valid characters|
|''.'' |Any character except new line|
|''\d'' |Any decimal digit, equal to ''[0-9]'' |
|''\D'' |Any character other than a decimal digit, equal to ''[^0-9]'' |
|''\h'' |Any horizontal white space character|
|''\H'' |Any character that is not a horizontal blank space|
|''\s'' |Any whitespace character, equal to ''[\t\r\n\f]'' |
|''\S'' |Any character that is not a blank space, equal to ''[^\t\r\n\f]'' |
|''\w'' |Any "word" character|
|''\W'' |Any "non-word" character|
=== Modifiers ===
^Expression ^Behavior|
|''?'' |0 or 1, greedy|
|''?+'' |0 or 1, possessive|
|''??'' |0 or 1, lazy|
|''*'' |0 or more, greedy|
|''*+'' |0 or more, possessive|
|''*?'' |0 or more, lazy|
|''+'' |1 or more, greedy|
|''++'' |1 or more, possessive|
|''+?'' |1 or more, lazy|
|''{n}'' |Exactly n|
|''{n,m}'' |At least n, no more than m, greedy|
|''{n,m}+'' |At least n, no more than m, possessive|
|''{n,m}?'' |At least n, no more than m, lazy|
|''{n,}'' |n or more, greedy|
|''{n,}+'' |n or more, possessive|
|''{n,}?'' |n or more, lazy|
=== Escape characters ===
^Expression ^Behavior|
|''\f'' |Next page (hexadecimal ''0C'')|
|''\n'' |New line (hexadecimal ''0A'')|
|''\r'' |Carriage return (hexadecimal ''0D'')|
|''\t'' |Tabulation (hexadecimal ''09'')|
|''\0dd'' |Character with octal code ''0dd'' |
|''\o{ddd..}'' |Character with octal code ''ddd..'' |
|''\xhh'' |Character with hexadecimal code ''hh'' |
|''v\x{hhh…}'' |Character with hexadecimal code ''hh..'' |
[[:en:documentation:start|Go back to Pandora FMS documentation index]]