Alert system

An alert is the reaction of Pandora FMS to an incorrect value of a Module. Said reaction is configurable and can consist of anything that can be triggered by a script configured in the Operating System where Pandora FMS server that processes the Module runs.

In Pandora FMS, alerts work by defining some trigger conditions, some actions chosen for that alert, and finally the execution of some commands in Pandora FMS server, which will be in charge of carrying out the configured actions.

There are several types of alerts:

• Simple alerts.
• Alerts about events.
• Alerts about traps SNMP.

Pandora FMS has a system for managing planned or scheduled service stops in Management → Alerts → Scheduled downtime menu. This system allows you to deactivate the alerts in the intervals that there is a service stoppage.

• Commands: Specify what will be done; It will be the execution that Pandora FMS server will carry out when triggering the alert.
• Actions: Specify how it will be done; they are the customizations of the command arguments.
• Templates: Specify when it will be done; define the conditions for triggering the action(s).

Templates and actions have a series of generic fields called Field1 , Field2 , Field3, (…) , Fieldn which are used to transfer the information from the template to the action and from the action to the command, to finally be used as parameters in the execution of said command.

Said information is transferred whenever the next step does not already bring information defined in its Fieldn fields. That is, in case of overlapping fields or parameters, it overwrites the action to the template (for example, if the template has Field1 defined and the action as well, the Field1 of the action overwrites the template action).

Menu Management → Alerts → Commands.

The actions that Pandora FMS will carry out in the event of alert situations will ultimately be translated into executions on the server, in the form of commands.

Menu Management → Alerts → Commands → Create.

• Command: Command that will be executed when triggering the alert. It is possible to use macros to replace the parameters configured in the alert declaration.
• Group: This determines which group of alerts you may associate the command with. You may only assign a group the user creating the alert command belongs to, unless that user explicitly belongs to group ALL.
• Field description and Field values:
  • Available Field Values: A collection of possible values for that field. If this field is set (not empty), the field will be a combo select instead of a text box. The combo needs for each possible value a label (the visible option) and a value (the sent option). The syntax is as follows: value1,label1;value2,label2;value3,label3;valueN,labelN.
  • Hide: If the field contains any passwords, this option hides the contents with asterisks.
• It is possible to display an HTML editor in a command field when creating or editing an alert action if that command field has the special token value _html_editor_ .

• eMail: Sends an email from the Pandora FMS Server. The email messages are sent in HTML format. Keep in mind that the recipient must be able to access the resources used in the template, such as images.
• Internal audit: Generates an entry in the Pandora FMS internal audit system. This is stored in the Pandora FMS database and can be reviewed with the event viewer from the console.
• Monitoring Event: Creates a custom event in the Pandora FMS event console.
• Alertlog: A predefined alert that writes alerts in plain ASCII format to the file /var/log/pandora/pandora_alert.log.
• SNMP Trap: Sends an SNMP trap with the parameters that are used.
• Syslog: Sends an alert to the system log using the system command logger.
• Sound Alert: Plays a sound on the event sound console when an alert occurs.
• Jabber Alert: Sends a Jabber alert to a chat room on a predefined server (the .sendxmpprc file must first be configured). Place the user alias in field1, the chat room name in field2, and the text message in field3.
• SMS Text: Sends an SMS to a specific mobile phone. First, you need to define an alert and configure an SMS gateway that is accessible from the Pandora FMS Server.
• Validate Event: Validates all events related to a module. The agent name and module name must be passed.
• Remote agent control: Sends commands to agents with the enabled UDP server. The UDP server is used to instruct the agents (MS Windows® and UNIX®) to refresh the agent's execution: in other words, to force the agent to execute and send data.
• Generate Notification: Allows sending an internal notification to any user. Recipients must be added manually, and each user will delete their notification when possible and/or deemed appropriate.
• Send report by e-mail and Send report by e-mail (from template): Both options allow sending a report in different formats (PDF, JSON, CSV) by email. The second option allows using a template for the attached report.

• Console notification: Allows sending a notification via the Web Console to any user. The available recipients will be added interactively. Notifications will appear and disappear as the corresponding alert is triggered or recovered. Additionally, messages will be permanently deleted according to the token Max. days before delete old messages. Pre-configured macros will display agent, module, etc. information. Ensure that the recipient has sufficient read rights over the alerted items.
• API request: Makes an API query through an alert action based on this command. The following parameters are required in this order:

1. URL: IP address or web link to the API server where the query will be made.
2. Method: Method to use, from a list of options (GET, POST, PUT, PATCH, DELETE), similar to those used by the PFMS API (except for PATCH).
3. Headers: To include request format (usually in JSON format), authorization token, etc.
4. Data: The query itself and its respective parameters.
5. SSL: Indicates whether a secure connection will be used for the connection.

A typical request includes code similar to this:

• HEADER:

Authorization: Bearer abc123token; Content-Type: application/json; X-Request-ID: 123456

• DATA:

{'title': 'foo', 'body': 'bar'}

Management → Alerts → Commands menu → click on the name of the command to be edited. Once the chosen alert has been modified, click Update button.

Actions are the alert components in which a command is related to the generic variables Field 1, Field 2, … , Field 10.

Actions allow you to define how to launch the command.

Menu Management → Alerts → Actions → Create.

• Group: The group of the action. You may only assign a group to which the user creating the alert command belongs, unless that user explicitly belongs to the group ALL. If the associated command has a group other than All, only the group associated with the command or the All group can be set as the action's group. If for some reason this defers, you will see a warning message for prompt correction by a user with the necessary rights.
• Command: Command that will be used in the event that the alert is executed. You may choose between different predefined commands in Pandora FMS.
• Threshold: An alert action is executed only once within this time interval, regardless of how many times the alert is triggered.
• Command Preview: In this field, not editable, the command to be executed in the system will automatically appear.
• Field 1 ~ Field 10: If necessary, these fields define the value of the macros, from _field1_ to _field10_, to be used in the command. These fields can be a text field or a select combo if configured.

• Console notification: This command allows you to generate a notification to the supervisor when an alert is triggered. It uses the predefined alert commandConsole notification. By modifying this predefined action, you may select the users who will be notified.
• Create Pandora ITSM ticket: When integration with Pandora ITSM is enabled, incidents may be created automatically in that application.
• Mail to Admin: Use the predefined command eMail to send an email message as configured in the Destination address field.
• Monitoring Event: Use the command of the same name to configure event types and severity (triggered and recovered), among other details.
• Pandora ilert, Pandora Slack, Pandora Telegram, Pandora Vonage: Pandora FMS can send notifications, after configuration, to multiple instant messaging applications.
• Restart agent: It allows you to send instructions (restart by default) according to the Remote agent control command to PFMS EndPoints.
• Send Report by e-mail and Send Report by e-mail (from template): For sending reports by email. Here you may configure the recipients, the report itself (or template), and its format (PDF, JSON, or CSV).

Menu Management → Alerts → Actions → click on the name of the action to modify.

Menu Management → Alerts → Actions → click on the corresponding trash can icon (Delete column).

Templates define the conditions for triggering the alert (when to run the action). They are associated to Modules, so that when the conditions of the template are met, the associated action(s) will be executed. Its design allows to generate a reduced group of generic templates that may be used for most possible cases in Pandora FMS.

Management → Alerts → Templates → Create.

Then follow the three guided steps.

• Group: The group the template will be applied to. You may only assign a group to which the user who creates the template belongs, unless said user explicitly belongs to the group ALL.
• Priority: Informative field about the alert. The event generated when the alert is triggered will inherit this priority, useful for filtering alert searches.

• Use special days list: It sets the special days calendar to be used in the template.
• Time Threshold: Time that must elapse to reset the alert counter. It defines the time interval in which an alert is guaranteed not to fire more times than the number set in Max. number of alerts. After the defined interval, the counter will be restarted. The shot counter reset will not be reset if the alert recovers upon reaching a correct value, unless the value Reset counter for non-sustained alerts is enabled, in which case the counter will be reset immediately after receiving a correct value.
• Min number of alerts: Minimum number of times that the situation defined in the template must take place (always counting from the number defined in the FlipFlop parameter of the Module) to start triggering an alert. The default value is 0, which means that the alert will be triggered when the first value that meets the condition is reached. It works like a filter, useful for ignoring false positives.
• Max number of alerts: Maximum number of alerts that can be sent consecutively in the same time interval (Time Threshold). It is the maximum value of the alert counter. No more alerts will arrive per time interval than those indicated in this field.
• Default Action: This list defines the default action that the template will have. This is the action that will be automatically created when you assign the template to the module. Add one action or none, however you cannot add multiple actions by default.
• Schedule: It establishes the days on which the alert can be triggered. It is possible to see and configure when the alert will be active each day of the week thanks to the built-in editor that is displayed by default in simple mode. In addition, by accessing the detailed mode you may configure the schedules with higher precision.
• Reset counter for non-sustained alerts: Its activation depends on whether the number indicated in Min. number of alerts is higher than 0. Enabling this token resets the alert counter when the indicated condition does not take place consecutively. For example, if the field Min. number of alerts has a value of 2, it means that the module has to go through the state assigned in Condition type 3 times to trigger the alert. There are two scenarios with the latter token:
• If the reset token is checked, the number of critical states will need to be consecutive, otherwise the counter will be reset.

• If the reset token is not checked, the alert will be triggered after an alternate or continuous sequence of critical states:

• Disable event: By checking this token, the event generated in the alert trigger event view will not be created.
• Condition type: It allows you to specify the element that will trigger the alert, e.g. a critical state (Critical state option) or just different from the normal state (Not normal state option). You may also define complex alerts (Complex alert option), for example if the sum is exactly equal to two over the last thirty days:

• Alert recovery: Combo where you may define whether or not to enable alert recovery. In the event that alert recovery is enabled, when the module no longer meets the conditions indicated by the template, the action associated with the arguments specified by the field fields defined in this column will be executed.
• In all instances of the fields field1 … field10 (both in the alert template, as well as in the command and in the action) those defined in the macro_list.

Once the configuration is complete, finish by clicking Finish.

The templates created by default are:

1. Critical condition: Configured with critical severity, a condition type in critical state, it has as default action that of sending an e-mail message to the administrator and with alert recovery enabled.
2. Manual alert: This is a template used to trigger manual alerts, the condition defined here will never be executed. This template is used to map actions and commands used to do remote management (restarting the EndPoint, executing commands on the server, etc.).
3. Warning condition: Configured with warning severity, a condition type in warning status, it has as default action that of sending an email message to an administrator and with alert recovery enabled.
4. Unknown condition: Configured with warning severity, a condition type in unknown_status, it has as default action that of sending email messages to administrators and with alert recovery enabled.
5. Default critical condition: The four previous templates may be customized. This template is read-only (system template) and is a copy of the Critical condition template. It is included due to the importance of critical status.

Management → Alerts → Module Alerts menu → click on pencil icon Builder alert.

• Agent: Autocomplete to choose the Agent.
• Module: List of modules of the previously selected Agent.
• Actions: Action that will be executed when the alert is triggered. If the template already has a default action, it can be left at Default.
• Template: Template that will contain the trigger conditions of the alert.
• Threshold: An alert action will not be executed more than once every action_threshold seconds, regardless of the number of times the alert is fired. This threshold takes precedence over the action threshold setting.

Once an alert has been created, it will only be possible to modify the actions added to the action that has the template.

It is also possible to delete the action that was selected when the alert was created by clicking the gray trash can icon to the right of the action, or add new actions by clicking the + button.

As of version 781 the default action is only shown if it is the only one existing.

From the agent administration section you may add new alerts by navigating to the corresponding tab:

There you may:

• Edit or delete each and every one of the actions of each alert assigned to the agent (column Actions).
• From the options column (Op.):
  • You may disable or enable.
  • You may set the alert in standby mode.
  • You may add an action.
  • You may clear the alert completely.
  • You may see the alert in detail.

• Define critical and warning thresholds in module.
• Associate the alert to the module, to do so, go to the alerts tab within the Agent where the Module is.

If necessary, you may create a new action and/or new template, by clicking on those buttons you will be redirected to the corresponding sections. Once the new components have been created, return to the previous step.

• With the Add alert button, the new alert is saved.
• Alert escalation: An alert escalation is additional actions that are executed if the alert is repeated a certain number of times consecutively.
  • It is only necessary to add the additional actions and determine between which consecutive repetitions (Number of matching alerts) of the alert this action will be executed.
  • When an alert recovers, all actions that were executed up to that point will be executed again, not just those that correspond to the current Number of alerts match from setting.
  • Additionally, a Threshold can be placed as a second parameter, for which an alert cannot be launched more than once during said interval.
• Finally you may configure alert message forwarding through instant messaging like Telegram, for example.

Alerts can be on, off, or in standby mode (standby). The difference between alerts that are disabled and alerts on standby is that alerts that are disabled simply will not work and therefore will not show up in the alerts view. Instead, alerts on standby will show up in the alert view and will work but only at display level. That is, it will show whether or not they are triggered but they will not carry out the actions that they have programmed nor will they generate events.

Alerts on standby are useful to be able to see them without disturbing other aspects.

Cascading protection is a Pandora FMS feature that allows avoiding massive alert overload when a group of Agents is not accessible, due to a failed main connection.

This happens, for example, when an intermediate network element such as a router or a switch fail, leaving most of the network managed by Pandora FMS inaccessible. Since network checks would fail in this scenario, alerts would start triggering for down devices but it would not actually be accurate.

For the agent to work with cascading protection enabled, the parent Agent it depends on (Advanced options, token Parent) must be correctly configured.

Cascade protection is activated from Agent settings, Advanced options section; click Cascade protection modules and/or Cascade protection services option.

Service-based cascade protection prevents elements of a service from triggering their alerts if the alert of the service to which they belong is triggered.

To enable this functionality, the Cascade protection services token must be activated in the advanced configuration of the agents for which this behavior is required, and enable the token Cascade protection enabled in the configuration of the service to which these agents belong.

When the service alert is triggered, information on which elements of the service are critical can be sent in the alert with the macro _rca_ which will indicate the root cause of the service status.

You may use the status of a Parent Agent Module to prevent them from sending Agent alerts in case it goes critical.

Safe operation mode can be enabled in the advanced configuration options of an Agent.

If the status of the selected Module goes into critical, the rest of the Agent Modules are disabled until it returns to normal or warning again. This allows, for example, to disable Remote Modules if connectivity is lost.

These specific macros can be added by expanding the macros section of any module.

• They are defined in the Module.
• Store the data in the database.
• They can have any name, for example _myMacro.
• They are not reflected in the local configuration (.conf).
• Used exclusively for alerts.
• They cannot be defined at the component level.
• They can be defined in monitoring policies.
• Set values can be used as part of the fields in the alert definition.

Pandora FMS by itself has the ability to send emails as explained in general Console settings. However, its flexibility allows sending an email with different mail platforms. Once both mechanisms have been set up, you can configure the sending action and create your alert.

Management → Alerts → Event Alerts menu.

Alerts can be built based on the events received. These alerts can be simple or complex, based on a set of rules with logical relationships.

This type of alerts allows working from a much more flexible perspective, since alerts are not generated based on the status of a specific Module, but on an event that may have been generated by several different Modules and even by different Agents.

Each event alert is configured to trigger on a certain type of event; when the logical equation defined by the rules and their operators is met, the alert will be triggered.

Management → Alerts → Event Alerts menu.

With the Create button a new event alert is added and the process is similar to the creation of a alert template. There are five steps for the complete creation of an event alert, some important aspects are:

• Paso 1, Configure: It contains the basic data such as name, group of agents to which the event alert will belong and its severity.
• Paso 2, Conditions: Step where a alert template, a list of special days, the Disable event option (the event generated in the alert trigger event view will not be created if this token is checked) and a rule evaluation mode will be assigned:

When there are two or more event alerts, they are evaluated one by one following the chronological order of creation and, if needed, establishing a hierarchy.

Each event alert has two specific configuration parameters for this purpose:

• Rule evaluation mode:It can be Pass or Drop. The former means that, in case an event meets the rules of an alert, the rest of the alerts are still evaluated below. This is the default behavior. Drop, otherwise, means that when an event meets an alert, stop evaluating the rest of the alerts.
• Grouped by: Allows grouping the rules by Agent, Group, Module or Module Alert. Thus, if a rule is configured to be triggered when two critical events are received and grouped by Agent, two critical events should arrive from the same Agent.

• Step 3, Rules: Rules within an event alert.
• Step 4, Fields: Fields within an event alert
• Step 5, Triggering: Triggered within an event alert.

When finishing the creation and returning to the global view you will have the list of registered event alerts and information about them, as well as options about them (operate with the action disabled, in standby mode, add more actions, edit or delete the corresponding event alert). It will also be possible to change the order of the different event alerts between them.

Event alerts are based on filtering rules using the following logical operators:

• and
• nand
• or
• nor
• xor
• nxor

These logical operators are used to search for events and/or expressions that match the configured filtering rules and if matches are found, the alert will be triggered. To define the alert rules it will be necessary to drag the elements on the left side to the drop area on the right side to build its rule.

Available configuration items:

These elements will be enabled to guide the user in complying with the grammar of the rule. The following is a simplified explanation of the grammar to be used:

Where S is the set of rules defined for the event alert.

Two buttons are available for clearing and undoing all changes: Cleanup and Reset.

It forces the analyzed element (event) to comply simultaneously with A and B.

It forces both rules (A) and (B) to be satisfied in the evaluation window. This means that there must be entries satisfying both rules in the last seconds (defined by the event_window parameter).

Field2 , Field3, (…) , Fieldn must be configured, which are used to transfer the information from template to action and from action to command, to finally be used as parameters in the execution of that command.

This information is transferred as long as the next step does not already have information defined in its Fieldn fields. That is, in case of overlapping fields or parameters, it overwrites the action to the template (for example, if the template has Field1 defined and the action also, the Field1 of the action overwrites the action of the template).

In this section you must configure the actions to be performed when the alert is triggered and indicate at what intervals and how often this action will be executed.

• Actions: Action that needs to be executed.
• Threshold: Time interval that has to elapse for the action to be executed again after the alarm has been triggered.

Once the above parameters have been selected, press the Add button and then you can select and view the list of configured actions (section Select the desired action and mode to view the Triggering fields for this action).

The macros that can be used within the configuration of an event alert are in the list of macros.

Management → Alerts → Log Alerts menu.

Alerts can be built based on the received logs. These alerts can be simple or complex, based on a set of rules with logical relationships.

This type of alerts allows working from a much more flexible perspective, since alerts are not generated based on the status of a specific Module, but on a log that may have been generated by several different Modules and even from different Agents.

Each log alert is configured to trigger on a certain type of event; when the logical equation defined by the rules and their operators is met, the alert will be triggered.

Management → Alerts → Log Alerts menu.

By means of the Create button, a new log alert is added and the process is similar to the alert template creation. There are five steps for creating a log alert, some important aspects are:

• Step 1, Configure: It contains basic data, such as the group of agents the log alert will belong to, the alert name and its severity.
• Step 2, Conditions: Step where an alert template, some list of special days, Disable event option (the event generated in the alert trigger event view will not be created if this token is checked) and rule evaluation mode will be assigned:

When there are two or more log alerts, they are evaluated one by one, following the chronological order of creation and, if necessary, establishing a hierarchy.

Each log alert has two specific configuration parameters for this purpose:

• Rule evaluation mode: Choosing Pass means that, in case a log meets the rules of an alert, all other log alerts are still evaluated below. This is the default performance. In the case of choosing Drop, when a log meets an alert, all other log alerts will no longer be evaluated.
• Grouped by: It allows grouping rules by Agent, Group, Module or Module Alert. Therefore, if a rule is configured to be triggered when two critical events are received and grouped by Agent, two critical events should come from the same Agent.

• Step 3, Rules: Rules within a log alert.
• Step 4, Fields: Fields within a log alert
• Step 5, Triggering: Triggered within a log alert.

Once creation is done and you go back to the global view, you will see the list of registered log alerts and information about them, as well as options about them (operate with the action disabled, in standby mode, add more actions, edit or delete the corresponding log alert). It is also possible to change the order of the different log alerts.

Event alerts are based on filtering rules using the following logical operators:

• and
• nand
• or
• nor
• xor
• nxor

These logical operators are used to search for logs and/or expressions that match the configured filtering rules and if matches are found the alert will be triggered.

To define the rules of the alert, it will be necessary to drag the elements on the left side to the drop area on the right side to build your rule.

Available configuration items:

These elements will be enabled to guide the user in complying with the grammar of the rule. The following is a simplified explanation of the grammar to be used:

Where S is the set of rules defined for the log alert.

Two buttons are available for cleaning and undoing all changes: Cleanup and Reset.

It forces the analyzed element (log) to fulfil simultaneously A and B.

It forces both rules (A) and (B) to be satisfied in the evaluation window. This means that there must be entries satisfying both rules in the last seconds (defined by the log_window parameter).

Field2 , Field3, (…) , Fieldn must be configured, which are used to transfer the information from template to action and from action to command, to finally be used as parameters in the execution of that command.

This information is transferred as long as the next step does not already have information defined in its Fieldn fields. That is, in case of overlapping fields or parameters, it overwrites the action to the template (for example, if the template has Field1 defined and the action also, the Field1 of the action overwrites the action of the template).

In this section you must configure the actions to be performed when the log alert is triggered and indicate at what intervals and how often this action will be executed.

• Actions: Action that needs to be executed.
• Threshold: Time interval that has to elapse for the action to be executed again after the log alarm has been triggered.

Once you have selected the above parameters, press the Add button and then you can choose and view the list of configured actions (section Select the desired action and mode to view the Triggering fields for this action).

The macros that can be used within the configuration of an event alert are in the list of macros.

These alerts are evaluated by the SIEM event server at the time of their generation, so for their correct operation, the SIEM monitoring must be enabled and configured.

Management → Alerts → SIEM Alerts menu.

In this section it is possible to create, edit and delete SIEM alerts. The LW permission is required to access this section.

These alerts are based on the filter system of the SIEM event views, so that any event that was displayed with the configured filter conditions will trigger the alert.

For example, if a SIEM alert is configured with a critical event filter, just before the SIEM event server generates one with that condition the alert will be triggered.

SIEM alerts, like all other alerts, have global configuration options for their triggering.

Operation → SIEM → Alerts menu.

In this section it is possible to view, enable/disable and change the standby mode of the SIEM alerts available in the environment. The LM permission is required to access this section.

Command Macros, Action Macros, and Event Alert Macros are similar to each other, with some exceptions specified in each description:

Note:

For the macro _prevdata_, it is necessary to uncomment the following section in the Pandora FMS server configuration file:

# Default texts for some events. The macros _module_ and _data_ are supported.
text_going_down_normal Module '_module_' is going to NORMAL (_data_) with previous data (_prevdata_)
#text_going_up_critical Module '_module_' is going to CRITICAL (_data_)
#text_going_up_warning Module '_module_' is going to WARNING (_data_)
#text_going_down_warning Module '_module_' is going to WARNING (_data_)
#text_going_unknown Module '_module_' is going to UNKNOWN

The server process must be restarted for the new changes to take effect.

Back to Pandora FMS documentation index