Go back to Pandora FMS documentation index

1 Alert Configuration in Pandora FMS

1.1 Introduction

An alert is Pandora FMS's reaction to a module's value being incorrect. Such a reaction is configurable and can result in any action that can be triggered by a script configured in the Operating System where the Pandora FMS server that processes the module is running.

There are several alert types: simple alerts, avent alerts and SNMP trap alerts. This chapter discusses the alert system as a whole and particularly the first type.

1.2 Introduction to the alert system

In Pandora FMS, alerts work by defining some triggering terms, some actions selected for that alert, and finally the execution of some commands in the Pandora FMS server, which will be in charge of carrying out the configured actions.

The general alert system associates a single alert for each module, this alert can carry out one or more actions.

1.2.1 The Alert Structure

An alert consists on:

Commands: They specify what will be done. This will be the execution done by Pandora FMS server when triggering an alert. This could be writing in a log, sending an email or an SMS, executing a script, etc.

Actions : They specify how it will be done. They are the customizations of the command's arguments, they allow to customize the execution itself, adding to the command particular parameters like module name, agent, etc.

Templates: They specify when it will be done, defining the conditions for triggering the action(s). For example: when the module becomes critical.

1.2.2 The Alert System's Information Flow

When defining actions and templates, both have some general fields called Field1, Field2 and Field3 that are used to customize the alert's triggering.

These fields are applied according to a precedence order, "transferring" information from template -> action -> command, to finally be used as parameters in the execution of this command.

The precedence works like follows:

Template < Action < Command

Where the field value overwrites the content specified in the previous layer:

If a template has any content in Field1, but the action has no content in its Field1, it inherits the content of Field1 from the template. However, if the action already has its own content (configured previously while creating the action) in its Field1, it will prevail over the one that is transfered from the template. So following the succession Template -> Action -> Command, the information will be transferred from the first one to the second one and from the second one to the third one if the next step does not already have any defined information in its fields Field1, Field2, Field3...

The following diagram shows this parameter transfer from the template to the command:

This is an example for how template values are overwritten by the action's values:

For example, this is a template that triggers an alert and sends an email, containing the following fields:

Template:
- Field 1: [email protected]
- Field 2: [Alert] The alert was fired.
- Field 3: The alert was fired!!! SOS!!!

Action:
- Field 1: [email protected]
- Field 2: <left blank>
- Field 3:<left blank>

The values transferred to the command are:

Command:
- Field 1: [email protected]
- Field 2: [Alert] The alert was fired
- Field 3: The alert was fired!!! SOS!!!

Field2 and Field3 keep the values defined in the template, but Field1, uses the value defined in the action.

1.3 The Alert Command

1.3.1 Introduction

Pandora FMS's acctions in face of alert situations at the end mean server executions, in the form of commands.

1.3.2 Command creation for an alert

The form will request some descriptive information about the command:

Next, the following fields are further explained:

Name: The command's name. It must be descriptive but short.

Command: The command to be executed when the alert is triggered. Macros can be used to replace the parameters configured in alert declaration. The macros that can be used are detailed below in a specific section.

When creating the commands for alerts, it is necessary to take into account that these commands are executed by Pandora FMS server. The alerts are also executed with the privileges of the user that executes the Pandora FMS server.

When defining a command, it is convenient to test, from the command line, that the execution of the command is successful and produces the desired result (sending an e-mail, generating an entry in a log file, etc.).

Group

Command group. This will determine which alerts the command can be associated to.

Description

Long description of the alert command just for further information.

Description of the fields and possible values

For each field it is possible to configure:

Description: The label next to the text box in the configuration form of the command action.

Possible values: A collection of the possible values for that field.

If the field is configured (it is not empty), it will be a selection combo instead of a text box. The combo needs a tag (the visible option) for each possible value and a value (the sent option).

This is the supported syntax:

value1,tag1;value2,tag2;value3,tag3

For example:

A simple field where it will be possible to choose the first four numbers:

1,Number one;2,Number two;3,Number three;4,Number four

From 6.0 version on, it will be possible to show an HTML editor in a command field in the creation or edition of an alert action if that command field has as value the special token _html_editor_

Hide: This box must be checked when the field value would rather be hidden or blurred. This option will be used in case the field contains a password.

Once it created, click on the 'Create' button.

1.3.2.1 Command macros

The macros that can be used in the commands are:

_address_: Address of the agent that triggered the alert.
_address_n_ : The address of the agent that corresponds to the position indicated in "n". E.g: address_1_ , address_2__
_agent_: Alias of the agent that triggered the alert. If there is no alias assigned, the name of the agent will be used instead.
_agentalias_: Alias of the agent that triggered the alert.
_agentcustomfield_n_: Agent custom field number n (e.g. _agentcustomfield_9_).
_agentcustomid_: Agent custom ID.
_agentdescription_: Description of the agent that triggered the alert.
_agentgroup_ : Agent group name.
_agentname_: Name of the agent that triggered the alert.
_agentos_: Agent's Operative System.
_agentstatus_ : Current agent status.
_alert_critical_instructions_: Instructions for CRITICAL status contained in the module.
_alert_description_: Alert description.
_alert_name_: Alert name.
_alert_priority_: Alert’s numeric priority.
_alert_text_severity_: Text alert priority level (Maintenance, Informational, Normal Minor, Major, Critical).
_alert_threshold_: Alert threshold.
_alert_times_fired_: Number of times the alert has been triggered.
_alert_unknown_instructions_: Instructions for UNKNOWN status contained in the module.
_alert_warning_instructions_: Instructions for WARNING status contained in the module.
_all_address_ : All address of the agent that triggered the alert.
_data_: Module data that caused the alert to be triggered.
_email_tag_: Emails associated to the module’s tags.
_event_cfX_: (Only event alerts) Key of the event custom field that triggered the alert. For example, if there is a custom field whose key is IPAM, its value can be obtained using the _event_cfIPAM_ macro.
_event_description_: (Only event alerts) Text event description.
_event_extra_id_: (Only event alerts) Extra id.
_event_id_: (Only event alerts) ID of the event that triggered the alert.
_event_text_severity_: (Only event alerts) Event text severity (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
_eventTimestamp_: Timestamp where the event was created.
_fieldX_: User defined field C.
_groupcontact_: Group contact information. Configured when the group is created.
_groupcustomid_: Group custom ID.
_groupother_: Other information about the group. Configured when the group is created.
_homeurl_ : It is a link of the public URL that must be configured in setup general options.
_id_agent_: Agent’s ID, useful for building a direct URL to access Pandora FMS console webpage.
_id_alert_: Alert’s ID, used to correlate the alert with third party software.
_id_group_ : Agent group ID.
_id_module_: Module ID.
_interval_: Module’s execution interval.
_module_: Module name.
_modulecustomid_: Module custom ID.
_moduledata_X_: By using this macro ("X" is the module name) it is possible to retrieve that module's last data and if t is numeric, it is returned under the decimal format specified in the console's setup and with its unit (if it has it). It would be useful, for instance, for sending an email when an alert is triggered at the same time it sends additiona information about other modules from the same agent (which might be quite relevant).
_moduledescription_: Module description.
_modulegraph_nh_: (Only for alerts that use the command eMail) It returns an image encoded in base64 of a module’s graph with a period of n hours (e.g. _modulegraph_24h_). A correct setup of the connection between the server and the console's API is required. This setup is done on the server's configuration file.
_modulegraphth_nh_: (Only for alerts that use the command eMail) Same operation as the previous macro, only with the critical and warning thresholds of the module, provided they are defined.
_modulegroup_: Module’s group name.
_modulestatus_: Module status.
_modulelaststatuschange_: timestamp from the last status change in the module.
_moduletags_: URLs associated to the module tags.
_name_tag_: Names of the tags related to the module.
_phone_tag_: Phone numbers associated to the module tags.
_plugin_parameters_: Module plugin parameters.
_policy_: Name of the policy that the module belongs to (if it applies).
_prevdata_: Module previous data before the alert was triggered.
_rca_: Root cause analysis chain (only for services).
_secondarygroups_: It displays the agent's groups.
_server_ip_: Ip of server assigned to agent.
_server_name_: Name of server assigned to agent.
_target_ip_: IP address for the module’s target.
_target_port_: Port number for the module’s target.
_timestamp_: Time and date on which the alert was triggered (yy-mm-dd hh:mm:ss).
_timezone_: Timezone represented on _timestamp_.

1.3.3 Editing an Alert Command

You may edit the newly created alert commands by clicking on Alerts -> Commands.

To edit an alert command, click on the command's name.

Once the chosen alert has been modified, click on the 'Update' button.

The commands named eMail, Internal Audit and Pandora FMS Event cannot be modified.

1.3.4 Operations of an alert command

Delete: To delete an alert, click on the grey trash can icon to the right of the alert.

Copy: Alerts can be copied. It is especially useful to generate commands similar to other existing ones by changing some detail such as a field or group.

The alerts ”eMail”, “Internal Audit” and “Monitoring Event” cannot be deleted nor copied.

1.3.5 Predefined Commands

There is a series of predefined commands ready to use in Pandora FMS alert system.

eMail

It sends an email from Pandora FMS Server and uses the Perl "sendmail" command to do so. Pandora FMS uses the system-specific tools to execute almost all of the alerts. In this case, it will be necessary to check whether the "libmail-sendmail-perl" (an "xprobe2" package) is already installed on your system or not.

Emails are sent in HTML format, which allows creating more visually attractive templates. Notice that the receiver of the email should have access to the resources used in the template such as images.

Internal Audit

This generates a small entry within the internal Auditing System of Pandora FMS. It is saved in Pandora FMS database and it can be reviewed by the console's event viewer.

Pandora FMS Event

This creates a custom event within Pandora FMS Event Manager.

Pandora FMS Alertlog

This is a default alert to write alerts in a standard ASCII plain-text log file located in /var/log/pandora/pandora_alert.log.

SNMP Trap

It sends a parametrized SNMP trap with the arguments being used.

Syslog It sends an alert to the system registry and uses the system command named "logger" to do so.

Sound Alert

It plays a sound if an alert is received.

Jabber Alert

It sends a jabber alert to a chat room on a predefined server (configure the file named .sendxmpprc first). It uses field3 for the text message, field1 for the user's alias, and field2 for the chat-room's name.

SMS Text

It sends an SMS to a specific cellphone. Define an alert and a gateway for sending configured and accessible SMS from Pandora FMS before being able to do so. It is also possible to install one using "Gnokii" to send an SMS directly by using a Nokia phone with an USB connection. Further information on the detailed procedure is provided below.

Validate Event

It validates all events regarding a module. The agent's and module's names will be given.

Remote agent control

This sends commands to the agents with the UDP server enabled. The UDP server is used to order agents (Windows and UNIX) to "refresh" agent execution: that means, to force the agent to execute and send data

Generate Notification

This command allows you to send an internal notification to any user or group.

1.3.6 Examples of Commands

1.3.6.1 Sending alerts with Jabber

It is very easy to set up Pandora FMS to send alerts by using a Jabber Server. Jabber can be used as a system to get real time alerts as well as a history log, allowing a group of people to receive those alerts simultaneously.

1.3.6.1.1 Installing Jabber Services

Procedure for the Client:

Please install a Jabber client like Pidgin.
Register an account under 'Pidgin' by clicking on the 'Accounts' tab to configure the account.
Login to that account.

Procedure for Pandora FMS Server:

Install the package named 'sendxmpp'. This tool enables sending Jabber messages.
Create a file named '.sendxmpprc' within the '/home' folder.
Edit that file and insert the following text:

  [email protected] password

Grant permissions to the folder:

 chmod 0600 .sendxmpprc

Private messages can be sent using the command line, for instance:

  $ echo "Hello" | sendxmpp -s pandora [email protected]

To register the alert within the Pandora FMS Web Console and to add a new command and configure its variables, do the following:

Field_1: The Jabber address.
Field_2: The text to be sent.

The alert is defined as follows:

  echo _field2_ | sendxmpp -s pandora _field1_

1.3.6.1.2 Additional examples of Jabber usage

Send this to a chat room:

  $ echo "Dinner Time" | sendxmpp -r TheCook --chatroom [email protected]

Send the log entries to a Jabber destination:

  $ tail -f /var/log/syslog | sendxmpp -i [email protected]

NOTE:|Be careful not to overload public Jabber servers or they will cut off access.

1.3.6.2 Sending Emails by the Expect Script

Sometimes, it is necessary to use an authenticated SMTP to send emails. It is probably an easier and more versatile method using a simple EXPECT script instead of configuring 'sendmail' to use an authenticated SMTP. This is an example using EXPECT to send emails by using an Exchange Server:

A file called '/etc/snmp' containing the following script is created:

#!/usr/bin/expect -f
set arg1 [lindex $argv 0] 
set arg2 [lindex $argv 1]
set arg3 [lindex $argv 2]
set timeout 1 
spawn telnet myserver.com 25 
expect "220"
send "ehlo mymachine.mydomain.com\r"
expect "250"
send "AUTH login\r"
expect "334"
send "2342348werhkwjernsdf78sdf3w4rwe32wer=\r"
expect "334"
send "YRejewrhneruT==\r"
expect "235"
send "MAIL FROM: [email protected]\r"
expect "Sender OK"
send "RCPT TO: $arg1\r"
expect "250"
send "data\r"
expect "354"
send "Subject: $arg2\r"
send "$arg3 \r\r"
send ".\r"
expect "delivery"
send "quit"
quit

File permissions are changed to allow execution:

chmod 700 /root/smtp

Before trying to use it, make sure that /usr/bin/expect works properly.

To use this toghether with Pandora FMS, create a new command (or modify the already existing email alert-sending command) and specify the following fields within the Pandora FMS Alert Command definition in the field named 'Command'. Write the following:

/root/smtp _field1_ _field2_ _field3_

The script can be located anywhere on the system. Just keep in mind that the alert script is launched by the server that processes the data. If it is a network data, it will be the network server, if it is a data coming from an agent, through an XML file, then the dataserver will be the one to launch it.

If you have several physical servers, it is possible that you need to copy the same script to the same location, along with the same permissions and the same owner on all the systems where you have a Pandora FMS Server running and want to execute this alert on. Remember that Pandora FMS Network Servers must be executed as 'root' (e.g. for being able to execute ICMP latency tests). However, dataservers can be started by any user without special privileges.

The alert will be executed by the user who is executing the Pandora FMS Server process.

1.3.6.3 Sending SMS through Gnokii

To use Gnokii, use a Nokia cellphone or one compatible with Gnokii (check the compatible hardware list on the Gnokii Project Page). You will also need a USB data cable connected the cellphone and a connection to the Pandora FMS Server you intend to send the SMS Alerts from.

Gnokii supports a large variety of Nokia cellphones and some models by other manufacturers.

By using Gnokii, you may also send SMS directly from the command line. This is a very easy and quick way to send any SMS directly from a Pandora FMS Server, thereby avoiding the use of gateways sending SMS by using the Internet (which is not very useful if the network is down) or GSM hardware solutions for sending messages which are very expensive in some countries.

An alternative is the Gammu Project.

This is an example of sending an SMS from the command line using Gnokii:

echo "PANDORA: Server XXXX is down at XXXXX" | gnokii --sendsms 555123123

Gnokii is unable to send an SMS with images attached, but it can send a URL via HTTP or WAP for it to be displayed when receiving a message, for example:

echo "Image capture sample" | gnokii --sendsms 555123123 -w http://artica.homelinux.com/capture.jpg

It can also send an image's URL or one that leads to a 'light version' of the console in order to provide console access and data analysis to the user.

The development team has tested sending SMS alerts from a Nokia 6030 cellphone when Internet connection was not available. The Nokia 6030 cellphone uses the module's 6510 definition within the 'gnokiirc' file. It takes about four seconds to send an SMS.

It is possible to install a much more powerful sending gateway using Gammu.

1.3.6.4 Executing a Remote Command on another System (UNIX)

Sometimes, it is pretty interesting to execute the command on another system by means of the ssh command. The system where the command is executed should be a UNIX system and have the SSH daemon installed, started and accessible.

To avoid storing the access password on the machine that executes the command within the Pandora FMS Console, copy the server's public key to where you intend to execute the remote command on the Pandora FMS Server.

Once done, execute the following command:

ssh [email protected] [_field1_]

By using '_Field1_' as a variable, you may use any command you want.

1.4 Alert Actions

1.4.1 Introduction

Actions are the components of alerts where a command is related to the generic variables Field 1, Field 2,..., Field 10.

Actions allow defining how to launch the command.

1.4.2 Creating an Action

New actions are created by clicking on Alerts -> Action and Create.

Once you have clicked on 'Create', this window will appear:

Here is an explanation about the fields to be filled out:

Name: The action name.
Group: The action group. If the associated command has a group other than All, only the group associated with the command or the All group can be set as the action group.
Command: The command used in case of a triggered alert. You may choose among several predefined commands under Pandora FMS.
Threshold: The action's execution threshold.
Command Preview: The command to be executed on the system appears here automatically.
Field 1-10: The values of the macros from '_field1_ through '_field10_' are defined here. They are intended to be used along with the command if necessary. These fields might be a text field or a selection combo if configured.

Once you have filled out the fields, click on the 'Create' button.

To edit newly created actions, click on Alerts and Actions.

1.4.2.1 Action macros

The macros that can be used in the actions are:

_address_: Address of the agent that triggered the alert.
_address_n_ : The address of the agent that corresponds to the position indicated in "n" e.g: address_1_ , address_2__
_agent_: Alias of the agent that triggered the alert. If there is no alias assigned, the name of the agent will be used instead.
_agentalias_: Alias of the agent that triggered the alert.
_agentcustomfield_n_: Agent custom field number n (eg. _agentcustomfield_9_).
_agentcustomid_: Agent custom ID.
_agentdescription_: Description of the agent that triggered the alert.
_agentgroup_ : Agent group name.
_agentname_: Name of the agent that triggered the alert.
_agentos_: Agent's operative system.
_agentstatus_ : Current status of the agent.
_alert_critical_instructions_: Instructions for CRITICAL status contained in the module.
_alert_description_: Alert description.
_alert_name_: Alert name.
_alert_priority_: Alert’s numeric priority.
_alert_text_severity_: Priority level, in text, for the alert (Maintenance, Informational, Normal Minor, Major, Critical).
_alert_threshold_: Alert threshold.
_alert_times_fired_: Number of times the alert has been triggered.
_alert_unknown_instructions_: Instructions for UNKNOWN status contained in the module.
_alert_warning_instructions_: Instructions for WARNING status contained in the module.
_all_address_ : All addresses of the agent that triggered the alert.
_data_: Module data that caused the alert to be triggered.
_email_tag_: Emails associated to the module’s tags.
_event_cfX_: (Only event alerts) Key of the event custom field that triggered the alert. For example, if there is a custom field whose key is IPAM, its value can be obtained using the _event_cfIPAM_ macro.
_event_description_: (Only event alerts) Text description of the Pandora FMS event.
_event_extra_id_: (Only event alerts) Extra id.
_event_id_: (Only event alerts) ID of the event that triggered the alert.
_event_text_severity_: (Only event alerts) Event text severity (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
_eventTimestamp_: Timestamp when the event was created.
_fieldX_: User defined field C.
_groupcontact_: Group contact information. Configured when the group is created.
_groupcustomid_: Group custom ID.
_groupother_: Other information about the group. Configured when the group is created.
_homeurl_ : It is a link of the public URL. It must be configured in the general options of the setup.
_id_agent_: Agent’s ID, useful for building a direct URL that redirects to a Pandora FMS console webpage.
_id_alert_: Alert’s numeric ID (unique), used to correlate the alert with third party software.
_id_group_ : Agent group ID.
_id_module_: Module ID.
_interval_: Module’s execution interval.
_module_: Module name.
_modulecustomid_: Module custom ID.
_moduledata_X_: Using this macro ("X" is its actual name) the last module data is retrieved. If it is umerical it is returned under the decimal format specified in the console's setup along with is unit (if it has it). I could be used, for instance, to send an email when a module alert is triggeres anlong with additiona information from other modules from the same agent (which might be quite relevant).
_moduledescription_: Description of the module.
_modulegraph_nh_: (Only for alerts that use the command eMail). It returns an image encoded in base64 of a module’s graph with a rango of n hours (e.g. _modulegraph_24h_). A correct setup of the connection between the server and the console's API is required. This setup is done on the server's configuration file.
_modulegraphth_nh_: (Only for alerts that use the command eMail). Same operation as the previous macro, only with the critical and warning thresholds of the module, provided they are defined.
_modulegroup_: Module’s group name.
_modulestatus_: Module status.
_moduletags_: URLs associated to the module tags.
_name_tag_: Names of the tags related to the module.
_phone_tag_: Phone numbers associated to the module tags.
_plugin_parameters_: Module plugin parameters.
_policy_: Name of the policy that the module belongs to (if applies).
_prevdata_: Module's previous data before the alert has been triggered.
_rca_: Root cause analysis chain (only for services).
_secondarygroups_: It displays the agent's secondary groups.
_server_ip_: Ip of server assigned to the agent.
_server_name_: Name of server assigned to the agent.
_target_ip_: IP address for the module’s target.
_target_port_: Port number for the module’s target.
_timestamp_: Time and date when the alert was triggered (yy-mm-dd hh:mm:ss).
_timezone_: Timezone represented on _timestamp_.

1.4.3 Editing an Action

To edit an action, click on the action's name.

Once you have completed the changes, update them by clicking on the 'Update' button.

1.4.4 Deleting an Action

To delete an action, click on the gray trash icon which is located at the right side.

1.5 Alert Templates

1.5.1 Introduction

Templates define alert triggering terms (when the action should be executed).

Alert templates are associated to modules, so theta ehen the template terms are met, the linked action(s) are executed.

Their design allows generating a small generic template group that is used for most cases in Pandora FMs.

1.5.2 Creating an Alert Template

Go to the Alerts menu > Templates. You can create a new template by clicking on the Create button.

1.5.2.1 Step 1: General

You may specify in the template wizard:

Name: The name of the template.
Description: It describes the template's function. It is useful to distinguish the template from others within the alert's general view.
Priority: The field which provides information about the alert. The generated event will inherit this priority whenthe alert is triggered. It is useful for filtering when searching for alerts.

You may choose among the following priorities:

- Maintenance
- Informational
- Normal
- Warning
- Critical

1.5.2.2 Step 2: Conditions

The fields you may find in this part of the configuration are the following:

Days of Week

Days when the alert could possibly be triggered. The alerts will apply to those checked.

Use special days list

It is used to enable or disable the use of the special days list, e.g. holidays and special working days.

Time From

Time from which the alert action is going to be executed.

Time To

Alert action time limit.

Time Threshold

Time required to reset the alarm counter.

It defines the time interval where it is guaranteed that an alert will not be triggered more than the maximum number of alerts.

After the defined interval, the counter will be reset. The restart of the trigger counter will not be restarted if the alert recovers when a correct value arrives, unless the value Alert recovery is activated, in which case the counter will restart immediately after receiving a correct value.

Min. number of Alerts

The minimum number of times the terms set on the template have to be met (counting from the number defined in the module's FlipFlop parameter) so that an alert is triggered. The default value is '0', which means the alert will be triggered when the first value that meets the terms is received. It is intended to work as a filter, which might be useful to ignore false positives.

Max number of Alerts

Maximum number of alerts which could be sent consecutively within the same time interval (time threshold). It is the maximum alert counter value. No more alerts than the specified number will be received by interval.

Reset counter for non-sustained alerts

Activating this token will reset the alert counter when the indicated condition is not repeated consecutively. Its activation will also depend on the number indicated in "Min. number of alerts" being higher than 0.

If the field "Min. number of alerts" has a value of 2, it will mean that the module must go through the status assigned in "Condition type" 3 times to fire the alert.

There are two options to work with this:

If you do not check the restart token, the alert will fire after a sequence like this one:

 normal -> critical -> normal -> critical -> normal -> critical

If the reset token is checked, the number of critics must be consecutive, otherwise the counter will be reset.

 normal -> critical -> critical

Disable event

By checking this token, the event generated in the alert trigger event view will not be created.

Default Action

The default action the template will have is defined in this combo. This will be the action to be automatically created when the template is assigned to the module. You may assign one or none, but you cannot assign several default actions here.

This section offers the possibility of customizing the template itself, when it must be launched:

Condition Type: The field where the type of condition to be applied on the alert is defined. The required combos will be added according to the defined type, which are:

Regular Expression: A regular expression is used. The alert will be triggered if the module's value meets certain requirement.

By choosing the 'regular expression' condition, the possibility of selecting the trigger box appears if it matches the value. If you select it, the alert will be fired if the value matches. If not, the alert will be fired if the value does not match.

Max and Min: The used maximum and a minimum values.

When checking 'Trigger when matches the value', the alert will be launched when the value is within the indicated range between maximum and minimum and, if not marked, the alert will be launched when the value is outside the selected range.

Max: The used 'maximum' value. The alert will be triggered if the module's value is higher than the defined 'maximum' value.

Min: The used 'minimum' value. The alert will be triggered if the module's value is lower than the defined minimum value.

Equal to: Used to trigger the alert when a value must be equal to the received data. This condition, like max/min, is used only for numeric values, e.g. 234 or 124.35.

Not Equal to: It is the same as previous one but denying the condition (logic operator NOT).

Warning/Critical/Unknown status: The module status is used. The alert will be triggered when the monitor status is appropriate one:

1.5.2.3 Step 3: Advanced fields

You may adjust the following options:

Alert Recovery

The combo where alert recovery is enabled or disabled.

In the event that alert recovery is enabled, when the module no longer meets the terms specified in the template, the action linked to the arguments specified by the fields defined in this column will be executed.

Field 1 It defines the value for the '_ﬁeld1_' variable. The list of macros (which are described below) may be used here.

Field n It defines the value for the '_ﬁeldn_' variable in n is a number between 1 and 10.

Once all appropriate fields have been filled out, click on 'Finish'.

1.5.3 Replaceable Macros within Field1, Field2, Field3... Field10

It is possible to use the following macros in all cases the instances of 'Field1', 'Field2'... 'Field10' (both in the alert template, as well as the command and the action). These are 'words' that will be replaced if executed by a value. That value is depends on the time, value or agent that triggered the alert, etc.

_address_: The address of the agent that triggered the alert.
_address_n_: The address of the agent that corresponds to the position indicated in "n" e.g: address_1_ , address_2_.
_agent_: Alias of the agent that triggered the alert. If there is no alias assigned, the agent name will be used instead.
_agentalias_: Alias of the agent that triggered the alert.
_agentcustomfield_n_: Agent custom field number n (eg. _agentcustomfield_9_).
_agentcustomid_: Agent custom ID.
_agentdescription_:Description of the agent that triggered the alert.
_agentgroup_: Agent group name.
_agentname_: Name of the agent that triggered the alert.
_agentos_: Agent's operative system.
_agentstatus_: Current agent status.
_alert_critical_instructions_: Instructions for the CRITICAL status contained in the module.
_alert_description_: Alert description.
_alert_name_: Alert name.
_alert_priority_: Alert’s numeric priority.
_alert_text_severity_: Priority level, in the alert's text (Maintenance, Informational, Normal Minor, Major, Critical).
_alert_threshold_: Alert threshold.
_alert_times_fired_: Number of times the alert was triggered.
_alert_unknown_instructions_: Instructions for UNKNOWN status contained in the module.
_alert_warning_instructions_: Instructions for WARNING status contained in the module.
_all_address_: All addresses of the agent that triggered the alert.
_data_: Module data that caused the alert to be triggered.
_email_tag_: Emails associated to the module's tags.
_event_cfX_: (Only event alerts) Key of the event custom field that triggered the alert. For example, if there is a custom field whose key is IPAM, its value can be obtained using the _event_cfIPAM_ macro.
_event_description_: (Only event alerts) Textual description of the event that triggered the alert.
_event_id_: (Only event alerts) Id of the event that fired the alert.
_event_extra_id_: (Only event alerts) Extra id.
_event_text_severity_: (Only event alerts) Priority in the text of the event that triggered the alert (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
_eventTimestamp_: Timestamp when the event was created. v733
_field1_: User defined field 1.
_field2_: User defined field 2.
_field3_: User defined field 3.
_field4_: User defined field 4.
_field5_: User defined field 5.
_field6_: User defined field 6.
_field7_: User defined field 7.
_field8_: User defined field 8.
_field9_: User defined field 9.
_field10_: User defined field 10.
_field11_: User defined field 11.
_field12_: User defined field 12.
_field13_: User defined field 13.
_field14_: User defined field 14.
_field15_: User defined field 15.
_groupcontact_: Group contact information. Configured when the group is created.
_groupcustomid_: Group custom ID.
_groupother_: Other information about the group. Configured when the group is created.
_homeurl_: It is a link to the public URL. Configured in the setup general options.
_id_agent_: Agent ID, useful for building a direct URL that redirects to a Pandora FMS console webpage.
_id_alert_: Alert numeric ID (unique), used to correlate the alert with third party software.
_id_group_: Agent group ID.
_id_module_: Module ID.
_interval_: Module’s execution interval.
_module_: Module name.
_modulecustomid_: Module custom ID.
_moduledata_X_: Use this macro (named "X") to get the most recent data from the module and, if it is numeric, it will return formatted with the decimals specified on the console setup and that of its unit (if it has one). It would be useful in case of sending an email when an alert is triggered and send additional (and possibly relevant) information from other modules belonging to the same agent.
_moduledescription_:Description of the module that triggered the alert.
_modulegraph_nh_: (Only for alerts that use the command eMail). It returns an image codified in base64 of a module graph with a period of n hours (e.g. _modulegraph_24h_). A correct setup of the connection between the server and the console's api is required. This setup is done in the server configuration file.
_modulegraphth_nh_: (Only for alerts that use the command eMail). Same operation as the previous macro but with module critical and warning thresholds, provided they are defined.
_modulegroup_: The module's group name.
_modulestatus_: Module status.
_moduletags_: URLs associated to module tags.
_name_tag_: Names of the tags associated to the module.
_phone_tag_: Phone numbers associated to the module tags.
_plugin_parameters_: Module plugin parameters.
_policy_: The policy's name the module belongs to (if applies).
_prevdata_: Module previous data before the alert was triggered.
_server_ip_: Ip of server assigned to agent.
_server_name_: Name of server assigned to agent.
_target_ip_: IP address for the module’s target.
_target_port_: Port number for the module’s target.
_timestamp_: Time and date when the alert was triggered (yy-mm-dd hh:mm:ss).
_timezone_: Timezone represented on _timestamp_.

Example: The agent_agent_ has module_module_ in status_modulestatus_ with data_data_

1.5.3.1 Complete example of an alert containing replacement macros

Suppose you intend to create a log entry where every line appears in the following format:

2009-12-24 00:12:00 pandora [CRITICAL] Agent <agent_name> Data <module_data> Module <module_name> in CRITICAL status

Command Configuration

echo _timestamp_ pandora _field2_ >> _field1_

Action Configuration

Field1 = /var/log/pandora/pandora_alert.log
Field2 = <left blank>
Field3 = <left blank>

Template Configuration

Field1 = <left blank>
Field2 = [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

In the recovering section:

Field2 = [RECOVERED] [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

If an alert is triggered, the following line will be added to the log:

2009-10-13 13:37:00 pandora [CRITICAL] Agent raz0r Data 0.00 Module Host Alive in CRITICAL status

And the this one to recover the alert:

2009-10-13 13:41:55 pandora [RECOVERED] [CRITICAL] Agent raz0r Data 1.00 Module Host Alive in CRITICAL status

1.5.4 Editing a template

You may edit newly created templates by clicking on Administration -> Manage Alerts ->Templates.

To edit a template, click on the template's name.

1.5.5 Deleting a template

To delete a template, click on the gray trash icon located at the right of the alert.

1.6 Assigning alert templates to modules

Once the basic information about the alert system is known, we will show you the possible ways to assign alerts to the modules.

1.6.1 Alert management from alert submenu

1.6.1.1 Assigning alerts from alert submenu

From section List of Alerts, create new alerts through the builder:

These are the fields to fill in:

Agent: Smart auto-completion to choose the agent.
Module: Module list of the previously selected agent.
Actions:: Action to be executed when the alert is triggered. If the template has a default action, Default may be kept.
Template: Template that contains alert triggering terms.
Threshold: The alert action to not be executed more than once every 'action_threshold' seconds, regardless of the number of times the alert is triggered.

1.6.1.2 Modifying alerts from an alert's submenu

Once an alert has been created, it is only possible to modify the actions that have been added to the template's action.

It is also possible to delete the action that was selected when the alert was created by clicking on the gray trash icon located at the right side of the action, or to add new actions by clicking on the 'Add' ('+') button.

1.6.1.3 Deactivating alerts from an alert's submenu

Once the alert has been created, it is possible to deactivate it by clicking on the light-bulb icon located at the right side of the alert's name.

1.6.1.4 Deleting alerts from the alert submenu

It is possible to delete any alert by clicking on the gray trash icon located at the right side of the alert.

1.6.2 Managing alerts from within the agent

1.6.2.1 Alert assignment within the agent

From the agent management section, new alerts can be added by clicking on the corresponding tab.

These are the form's available fields:

Module: Agent module list.
Actions: Action executed when the alert is triggered. If the template has a default one, leave it as Default.
Template: Template that contains alert triggering terms.
Threshold: The alert action will not be executed more than once every 'action_threshold' seconds, regardless of the number of times the alert is triggered.

1.6.2.2 Modifying alerts from within the agent

Once an alert has been created, it is only possible to modify the actions that have been added to the template's action.

It is also possible to delete the action selected at the moment the alert was created by clicking on the gray trash icon which is located at the right side of the action, or to add new actions by clicking on the 'Add' button.

1.6.2.3 Deactivating alerts from the agent

Once an alert has been created, it is possible to deactivate it by clicking on the light bulb icon located at the right side of the alert's name.

In the example image, the second alert is disabled (note that the font color and the disabled alert icon are light grey).

1.6.2.4 Deleting alerts from within the agent

It is possible to delete any alert by clicking on the gray trash icon which is located at the alert's right side.

1.6.2.5 Alert detail

By clicking on the magnifying glass icon on the button panel of alert options, you will see a summary page of the effective alert setup.

This is the screen where each of the settings selected for the alert can be confirmed:

Select a specific action from the Actions dropdown to see an example of the final command:

1.7 Defining a threshold

In the following screenshot, there is a module called "CPU Load" for which a critical threshold and a warning thresholds will be defined.

The module edit form will be accessed to set the thresholds as shown in the following screenshot.

It is important to remember that modifying local modules is only available from the console in the Enterprise version, otherwise it will have to be done directly in the agent configuration file:

Accept and save the changes. When the value of the module CPU Load is between 70 and 90, its status will change to WARNING, and when between 91 and 100, it will become CRITICAL, showing its status in red as seen here:

1.8 Configuring an action

Create an action that is "Send an email to the operator". Go to the menu: Alerts > Actions and click on the button to create a new action:

This action uses the command 'Send email' and its fields named 'Field1', 'Field2' and 'Field3' which correspond to the target address, email subject, and message body.

1.9 Configuring an alert template

A generic alert template will be created for any module in critical status, and its default action will be to notify the group of operators by email. Define the template from the Templates section:

The priority set here as "Informational" will be used to display the event in a certain color when the alert is triggered.

Step 2 specifies the parameters that determine the specific triggering terms, such as the state the module should have or the time intervals when the template will work.

The most important parameters in this step are:

Condition type: It determines whether the alert will be triggered by a status change, a value change, etc. It is the most important parameter for the alert to work as desired. The Critical status term would be used to trigger the alert when a module is in a critical status.

Default action: The action to be executed by default when the alert is triggered. It is optional.

Time threshold: Time during which the alert will not be repeated if the incorrect status is continuously kept. If left at one day (24 hours), it will only send the alert once every 24 hours even if the module remains longer in faulty status.

Min. Number of alerts: The minimum number of times that the term has to be met (in this case, that the module is in CRITICAL state) before Pandora FMS executes the actions linked to the alert template. With a value of 0, the first time the module is faulty, the alert will be triggered.

Max. Number of alerts: 1 means that the action will be executed only once. If the value is 10, the action will be run 10 times. This is a way to limit the number of times an alert can be executed.

In section 3, Field1, Field2, Field3, etc. will be used to transfer parameters from the template to the action, and from the action to the command. In addition, in this third section alert recovery can be enabled or disabled, which consists of executing another action when the situation goes back to normal.

1.10 Associating an alert to a module

Now just link the alert template to the module. To do so, go to the 'Alert' tab within the agent where the module is located:

Here the module named 'cpu_user' and the 'critical condition' alert template have been linked. It will show the predefined action in this template ('Send email to XXX') by default.

1.11 Scaling alerts

Once a complete alert has been associated with a module, it is possible to add additional actions that are executed if the alert is repeated a certain number of times consecutively. That is called scaling alerts.

Add the additional actions and determine between which consecutive alert repetitions this action will be executed, as shown in the following capture:

When an alert is recovered, all actions that have been executed up to that point will be re-executed, not just those that correspond to the "Number of alerts match from" current setting.

In addition, a second threshold on the basis of which it will not be possible to send an alert more than once within said interval.

1.12 Stand-By Alerts

Alerts can be defined as 'active', 'deactivated' or in 'standby'. The difference between 'deactivated' and 'standby' alerts is that 'deactivated' alerts are not fired at all. They will not be shown in the alert's view either. 'Standby' alerts will be shown in alert view. They will also work - but only on the display level. They will show whether they are triggered or not, but they neither perform the assigned actions nor generate events.

Standby alerts are useful to see what happens, but they have notifications and actions disabled.

1.13 Cascade Protection

Cascade Protection is a Pandora FMS feature which allows to avoid alert overload if a group of agents cannot be reached due to a connection failure. These kinds of things tend to happen if an intermediate device such as a router or a switch is down and most of the network managed by Pandora FMS becomes unaccessible. Since network checks would not work in that situation, alerts would start being triggered due to down devices, despite not being true.

Cascade protection is enabled from agent configuration. Click on the Cascade protection option. In order for the agent to work with cascade protection, the parent agent it depends on must be correctly configured. If the parent agent currently has any critical-state module alert triggered, the lower agent with cascade protection will not execute its alerts.

This does not apply to module alerts in WARNING or UNKNOWN status.

1.13.1 Examples

These are the agents at your disposal:

Router: ICMP check module and SNMP check module, using a standard OID to verify the status of an ATM port. Latency towards your provider's router can be verified too.

Web Server: It has several modules executed by the agent: CPU, Memory, Apache process verification. It also has a four-step WEB latency check.

Database Server: It has several modules executed by the agent: CPU, Memory, MySQL process verification. It has some additional BBDD integrity checks. It can also verify connectivity to another database remotely, using a specific plugin that logs in, queries and exits, measuring the total time.

In WEB SERVER and DATABASE SERVER, ROUTER is defined as parent. Check the cascade protection check-box in WEBSERVER and DATABASE SERVER. Now define several simple alerts:

ROUTER

SNMP Check / CRITICAL -> Action, send MAIL. Latency > 200ms / WARNING -> Action, send MAIL.

WEB SERVER

CPU / WARNING MEM / WARNING PROCESS / CRITICAL -> Action, send MAIL. HTTP LATENCY / CRITICAL -> Action, send MAIL.

DATABASE SERVER

CPU / WARNING MEM / WARNING PROCESS / CRITICAL -> Action, send MAIL. SQL LATENCY / CRITICAL > Action, send MAIL.

If ROUTER connection fails, by means of which Pandora FMS connects with WEB SERVER and DATABASE, and if cascade protection is disabled, you will receive six alerts. Imagine what would happen if you had 200 Servers connected instead of six. That is the reason for why it is sometimes called an 'Alert Storm'. In worst-case scenarios, this problem may kill your Mail and Monitoring Servers or your cellphone, because they get flooded by lots of alerts or SMS messages.

However, if you have the cascade protection enabled, you will only receive one alert, saying that the router ATM interface is down. You will still see the Web and Database Servers in red, but you will not receive the alerts .

1.13.2 Service-based cascade protection

From version 7.0 OUM727 onwards, services can be used to avoid multiple source alerts reporting the same incident.

If service-based cascade protection is enabled, the service elements (agents, modules or other services) will not report problems, but the service itself will alert on behalf of the affected element.

That is to say, in the following image:

If the element 192.168.10.149 goes into critical status without affecting the rest of the service, the operator will receive an alert pointing out that 192.168.10.149 is down, but the service Service works normally.

In case that 192.168.10.149 affects the service performance, the operator will receive an alert indicating that the Service is affected by 192.168.10.149 being down.

In order to receive this information, edit or create a new alert template, using the _rca_ root cause analysis macro.

_rca_

From Pandora FMS 7.0 OUM727 onwards, this macro will provide info about the affected 'path' in the service to the operator.

For example, the value of the macro _rca_ corresponding to the service status in the image would be:

[Service -> web_service -> 192.168.10.149]

Though the status of the service would be correct.

Observation: The chain of events represented in root cause analysis represents the critical elements within a service, allowing to see what elements are affecting the service.

1.13.3 Cascade protection based on modules

The state of a module of a parent agent can be used to avoid agent alerts in case the module of the parent agent goes into critical state.

1.14 Safe operation mode (Version >= 7.0)

Safe operation mode can be enabled in agent advanced configuration options.

If the selected module's status becomes critical, the rest of the agent's modules are disabled until it goes back to warning or normal again. This allows, for example, to disable remote modules if connectivity is lost.

1.15 List of special days

Pandora FMS allows to define a list of special days for holidays and vacations that can be used in the template configuration, so that during those days alerts are not triggered.

1.15.1 Creating a Special Day

New special days are created in the "Alerts" -> "List of special days" section, by clicking on "More" or "Create" underneath the calendar.

Once one of them has been clicked, a window like this one will appear:

This is an explanation of the given options:

Date: The special day's date. The data format is 'YYYY-MM-DD'. If you want to define the same day every year, you may use '*' for the 'YYYY' entry.
Group: Select here the group to which the special day applies.
Same Day of the Week: Select a day. The date in field "Date" is dealt with the same as that weekday.
Description: Special Day's description.

Suppose that May 3, 2020 is holiday. If you define the "2012-05-03" as a "Sunday", that day becomes the same as a Sunday. Bearing in mind that templates have configuration options and will work in one way or another depending on the day of the week, this will help you make them work the way you want.

Practical example

There is a template that alerts you from Monday to Friday from 8 am to 6 pm but on Saturdays and Sundays this template will not cause any alert to go off. The 15th August is Wednesday and it is holiday, so a special day is created and in the field Same day of the week Saturday or Sunday is chosen, so alerts of any kind will not be received on August 15th as it will be as if it was Saturday or Sunday, when the template is configured to not trigger any alerts.

Once the fields have been selected, click on "Create".

1.15.2 Creating special days in bulk from an .ics file

Special days can also be created using an iCalendar file (. ics). These can be imported at the top of the window. Once imported, the corresponding data will be recorded in the current month.

1.15.3 Editing a special day

You may edit special days created within the "List of Special Days" option by clicking on "Alerts".

To edit a special day, click on the wrench icon next to the corresponding special day.

Once the changes are completed, click on the 'Update' button.

1.15.4 Deleting a Special Day

In order to delete a Special Day, click on the gray trash icon located next to the Special Day wrench icon.

1.16 Complete Alert Examples

1.16.1 Sending SMS Alerts

This example exemplifies something quite common: how to send an SMS either if something happens or it is about to happen.

You must have a tool that allows sending SMS installed, such as smstools. Suppose you have already configured your SMS account. Enter the following command:

> sendsms

Enter three parameters: <source>, <destination> and 'complete message'.
Do not forget to enclose the message in single quotes (') and to enter the 
destination number by using the international code format 
(e.g. 346276223 for Spanish phone numbers).

Next, use the "alert command" in the Pandora FMS management interface:

Within this command, send "346666666666" as the source of the message. An alphanumerical word can be used here, but some mobile phone providers cannot handle alphanumeric IDs very well. 'Field 1' and 'Field 2' will be used to define the command's performance. On the photo of the mobile phone that receives the SMS, a string identifier named 'Aeryn' is used. 'Field 1' is the target phone, while 'Field 2' is the text, defined within the alert's action.

Now define the alert's action. It will execute the predefined command and replace Field 1 and Field 2 by custom values. In this specific case, the template's alert does not return any data in the SMS. All information is defined in Alert Action.

Field 1 is the phone number. Field2 contains the text message. A few macros are used here, which will be replaced over time, when the alert goes off.

Final step: create an Alert Template (skip this if you already have a valid Alert Template). This is a very simple Alert Template, just to "go off" when a module goes into CRITICAL. This alert will be triggered once a day at the most, but if it recovers, it will be triggered again each time it recovers and is triggered again.

Now, assign a module along with an alert template and an alert action:

To have this alert field, the module must be in CRITICAL state. On the picture below, the module's configuration must be reviewed to see if its 'critical' thresholds are properly defined. If it is not, the alert will never be triggered because it is waiting for the moment to reach the 'critical' status. In this case, it is set to '20'. If a lower value is received, the module will go into CRITICAL state and the alert will be triggered.

Finally, the alert can be forced to execute it and test it. To force the alert, go to agent alert view and click on the green circle icon.

All set. We can now "force" the alert to run and test it. To force the alert, go to the agent alert view and click the green circular icon.

An SMS may appear on your mobile phone, as shown in the following picture. The obtained data is "N/A" because, when forcing the alert, no real data is received from the module.

1.16.2 Using alert commands different from email

The email is hosted as command in Pandora FMS, because 'Field 1', 'Field 2' and 'Field 3' are fields clearly intended to be used for 'addressee', 'subject' and 'message text' - but what if you intend to execute a user-defined action?

This is how a new command is custom defined. Suppose you intend to generate a log file entry for each alert alert find. The format of the log file should be something like this:

DATE_HOUR - NAME_AGENT -NAME_MODULE -VALUE- PROBLEM DESCRIPTION

VALUE is the module's value at that specific moment. There will be several log file entries, depending on the action that calls the command. The alert will define the description and the file to which the events are added.

To accomplish this, first create a command like this one:

Then, define an action:

If you take a look into the created log file, you will see the following:

2010-05-25 18:17:10 - farscape - cpu_user - 23.00 - Custom log alert #1

The alert was triggered at '18:17:10' within the agent named 'Farscape', in the module named 'cpu_sys' containing the data '23.00' and the description entered when the action was defined.

As for command execution, the field order and the other things might make you not to understand very well how the command is eventually executed, the easiest way is activating the Pandora FMS Server's debug traces within the server's configuration file located at '/etc/pandora/pandora_server.conf'. Restart the server by entering '/etc/init.d/pandora_server restart', look for the file named '/var/log/pandora/pandora_server.log' and look for the exact line which contains the execution of the user-defined alert command to see how Pandora FMS Server triggers it in detail.

1.16.2.1 Complete example of an alert with substitution macros

Suppose for a moment you intend to generate a log entry where each line is supposed to show its data in the following format:

2009-12-24 00:12:00 pandora [CRITICAL] Agent <agent_name> Data <module_data> Module <module_name> in CRITICAL status

Command configuration:

echo _timestamp_ pandora _field2_ >> _field1_

Action configuration:

Field1 = /var/log/pandora/pandora_alert.log
Field2 = <left blank>
Field3 = <left blank>

Template configuration

Field1 = <left blank>
Field2 = [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

In the recovery section:

Field2 = [RECOVERED] [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <left blank>

When executing an alert, the following line is entered in the log:

2009-10-13 13:37:00 pandora [CRITICAL] Agent raz0r Data 0.00 Module Host Alive in CRITICAL status

And the following line to recover the alert:

2009-10-13 13:41:55 pandora [RECOVERED] [CRITICAL] Agent raz0r Data 1.00 Module Host Alive in CRITICAL status

1.17 Custom module alert macros

Any amount of macros may be added to an agent module.

These macros have the following characteristics:

They are defined in the module configuration section.
They store information in the database.
They may have any name, for example: _joey_
They do not affect local configuration. files(pandora_agent.conf).
They are exclusively used in alerts.
They cannot be defined at local component level.
They can be defined in policies.

These specific macros can be added by just extending the module macros section.

Macro values can be used as part of the fields in alert definition. For Example: To include a macro to the mail sending action, the field with the e-mail body must be configured in the following fashion:

If a module is added to this alert without any defined custom macro, then no information will be displayed in the macro value section.

1.18 Event Alerts and Event Correlation

There is a specific chapter about this topic.

1.19 Quick guide to email configuration for alerts in Pandora FMS

1.19.1 Email configuration with a Gmail account

In order for Pandora FMS server to send alerts via Gmail, Pandora FMS and Postfix must be configured this way:

1.19.1.1 Pandora FMS setup

In order to properly configure your email with a Gmail account, all the fields must have the following comments in the Pandora FMS server configuration file (/etc/pandora/pandora_server.conf) except for the mta_address field, which will be configured with the IP server or localhost where the postfixserver is installed.

If Postfix is installed in the same server as Pandora FMS, the configuration in the pandora_server.conf would be like this:

mta_address localhost 
#mta_port 25
#mta_user [email protected]
#mta_pass mypassword
#mta_auth LOGIN
#mta_from Pandora FMS <[email protected]>

The following section shows briefly how to configure an alert in Pandora FMS console.

1.19.1.1.1 Action Setup

To set the mail recipient, use the mail action to XXX so you can add an email recipient to which all mail alerts will be sent.

1.19.1.1.2 Alert setup

In this case, a new alert with the module seen on the screenshot has been generated in module configuration > Alerts.

Once the alert is triggered, you can see how the alert reaches the e-mail chosen in the action:

1.19.1.2 Postfix Installation

The following packages must be installed in Pandora FMS server for postfix server to work properly together with a GMAIL account.

yum install postfix mailx cyrus-sasl-plain cyrus-sasl cyrus-sasl-lib cyrus-sasl-md5 cyrus-sasl-scram cyrus-sasl-gssapi

1.19.1.3 Postfix Configuration

Once Postfix has been installed within the server and everything works properly, except for sending emails through Gmail, follow these steps:

1-- Check that the "less secure pass" option is enabled in your Gmail account. It can be enabled through this link.(https://myaccount.google.com/lesssecureapps)

2-- Edit the /etc/postfix/main.cf file and add the following lines at the end of said file:

myhostname = <hostname> #Add here server hostname
relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
smtp_sasl_security_options = noanonymous
smtp_use_tls = yes
smtp_tls_CAfile = /etc/pki/tls/cert.pem
smtp_tls_security_level = encrypt

3-- Create the /etc/postfix/sasl_passwd file with its corresponding Gmail address and password.

nano /etc/postfix/sasl_passwd

Add the following line with the Gmail address and password to the file:

[smtp.gmail.com]:587 [email protected]m:PASSWORD

Secure it accordingly:

chmod 600 /etc/postfix/sasl_passwd
chown root:root /etc/postfix/sasl_passwd

4-- Create the /etc/postfix/tls_policy file with the following information:

nano /etc/postfix/tls_policy

[smtp.gmail.com]:587 encrypt

Secure it accordingly:

chmod 600 /etc/postfix/tls_policy
chown root:root /etc/postfix/tls_policy

5-- Turn /etc/postfix/sasl_passwd and /etc/postfix/tls_policy into a hash-type indexed file through this command:

postmap /etc/postfix/sasl_passwd && postmap /etc/postfix/tls_policy

It will create the /etc/postfix/sasl_passwd.db and /etc/postfix/tls_policy.db file.

6-- Finally, restart postfix to apply the changes as it follows:

/etc/init.d/postfix restart

7-- The performance can be checked by logging in two consoles. Execute the following command to monitor mail performance:

tail -f /var/log/maillog

The other one will send an email:

echo "Mail test" | mail [email protected]

If the preceding steps have been carried out correctly, the other console should show something like this:

Dec 18 18:33:40 OKComputer postfix/pickup[10945]: 75D4A243BD: uid=0 from=
Dec 18 18:33:40 OKComputer postfix/cleanup[10951]: 75D4A243BD: message-id=
Dec 18 18:33:40 OKComputer postfix/qmgr[10946]: 75D4A243BD: from=, size=403, nrcpt=1 (queue active)
Dec 18 18:33:44 OKComputer postfix/smtp[10953]: 75D4A243BD: [email protected], relay=smtp.gmail.com[74.125.93.109]:587, delay=3.7,  delays=0.15/0.14/1.8/1.6, dsn=2.0.0, status=sent (250 2.0.0 OK 1324249500 eb5sm36008464qab.10)
Dec 18 18:33:44 OKComputer postfix/qmgr[10946]: 75D4A243BD: removed

If this is the result, Pandora FMS will point to the Postfix server to send emails and they will be successfully sent.

1.20 Alert correlation: event and log alerts

From Pandora FMS version 741, alerts can be build based on received events or logs collected by means of the log collection system. Simple or complex alerts may be created based on a set of terms with logic relations. This feature replaces the previous one of event alerts.

Pandora FMS allows working from a much more flexible perspective, since alerts are not generated according to the status of a specific module, but on an event -which may have been generated by several different modules of different agents.

Event alerts are based on filtering rules using logical operators (and, or, xor, nand, nor, nxor) to look for events matching the configured filtering rules and if matches are found, the alert will be triggered.

They also use templates to define some parameters, such as the days on which the alert will work. However, in this case the templates do not determine when the event alert is triggered, but rather it is through the filtering rules that the events that match will be searched and the corresponding alerts will be triggered.

From version NG741, a new rule editor that allows building alerts visually can be used. The old event alert editor will still be available for some time.

Given the high number of events Pandora FMS Database is able to store, the server works on an maximal event window which is defined in the pandora_server.conf configuration file by a parameter named 'event_window'. Events generated outside the specified time range will not be processed by the server. So it does not make any sense to specify in a rule a time range wider than the one configured within the Server.

In a similar way, there is another server parameter called log_window that works in a similar fashion as the previously described one.

In order for event correlation alerts to work, it is necessary to activate the event correlation server with the parameter eventserver 1 in the Pandora FMS server configuration file.

To create event alerts it is mandatory to define the following parameters: agent, moduleand event.

1.20.1 Correlation alert creation

1.20.1.1 Correlation alerts / templates

To configure a correlated alert you will have to access through the menu to the section 'Alert correlation'.

In this overview we can see the fields:

Sort: marks the order of the evaluation of correlated alerts valuing if it is configured as "pass" or "drop". The higher it is in the list the sooner the alert will be evaluated.
Name: name of the alert.
Group: group where it has been organized.
Matched: how many times has been detected an event that coincides with the trigger rule in the current threshold.
Triggered: how many times the alert has been launched in the threshold that has been configured.
Action: Shows the actions configured in the alert.
Options: allows to operate with the action disabling it, putting it in standby, adding more actions, editing or deleting.

We create a rule and define its behavior (similar to the Alert Templates):

The configuration parameters of the templates for correlation alerts are similar to those of a module alert. There are only two specific parameters for event alerts:

Rule evaluation mode: It can be Pass or Drop. Pass means that, in case an event coincides with an alert, the rest of the alerts are still evaluated. Drop means that, in case an event coincides with an alert, the rest of alerts are not evaluated.
Group by<b>: Allows to group the rules by agent, module, alert or group. For example, if a rule is configured to jump when two critical events are received and it is grouped by agent, two critical events should arrive from the same agent. It can be deactivated.

{warning|In case of alerts that contain log rules, it will only affect the grouping per agent. If you choose a different grouping, <b>alerts based on log entries will never be fulfilled.}
Each rule is configured to jump to a certain type of event or log match; when the logical equation defined by the rules and their operators is met, the alert is fired.

1.20.1.2 Rules within a correlation alert

To define these rules, drag the elements from the left side to the "drop area" at the right to create your rule.

<b>'Configuration available items:

These elements will be enabled to guide the user to meet the rule's grammar. Hereon the grammar to be used is further explained:

S -> R | R + NEXUS +R
R -> FIELD + OPERATOR + C | FIELD + OPERATOR + C + MODIFIER
C -> VARIABLE

Where S is the set of rules defined for the correlated alert.

Drag the element to the rule definition area:

To clean up and undo all changes, the 'Cleanup' and 'Reset' buttons are available.

No changes will be saved until you click on the 'Next' button.

REMEMBER: Blocks are simultaneous when meeting a term.

(A and B)

It forces the analyzed item (whether event or log) to meet A and B simultaneously.

A and B

It forces both rules (A) and (B) to be met in the evaluation window. This means that during the last couple of seconds (defined by 'log_window' and 'event_window' parameters), entries that meet both rules must exist.

1.20.1.3 Fields within a correlation alert

To understand the operation of this section, see "Alert system".

1.20.1.4 Triggering within a correlation alert

In this section we configure the actions that we are going to perform when the alert is triggered and indicate at what intervals and how often it is going to execute such action.

In this section we see a preview of the configuration that we have done in the section "Conditions" to take into account when configuring the execution of an action.

At the bottom we can configure the actions through the fields:

Actions: action that we want to execute.
Number of alerts match: number of intervals that have to pass since the alert is fired to execute the action. If we want it to be always we should leave these fields blank.
Treshold: Interval that has to pass for the action to be executed again once the alarm is fired.

Then we visualize the list of configured actions. In this list, the field "triggering" shows us in which intervals of the alert will be executed the action as we configured in "number of alerts match". In addition, in the column "Options" we can delete or modify the configured actions.

1.20.1.5 Multiple correlated alerts

When you have multiple alerts, these have a specific evaluation order. They are always orderly evaluated, starting by the first of the list.

If the 'PASS' rule evaluation mode is configured, if a correlated alert is run, the next ones will also be evaluated. Tha is the "normal" mode.

If you configure the 'DROP' rule evaluation mode, if a correlated alrt configured through this mode is run, the evaluation of the next rules will come to a halt. This feature makes cascade alert protection possible.