Difference between revisions of "Pandora: Documentation en: Alerts"

From Pandora FMS Wiki
Jump to: navigation, search
(Command Creation for an Alert)
(Command Creation for an Alert)
Line 107: Line 107:
 
*_plugin_parameters_: Plug-in Parameters of the module.
 
*_plugin_parameters_: Plug-in Parameters of the module.
  
*_email_tag_: Emails associated to module tags.
+
*_email_tag_: Emails associated to module tags ''(Pandora 5.0)''.
 
   
 
   
 
When it comes to creating the commands for the alerts, one must bear in mind that such commands are executed by the Pandora FMS's server which processes the module of the processed agent. Be it a data server or a network server. Alerts will also be executed with the priviledges of the user executing the Pandora FMS's server. It is advisable to test in the command line interface at command definition time, if the command's execution is successful and if it produces the desired result (send an e-mail, generate an entry in the logfile, etc).
 
When it comes to creating the commands for the alerts, one must bear in mind that such commands are executed by the Pandora FMS's server which processes the module of the processed agent. Be it a data server or a network server. Alerts will also be executed with the priviledges of the user executing the Pandora FMS's server. It is advisable to test in the command line interface at command definition time, if the command's execution is successful and if it produces the desired result (send an e-mail, generate an entry in the logfile, etc).

Revision as of 10:42, 8 January 2013

Go back to Pandora FMS documentation index

Contents

1 Alerts

1.1 Introduction

An alert is a Pandora FMS's reaction to a module's value «out or range». Such reaction is configurable and can result in sending an e-mail or an SMS to the administrator, send an SNMP trap, record the incident in the system's log, etc. An alert is, basically, any action able to be triggered by a script configured in the Operating System where the Pandora FMS's server which processes the module runs. Pandora FMS 3.0 allows to «chain» alerts in a logic sequence, so called Composed Alerts. Alerts can be disabled individually or by disabling a whole agent group. If an agent is disabled, it won't trigger alerts either. Alert management is performed in section Administration -> Manage Alerts, located at the right hand side of Pandora FMS's web console:



File:Susi1.jpg



1.2 Command

Pandora FMS's reaction to a value “out of range” can be of diverse kinds: record in a syslog, e-mail or SMS sending, or the execution of any script hosted in Pandora FMS's machine that can be processed.

The different reactions Pandora can adopt are configured in option Command of Manage Alerts within the Administration part.



Susi2.png



In this section one can modify or add its own commandos for the Alerts.

1.2.1 Command Creation for an Alert

New alert commands are created clicking on the Create button in Command option of Manage Alerts menu located in Administration menu.

Once clicked on Create a screen as follows appears.



600px



Next, the fields are introduced:

Name

The command's name. It is important to be descriptive, yet short. For example: «Log, Comunications»..

Command

Command to be executed as reaction to a module “out of range”. It is possible to use macros to replace the parameters configured in the alert declaration. The available macros are:

  • _field1_: Usually assigned as user name, phone, file, or destination for an e-mail.
  • _field2_: Usually assigned as a short event description, such as an e-mail's subject.
  • _field3_: It is a descriptive field, in case of an e-mail or an SMS it can be used for the payload.
  • _agent_: Compelete agent's name.
  • _agentdescription_: Description of the agent who fired alert.
  • _agentgroup_: Agent group name.
  • _address_: Address of the agent that fired the alert.
  • _timestamp_: A standard representation of date and time. Automatically replaced at alert's execution.
  • _data_: The values of the data that triggered the alert.
  • _alert_description_: Alert description.
  • _alert_threshold_: Alert threshold.
  • _alert_times_fired_: Number of times the alert has been fired.
  • _module_: Module name.
  • _modulegroup_: Module group name.
  • _moduledescription_: Description of the module who fired the alert.
  • _alert_name_: Alert name.
  • _alert_priority_: Numerical alert priority.
  • _id_agent_: Id of agent, useful to build direct URL to redirect to a Pandora FMS console webpage.
  • _id_alert_: Numerical ID of the alert (unique), used to correlate on third party software.
  • _policy_: Name of the policy the module belongs to (if applies).
  • _interval_: Execution interval of the module.
  • _target_ip_: IP address of the target of the module.
  • _target_port_: Port number of the target of the module.
  • _plugin_parameters_: Plug-in Parameters of the module.
  • _email_tag_: Emails associated to module tags (Pandora 5.0).

When it comes to creating the commands for the alerts, one must bear in mind that such commands are executed by the Pandora FMS's server which processes the module of the processed agent. Be it a data server or a network server. Alerts will also be executed with the priviledges of the user executing the Pandora FMS's server. It is advisable to test in the command line interface at command definition time, if the command's execution is successful and if it produces the desired result (send an e-mail, generate an entry in the logfile, etc).

Description

Long description of the alert command for information purposes.



600px



Once created, click on the Create button.

1.2.2 Edition of a command for an alert

It is possible to edit the alert commands created from option Command at the Manage Alerts menu of Administration menu.



Susi5.jpg



To edit an alert command just click on the command name.



650px



Once the chosen alert has been modified, click on the Update button.

The ”eMail”, “Internal Audit” and “Pandora FMS Event” alerts cannot be modified.

1.2.3 Delete an Alert Command

In order to delete an alert click on the red cross located at the right hand side of the alert.



Susi7.png



”eMail”, “Internal Audit” and “Pandora FMS Event” alerts cannot be deleted.

1.2.4 Predefined Commands

There are some Predefined commands, which could be adjusted if the system don't have the internal commands for executing these alerts. The development team has tested these alerts with Red Hat Linux, CentOs, Debian and Ubuntu Server.

eMail

Sends an email from Pandora FMS server.Uses the Perl sendmail.Pandora FMS works with the system specific tools for execute almost all alerts. It will be necessary that you check that the libmail-sendmail-perl xprobe2 package is already installed in your system.


Internal audit

This is only an «internal» alert that generates an small entry in Pandora FMS internal audit system. This is kept in the Pandora FMS database and it could be check with the event viewer from the console.


Pandora FMS Event

This alert create an special event into Pandora FMS event manager.

Pandora FMS Alertlog

This is a default alert to write alerts in a standard ASCII plaintext log file in /var/log/pandora/pandora_alert.log

SNMP Trap

Sends a SNMP trap when the alert occurs.

Syslog

Sends an alert to the system registry.Uses the sytem command «logger».

Sound Alert

Plays a sound when an alert occurs.

Jabber Alert

Send jabber alert to chat room in a predefined server (configure first .sendxmpprc file). Uses field3 as text message, field1 as useralias for source message, and field2 for chatroom name.

SMS Text

Sends an SMS to an specific mobil telephone. But, of course, it will need to define an alert before doing this possible, and also a gateway for sending configured and accesible SMS from Pandora FMS. It is also possible to install one using Gnokii to send SMS, directly by using a Nokia telephone with an USB wire. The process is described further on.

Validate Event

Validate all the events relationed with a module. The name of the agent and the name of the module will be given.

1.2.5 Examples of Commands

1.2.5.1 Integrating alerts with Jabber IM

It is very easy to set up Pandora FMS to send alerts through a Jabber server. Jabber can be a system to get real time alerts as well as a historic log, allowing a group of people to receive those alerts simultaneously.

1.2.5.1.1 Installing Jabber services

From the client side:

  1. Install a Jabber client, like for example Gaim (now Pidgin).
  2. Register an account (using Pidgin: configure the account clicking on "Accounts" tab).
  3. Login that account.

From Pandora FMS Server side:

  1. Install sendxmpp. With this tool your Pandora FMS server can send messages to Jabber services.
  2. Create the file .sendxmpprc inside the folder /home.
  3. Edit that file and insert the following text:
  [email protected] password
  1. Change that file permissions:
  chmod 0600 .sendxmpprc

Now you can send private messages using the command line, for example:

  $ echo "Hello" | sendxmpp -s pandora [email protected] 

To register the alert at Pandora FMS Web Console, add a new command and configure its variables. It is a good idea to do as follows:

  • Field_1: Jabber address.
  • Field_2: Text.

The alert will be defined as follows:

  echo _field2_ | sendxmpp -s pandora _field1_
1.2.5.1.2 More examples of Jabber usage

Send a message to a chat room:

  $ echo "Dinner Time" | sendxmpp -r TheCook --chatroom [email protected]

Send the log lines to a Jabber destination, as they appear:

  $ tail -f /var/log/syslog | sendxmpp -i [email protected]

NOTA: Be careful not to flood public Jabber servers or you can be banned from them.

1.2.5.2 Sending emails with Expect

Sometimes it is necessary to use an authenticated SMTP to send emails. It will be probably easier and more versatile to use a simple EXPECT script instead to configure sendmail to use an authenticated SMTP. This is an exaple using EXPECT to send emails using an Exchange server.

Then, a file called /etc/snmp with the following content is created:

#!/usr/bin/expect -f
set arg1 [lindex $argv 0] 
set arg2 [lindex $argv 1]
set arg3 [lindex $argv 2]
set timeout 1 
spawn telnet myserver.com 25 
expect "220"
send "ehlo mymachine.mydomain.com\r"
expect "250"
send "AUTH login\r"
expect "334"
send "2342348werhkwjernsdf78sdf3w4rwe32wer=\r"
expect "334"
send "YRejewrhneruT==\r"
expect "235"
send "MAIL FROM: [email protected]\r"
expect "Sender OK"
send "RCPT TO: $arg1\r"
expect "250"
send "data\r"
expect "354"
send "Subject: $arg2\r"
send "$arg3 \r\r"
send ".\r"
expect "delivery"
send "quit"
quit

The file permissions are changed to allow the execution.

chmod 700 /root/smtp 

Before trying to use it, please make sure that /usr/bin/expect works right.

To use this with Pandora FMS, you will need to create a new command (or modify the one that already exists, this is, the email alert sending) and specify the following fields in the Pandora FMS Alert command definition, in the “Command” field. It will write:

/root/smtp _field1_ _field2_ _field3_

And of course,the script could be located in any place of the system.You will only need to consider that the alert script is launched by the server that processes the data: if it is a network data, then it will be the network server. If it is a data that comes from an agent, through an XML data file, then it will be the dataserver the one that will launch it.

If you have several physical servers, then it is possible that you will need to copy the same script in the same location, with the same permissions and the same user owner in all the systems where you have a Pandora FMS server that you want to execute this alert. Also consider that the Pandora FMS network servers need to be executed as root ( to could do ICMP latency tests) and the data server could be executed as a user without priviledges.

The alert will be executed by the user who is executing the process of the Pandora FMS server.

1.2.5.3 Sending SMS with Gnokii

You could use Gnokii, it is necessary to use a Nokia mobile or a mobile that should be compatible with Gnokii ( check the compatible hardware in the Gnokii project page). You will also need a USB data wire to which you have to connect the mobile phone and the Pandora FMS server you want to send SMS alerts.


Gnokii supports a large variety of Nokia phones ( and some from other manufacturers).

With Gnokii, you can send SMS from the command line. This way is very easy and quick to send SMS directly from a Pandora FMS server, avoiding the use of gateways sending SMS through the internet (not quite useful if the network is down) or GSM hardware solutions for sending messages that are very expensive.

Another alternative to the use of Gnokii is the Gammu project.


Example of SMS sending with Gnokii from the command line:

echo "PANDORA: Server XXXX is down at XXXXX" | gnokii --sendsms 555123123

Gnokii can not send SMS with attached images, but it can send a URL HTTP/Wap for it could be visualized when a message is received, such as:

echo "Image capture sample" | gnokii --sendsms 555123123 -w http://artica.homelinux.com/capture.jpg

It could send a URL from one image or a URL that leads to a light version of the console to have access to the console from the mobile device and analyze data.

The development team has tested SMS sending from a Nokia 6030 phone, sending SMS alerts when the internet connection was not possible. The Nokia 6030 phone uses the module 6510 definition in the gnokiirc file, and it takes about four seconds to send an SMS.

It is possible to install a much more powerful sending gateway using Gammu.

1.2.5.4 Executing a Remote Command in Another System (UNIX)

Sometimes, it is interesting to execute the command in another system, to do it, use the ssh.command. The system in which the command will be executed should be UNIX, and it should have the ssh demon installed, started and accesible.

To avoid storing the password of access to the machine that executes the command in Pandora Console, the first thing you should do is to copy the server public key where you want to execute the remote command in the Pandora server.

Once this have been done, we should put as command:

ssh [email protected] [_field1_]

Using _field1_ as variable, you can use the command you want.

1.3 Command (>=5.0)

Pandora FMS's reaction to a value “out of range” can be of diverse kinds: record in a syslog, e-mail or SMS sending, or the execution of any script hosted in Pandora FMS's machine that can be processed.

The different reactions Pandora can adopt are configured in option Command of Manage Alerts within the Administration part.



Susi2.png



In this section one can modify or add its own commandos for the Alerts.

1.3.1 Command Creation for an Alert

New alert commands are created clicking on the Create button in Command option of Manage Alerts menu located in Administration menu.

Once clicked on Create a screen as follows appears.



Susi3 5.png



Next, the fields are introduced:

Name

The command's name. It is important to be descriptive, yet short. For example: «Log, Comunications»..

Command

Command to be executed as reaction to a module “out of range”. It is possible to use macros to replace the parameters configured in the alert declaration. The available macros are:

  • _field1_ - _field10_: Ten fields to customize the macro.
  • _agent_: Compelete agent's name.
  • _agentdescription_: Description of the agent who fired alert.
  • _agentgroup_: Agent group name.
  • _agentstatus_: Current status of the agent.
  • _address_: Address of the agent that fired the alert.
  • _timestamp_: A standard representation of date and time. Automatically replaced at alert's execution.
  • _data_: The values of the data that triggered the alert.
  • _alert_description_: Alert description.
  • _alert_threshold_: Alert threshold.
  • _alert_times_fired_: Number of times the alert has been fired.
  • _module_: Module name.
  • _modulegroup_: Module group name.
  • _moduledescription_: Description of the module who fired the alert.
  • _modulestatus_: Status of the module.
  • _moduletags_: Tags associated to the module.
  • _alert_name_: Alert name.
  • _alert_priority_: Numerical alert priority.
  • _alert_text_severity_: Text alert severity (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
  • _event_text_severity_: (Only event alerts) Text event (who fire the alert) severity (Maintenance, Informational, Normal Minor, Warning, Major, Critical).
  • _id_agent_: Id of agent, useful to build direct URL to redirect to a Pandora FMS console webpage.
  • _id_alert_: Numerical ID of the alert (unique), used to correlate on third party software.
  • _policy_: Name of the policy the module belongs to (if applies).
  • _interval_: Execution interval of the module.
  • _target_ip_: IP address of the target of the module.
  • _target_port_: Port number of the target of the module.
  • _plugin_parameters_: Plug-in Parameters of the module.
  • _groupcontact_: Group contact information. Configured when the group is created.
  • _groupother_: Other information about the group. Configured when the group is created.

When it comes to creating the commands for the alerts, one must bear in mind that such commands are executed by the Pandora FMS's server which processes the module of the processed agent. Be it a data server or a network server. Alerts will also be executed with the priviledges of the user executing the Pandora FMS's server. It is advisable to test in the command line interface at command definition time, if the command's execution is successful and if it produces the desired result (send an e-mail, generate an entry in the logfile, etc).

Description

Long description of the alert command for information purposes.



Susi4 5.png



Once created, click on the Create button.

1.3.2 Edition of a command for an alert

It is possible to edit the alert commands created from option Command at the Manage Alerts menu of Administration menu.



Susi5.jpg



To edit an alert command just click on the command name.



Susi6 5.png



Once the chosen alert has been modified, click on the Update button.

The ”eMail”, “Internal Audit” and “Pandora FMS Event” alerts cannot be modified.

1.3.3 Delete an Alert Command

In order to delete an alert click on the red cross located at the right hand side of the alert.



Susi7.png



”eMail”, “Internal Audit” and “Pandora FMS Event” alerts cannot be deleted.

1.3.4 Predefined Commands

There are some Predefined commands, which could be adjusted if the system don't have the internal commands for executing these alerts. The development team has tested these alerts with Red Hat Linux, CentOs, Debian and Ubuntu Server.

eMail

Sends an email from Pandora FMS server.Uses the Perl sendmail.Pandora FMS works with the system specific tools for execute almost all alerts. It will be necessary that you check that the libmail-sendmail-perl xprobe2 package is already installed in your system.


Internal audit

This is only an «internal» alert that generates an small entry in Pandora FMS internal audit system. This is kept in the Pandora FMS database and it could be check with the event viewer from the console.


Pandora FMS Event

This alert create an special event into Pandora FMS event manager.

Pandora FMS Alertlog

This is a default alert to write alerts in a standard ASCII plaintext log file in /var/log/pandora/pandora_alert.log

SNMP Trap

Sends a SNMP trap when the alert occurs.

Syslog

Sends an alert to the system registry.Uses the sytem command «logger».

Sound Alert

Plays a sound when an alert occurs.

Jabber Alert

Send jabber alert to chat room in a predefined server (configure first .sendxmpprc file). Uses field3 as text message, field1 as useralias for source message, and field2 for chatroom name.

SMS Text

Sends an SMS to an specific mobil telephone. But, of course, it will need to define an alert before doing this possible, and also a gateway for sending configured and accesible SMS from Pandora FMS. It is also possible to install one using Gnokii to send SMS, directly by using a Nokia telephone with an USB wire. The process is described further on.

Validate Event

Validate all the events relationed with a module. The name of the agent and the name of the module will be given.

1.3.5 Examples of Commands

1.3.5.1 Integrating alerts with Jabber IM

It is very easy to set up Pandora FMS to send alerts through a Jabber server. Jabber can be a system to get real time alerts as well as a historic log, allowing a group of people to receive those alerts simultaneously.

1.3.5.1.1 Installing Jabber services

From the client side:

  1. Install a Jabber client, like for example Gaim (now Pidgin).
  2. Register an account (using Pidgin: configure the account clicking on "Accounts" tab).
  3. Login that account.

From Pandora FMS Server side:

  1. Install sendxmpp. With this tool your Pandora FMS server can send messages to Jabber services.
  2. Create the file .sendxmpprc inside the folder /home.
  3. Edit that file and insert the following text:
  [email protected] password
  1. Change that file permissions:
  chmod 0600 .sendxmpprc

Now you can send private messages using the command line, for example:

  $ echo "Hello" | sendxmpp -s pandora [email protected] 

To register the alert at Pandora FMS Web Console, add a new command and configure its variables. It is a good idea to do as follows:

  • Field_1: Jabber address.
  • Field_2: Text.

The alert will be defined as follows:

  echo _field2_ | sendxmpp -s pandora _field1_
1.3.5.1.2 More examples of Jabber usage

Send a message to a chat room:

  $ echo "Dinner Time" | sendxmpp -r TheCook --chatroom [email protected]

Send the log lines to a Jabber destination, as they appear:

  $ tail -f /var/log/syslog | sendxmpp -i [email protected]

NOTA: Be careful not to flood public Jabber servers or you can be banned from them.

1.3.5.2 Sending emails with Expect

Sometimes it is necessary to use an authenticated SMTP to send emails. It will be probably easier and more versatile to use a simple EXPECT script instead to configure sendmail to use an authenticated SMTP. This is an exaple using EXPECT to send emails using an Exchange server.

Then, a file called /etc/snmp with the following content is created:

#!/usr/bin/expect -f
set arg1 [lindex $argv 0] 
set arg2 [lindex $argv 1]
set arg3 [lindex $argv 2]
set timeout 1 
spawn telnet myserver.com 25 
expect "220"
send "ehlo mymachine.mydomain.com\r"
expect "250"
send "AUTH login\r"
expect "334"
send "2342348werhkwjernsdf78sdf3w4rwe32wer=\r"
expect "334"
send "YRejewrhneruT==\r"
expect "235"
send "MAIL FROM: [email protected]\r"
expect "Sender OK"
send "RCPT TO: $arg1\r"
expect "250"
send "data\r"
expect "354"
send "Subject: $arg2\r"
send "$arg3 \r\r"
send ".\r"
expect "delivery"
send "quit"
quit

The file permissions are changed to allow the execution.

chmod 700 /root/smtp 

Before trying to use it, please make sure that /usr/bin/expect works right.

To use this with Pandora FMS, you will need to create a new command (or modify the one that already exists, this is, the email alert sending) and specify the following fields in the Pandora FMS Alert command definition, in the “Command” field. It will write:

/root/smtp _field1_ _field2_ _field3_

And of course,the script could be located in any place of the system.You will only need to consider that the alert script is launched by the server that processes the data: if it is a network data, then it will be the network server. If it is a data that comes from an agent, through an XML data file, then it will be the dataserver the one that will launch it.

If you have several physical servers, then it is possible that you will need to copy the same script in the same location, with the same permissions and the same user owner in all the systems where you have a Pandora FMS server that you want to execute this alert. Also consider that the Pandora FMS network servers need to be executed as root ( to could do ICMP latency tests) and the data server could be executed as a user without priviledges.

The alert will be executed by the user who is executing the process of the Pandora FMS server.

1.3.5.3 Sending SMS with Gnokii

You could use Gnokii, it is necessary to use a Nokia mobile or a mobile that should be compatible with Gnokii ( check the compatible hardware in the Gnokii project page). You will also need a USB data wire to which you have to connect the mobile phone and the Pandora FMS server you want to send SMS alerts.


Gnokii supports a large variety of Nokia phones ( and some from other manufacturers).

With Gnokii, you can send SMS from the command line. This way is very easy and quick to send SMS directly from a Pandora FMS server, avoiding the use of gateways sending SMS through the internet (not quite useful if the network is down) or GSM hardware solutions for sending messages that are very expensive.

Another alternative to the use of Gnokii is the Gammu project.


Example of SMS sending with Gnokii from the command line:

echo "PANDORA: Server XXXX is down at XXXXX" | gnokii --sendsms 555123123

Gnokii can not send SMS with attached images, but it can send a URL HTTP/Wap for it could be visualized when a message is received, such as:

echo "Image capture sample" | gnokii --sendsms 555123123 -w http://artica.homelinux.com/capture.jpg

It could send a URL from one image or a URL that leads to a light version of the console to have access to the console from the mobile device and analyze data.

The development team has tested SMS sending from a Nokia 6030 phone, sending SMS alerts when the internet connection was not possible. The Nokia 6030 phone uses the module 6510 definition in the gnokiirc file, and it takes about four seconds to send an SMS.

It is possible to install a much more powerful sending gateway using Gammu.

1.3.5.4 Executing a Remote Command in Another System (UNIX)

Sometimes, it is interesting to execute the command in another system, to do it, use the ssh.command. The system in which the command will be executed should be UNIX, and it should have the ssh demon installed, started and accesible.

To avoid storing the password of access to the machine that executes the command in Pandora Console, the first thing you should do is to copy the server public key where you want to execute the remote command in the Pandora server.

Once this have been done, we should put as command:

ssh [email protected] [_field1_]

Using _field1_ as variable, you can use the command you want.

1.4 Actions

Actions are the components of alerts where a command (which is described in the previous section) is linked with the generic variables Field 1, Field 2 and Field 3. These actions will be used later in the alert templates that are the ones that associate a data condition with an specific action.

1.4.1 Creating an action

New Actions are created pressing the Create button from Action in the Manage Alerts menu from Administration menu.



Accion1.jpg



Once you have pressed on Create, one screen as the following will be shown:



Accion2.jpg



Next are the fields that you should fill in:

  • Name: Name of the action.
  • Command: In this field is defined the command that will be used in case the alert will be executed . You can choose between the different commands that are defined in Pandora.
  • Field 1: In this field is defined the Field 1 variable value, that will be used in the command if necessary.
  • Field 2: In this field is defined the Field 2 variable value, that will be used in the command if necessary.
  • Field 3: In this field is defined the Field 3 variable value,that will be used in the command if necessary,
  • Command Preview: In this field, not editable, will automatically appear the command that will be executed in the system.

Once you have filled the fields, press on the Create button.



Boton1.jpg



From Action in the Manage Alerts menu from Administration menu, it is possible to edit the actions that have been created.

1.4.2 Editing an action



Caca1.png



To edit the action, you will only have to press on the name of the Action.



Sugus.png



Once these changes have been done, update pressing on the “Update” button.

1.4.3 Deleting an Action

To delete an Action, press on the red "x" that is on the Action right.



Sipo.jpg



1.5 Actions (>=5.0)

Actions are the components of alerts where a command (which is described in the previous section) is linked with the generic variables Field 1, Field 2, ..., Field 10. These actions will be used later in the alert templates that are the ones that associate a data condition with an specific action.

1.5.1 Creating an action

New Actions are created pressing the Create button from Action in the Manage Alerts menu from Administration menu.



Accion1.jpg



Once you have pressed on Create, one screen as the following will be shown:



Accion2.jpg



Next are the fields that you should fill in:

  • Name: Name of the action.
  • Group: Group of the action.
  • Command: In this field is defined the command that will be used in case the alert will be executed . You can choose between the different commands that are defined in Pandora.
  • Threshold: The action execution threshold.
  • Command Preview: In this field, not editable, will automatically appear the command that will be executed in the system.
  • Field 1-10: In this fields are defined the values of the macros _field1_ to _field10_, that will be used in the command if necessary.

Once you have filled the fields, press on the Create button.



Boton1.jpg



From Action in the Manage Alerts menu from Administration menu, it is possible to edit the actions that have been created.

1.5.2 Editing an action



Caca1.png



To edit the action, you will only have to press on the name of the Action.



Sugus.png



Once these changes have been done, update pressing on the “Update” button.

1.5.3 Deleting an Action

To delete an Action, press on the red "x" that is on the Action right.



Sipo.jpg



1.6 Alert Template

Templates are alerts with all the parameters defined. They only need the agent to which they are assigned and the module that is used to activate the command or the reaction when a value is "out of range". Templates are used to do the administrator management easier, so when they are done they could be assigned easily to the required agents.


1.6.1 Creating a Template

The new Templates are created pressing on the Create button at Templates, in the Manage Alerts menu, from the Administration menu.



Planti.jpg



Once you have pressed on Create, a new screen as the following will appear.



Sabo.jpg



Here are detailed the fields to fill in:

  • Name: The name of the template.
  • Description:Describes the template function and is useful to identify the template from others in the alert general view.
  • Priority: Field that gives information about the alert. It is useful to search alerts. You can choose between the following priorities:
    • Maintenance
    • Informational
    • Normal
    • Warning
    • Critical
  • Condition Type: Field where the kind of condition that will be applied on the alert is defined.The required combos will be added according to the chosen kind.There are the following fields:
  • Regular Expression: The regular expression is used. The alert will be fired when the module value perform a fixed condition expresed using a regular expression, this is the condition used to fire on string/text data. The other conditions are for status or numerical data.



Regular.jpg



By choosing the regular condition it appears the possibility to select the Trigger box when matches the value. In case of select, the alert will be fired when the value matches, and in case of not selecting it, the alert will be fired when the value does not match.

  • Max and Min: A maximum and a minimum value are used.



Maximo.jpg



By choosing the regular condition the possibility to select the Trigger box when matches the value will appear.In case of selecting it, the alert will be fired when the value is out of the range selected between the maximum an the minimum.In case of not selecting it, the alert will be launched when the value would be between the range selected betweeb the maximum and the minimum.

  • Max: A maximum value is used. The alert will be fired when the module value would be bigger than the maximum value selected.



Minimo.jpg



  • Min: A minimum value is used. The alert will be fired when the module value would be lower than the minimum value selected.



Minimo1.jpg



  • Equal to: The value Equal to is used. The alert will be fired when the module value would be the same as the selected one. It is used ONLY for numerical values (for example 0 or 0.124).



Equal.jpg



  • Not Equal to: Similar to previous but adding a logical NOT.



Notequal.jpg



  • Warning Status: The module state is used.The alert will be fired when this state would be Warning.



Estupido.jpg



  • Critical Status: The module state is used.The alert will be fired when this state would be Critical.



Critical.jpg





Critical.jpg



Once the fields have been filled, press on the "Next" button and this way you will have access to the following screen.



Pincha.png



Next we are going to detail the fields to fill in:

Days of Week

Days when the alert could be fired.

Use special days list

Enable/disable use of special days (holidays and special working days) list.

Time From

Time from which the action of the alert will be executed.

Time To

Time until the action of the alert will be executed.

Time Threshold

Defines the time interval in which it is guaranteed that an alert is not going to be fired more times than the number fixed in Maximum number of alerts. If the defined interval is exceeded, an alert will not recover if it comes to an specific value, except if the alert Recover value would be activated. In this case it is recovered inmediatelly after receiving an specific value,regardless the threshold.

Min number of alerts

Minimum number of times that the data has to be out of range (always counting from the number defined in FlipFlop parameter of the module) to start firing an alert. Default is 0, which means that the alert will be fired when the first value satisfies the condition. It works as a filter, necessary to eliminate false positives.

Max number of alerts

Maximum number of alerts that could be sent consecutively in the same time interval (Time Threshold).

Field 1

Defines the value for the "_field1_" variable. Here could be used the list of macros that is described next.

Field 2

Defines the value for the "_field2_" variable.

Field 3

Defines the value for the "_field3_" variable.

Default Action

In this combo is defined the action by default that the template is going to have. This is the action that will be automatically created when the template would be assigned to the module. You can put none or one, but you can not put several actions by default.



Combo.jpg



Next are the fields that you should fill in:

Alert Recovery

Combo where you can define if the alert recovery is enabled or not.In case that the alert recovery is enabled, when the module would have again values out of the alert range, the alert that matches with the Field 1 defined in the alert and with the Field 2 and 3 that are defined next, will be executed.

Field 2

Defines the value for the "_field2_" variable in the alert recovery.

Field 3

Defines the value for the "_field3_" variable in the alert recovery.

Once the fields have been filled in, press on the "Finish" button.

1.6.2 Replaceable Macros in Field1, Field2 and Field3

It is possible to use the following macros in all cases of the fields Field1, Field2 and Field (in the alert template, in the command and in the action). These are "words" that are replaced when executing by a value, that will change depending on the moment, value, agent that fires the alert, etc.

  • _field1_ : User defined field 1.
  • _field2_ : User defined field 2.
  • _field3_ : User defined field 3.
  • _agent_ : Name of the agent that fired the alert.
  • _agentdescription_ : Description of the agent who fired alert
  • _agentgroup_ : Agent group name
  • _address_ : Address of the agent that fired the alert.
  • _timestamp_ : Time when the alert was fired (yy-mm-dd hh:mm:ss).
  • _data_ : Module data that caused the alert to fire.
  • _alert_description_ : Alert description.
  • _alert_threshold_ : Alert threshold.
  • _alert_times_fired_ : Number of times the alert has been fired.
  • _module_ : Module name
  • _modulegroup_ : Module group name.
  • _moduledescription_ : Description of the module who fired the alert
  • _alert_name_ : Alert name
  • _alert_priority_ : Numerical alert priority
  • _id_agent_ : Id of agent, useful to build direct URL to redirect to a Pandora FMS console webpage.
  • _id_alert_ : Numerical ID of the alert (unique), used to correlate on third party software
  • _policy_ : Name of the policy the module belongs to (if applies).
  • _interval_ : Execution interval of the module.
  • _target_ip_ : IP address of the target of the module.
  • _target_port_ : Port number of the target of the module.
  • _plugin_parameters_ : Plug-in Parameters of the module.

1.6.2.1 Orders for the replacement of the Macros and _field*_ fields

After describing what are commands, actions and templates, probably you are questioning yourself about the necessity of defining the fields Field1, Field2 and Field3 in each of them and what is the sense of all of this.

When an alert is fired, the field* values are brought from the action to the command, and from the template to the command. This is, if in the action the _field1_value is different from an empty string, then it will ignore the command that is brought from the template and this will not have effect. If the _field1_ value of the command is a value different to _field1_ this means that it will ignore any parameter that comes from the field1 from the action or from the template, and that neither the action nor the template will be able to redefine it. As it has got _field1- as value. this means that it is ordering the command that insert in this field whatever comes from the action or from the template.

In the action the same thing happens, but in a more subtle way. If this field is empty, this means than any thing that is brought to it from the alert screen will be brought to the command. But if this field is different from the empty string, then it will use the values from this field and the values that comes from the template will be ignored.

This has been thought this way to offer the possibility to establish some "fixed" parameters by command or action and have always the possibility of doing them flexibles.

1.6.2.2 Complete example of alert with replacement macros

Supposing you want to create an entry in a LOG where in each line appears the following format:

2009-12-24 00:12:00 pandora [CRITICAL] Agent <agent_name> Data <module_data> Module <module_name> in CRITICAL status

Command Configuration

echo _timestamp_ pandora _field2_ >> _field1_

Action Configuration

Field1 = /var/log/pandora/pandora_alert.log
Field2 = <En blanco>
Field3 = <En blanco>

Template Configuration

Field1 = <En blanco>
Field2 = [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <En blanco>

In the recovering section:

Field2 = [RECOVERED] [CRITICAL] Agent _agent_ Data _data_ Module _module_ in CRITICAL status
Field3 = <En blanco>

This way when executing an alert the following line will be placed in the LOG:

2009-10-13 13:37:00 pandora [CRITICAL] Agent raz0r Data 0.00 Module Host Alive in CRITICAL status

And the following line when recovering the alert:

2009-10-13 13:41:55 pandora [RECOVERED] [CRITICAL] Agent raz0r Data 1.00 Module Host Alive in CRITICAL status

1.6.3 Editing a Template

It is possible to edit the templates that have been created from Templates in the Manage Alerts menu from the Administration menu.



Plantilla.jpg



To edit the template you only need to press on the name of the template.

1.6.4 Creating a duplicate of a Template

Is possible to duplicate a template that has been created from Templates in the Manage Alerts menu from the Administration menu.

To duplicate the template you will only need to press on the icon that is on the right of the kind of template.



File:Plantiver.jpg



1.6.5 Deleting a Template

To delete a template press on the red cross that is on the right side of the alert.



Cruz.jpg



1.7 Assigning Alert Templates to Modules

Until now we have defined the commands and actions as the response that Pandora FMS gives for an "out of range" value.Through these templates we define when a value is "out of range" and which circumstances should be given in order that Pandora FMS could work.In this section it is described the way for relating the Templates and the Actions with Pandora agents and with the modules of these agents. This operation is the one that finally does that Pandora FMS "reacts" when it is a date out of an specific range.

The alerts could be assigned in two ways from the Alerts submenu or from the Manage Agents submenu, both of them in the Administration menu, but we can also assign them from the submenu Policy Administration menu as we will see in the chapter Monitoring with policies.


1.7.1 Alert management from Alert submenu

1.7.1.1 Assigning Alerts from Alert Submenu

Alert assignment for modules is done filling the required fields and pressing on the "Add" button at Manage Alerts from the Administration menu.



Pinar.jpg



Next are the fields that should be filled in:

  • Group: Through a combo you can choose the group the agent belongs to.
  • Agent: Writing the name of the Agent for assigning the alert to it.
  • Module: Writing the module that will be used in order the alert could be fired.
  • Template: Through a combo you can choose the template that you want to use to configure the alert.
  • Actions:Allows to choose between all the alerts that have been configured. The selected action will be added to the action that is defined in the template. It is possible to choose more than one action.

At the moment we select an action two new fields will appear.These fields are From and to. In these fields is defined the number of alerts that should be in order to could execute the action.



File:Ejecutar.jpg



1.7.1.2 Modifying alerts form the Alert Submenu

Once an alert has been created, it is only possible to modify the actions that have been added to the action that the template has got.

It is also possible to delete the action that was selected when you created the alert by doing click on the red cross that is on the right of the action or to add new actions selecting them from the combo. Filling the From and TO data and pressing on the "Add" button.



Modifica.jpg



1.7.1.3 Deactivating Alerts from the Alert Submenu

Once the alert has been created, it is possible to deactivate it by doing click in the light bulb that is on the right of the name of the alert.



Desha.jpg



The alerts that are availables are in blue and the alerts that are not availables are in yellow.

1.7.1.4 Deleting Alerts from the Alert Submenu

It is possible to delete any Alert pressing on the red cross that is at the right of the Alert.



Filter.jpg



1.7.2 Managing Alerts from the Agent

1.7.2.1 Alert assignment from the Agent

Other option to add an alert is doing it from the Agent. Press on the Manage Agents Submenu from the Administration Menu, where are all Pandora agents.



Tarantu.jpg



Choose an agent and click on Alerts box.



Vicho.jpg



Next we are going to detail the fields that should be filled in:

  • Module: To writte the module that will be use in order the alert fires.
  • Template:Trough a combo you can select the template that you want to configure the alert.
  • Actions: Allows to choose between all the actions that have been configured. The choosen action will be added to the action that is defined in the template. It is possible to select more than one action.

At the moment of selecting an action two new files are shown. These fields are From and To. In these fields are defined the number of alerts that should be appear to execute the action.



File:Lagarto.jpg



1.7.2.2 Modifying Alerts from the Agent

Once an alert has been created, it is only possible to modify the actions that have been added to the action that the template has got.

It is posible to delete the action that was choosen to create the alert by pressing on the red cross that is on the right side of the action or by adding new actions selecting them from the combo. Filling the From and To and pressing on the "Add" button.



Gusano.jpg



1.7.2.3 Deactivating Alerts from the Agent

Once an alert has been created, it is possible to deactivate it by pressing on the light bulb that is on the right side of the alert name.



Cuca.jpg



The alerts that are availables are in blue and the alerts that are not availables are in yellow.

1.7.2.4 Deleting Alerts from the Agent

It is possible to delete any Alert by pressing on the red cross that in on the right side of the Alert.



Adorno.jpg



1.8 Scaling Alerts

Alert scaling consists of the possibility or doing different actions depending on the severity of the situation. The severity of the situation is established by the number of times that a value out of range appears. For example, if an alert is fired when the CPU of a system is at 90% , then it is possible to configure it to send an email at any case and an SMS when the value out of range has been taken place more than 5 times.

This alert scaling is done by configuring more than one action in one alert and filling in well the fields From and To.



Ciempies.jpg



1.9 Full alert examples

1.9.1 Sending SMS alerts

In this example we are goint to see something very frequent: to send an SMS when something happen or it's about to happen.

To made this, we will use a script published in our website (http://pandorafms.org) in the Module Exchange Library section. This script uses a commecial Perl API to send SMS using a commercial HTTP gateway (you need to create an account and pay money). This is very easy, because once you've setup the account and configured the script (just put your user & pass), it's ready to be used.

Let's suppose you have configured your SMS account and installed the script in the Pandora FMS server. Run the command:

> sendsms 

You must give three parameters:  <source> <destination> 'Full message'
Don't forget to send the message with single quotes (), and put the destination number 
with international code (346276223 for spanish phones, for example)

Our first step, after being sure that SMS sendsms command is ready to be used, is to define the "alert command". We define the command in the Pandora FMS administration interface:



Smsalert sample1.png



In this command, we send "346666666666" as source of message. We could use a word (alphanumerical) but some mobile operators doesnt manage well the alphanumeric ID's. Field1 and Field2 will be used to defined the command behaviour. In the photo of the mobile phone receiving the SMS I use a string identifier: "Aeryn". Field1 will be the destination phone, and field2 the text, defined in the Alert Action.

Now define the alert action. This execute the command defined before, replacing field1 and field2 with custom values, in this specific case, template alert doesnt put any data on the SMS; all information is defined in Alert Action.



Smsalert sample3.png



Field1 is my phone number (a bit obfuscated, I don't want you to call me in the night ;). In Field2 it's the SMS text message, I'm using here a few macros, which will be replaced in the runtime, when alert is produced.

Final step!: We are going to create an Alert Template (skip this if you have a valid one). We want to create a very simple Alert Template, just to "fire up", when a module will be CRITICAL. That alert will fire once per day max, but if it recovers, will fire again each time it recover and fire again.



Smsalert sample5.png





Smsalert sample6.png



Now, just assign a module with an alert template and an alert action:



Smsalert sample4.png



To get this alert fied, module must be on CRITICAL. In following screenshot, I will review the module configuration to see if their CRITICAL threshold are defined. If it was not, alert will never fire because is waiting to have a CRITICAL status. In my case I've set to 5. When a value of 6 is received, module will set on CRITICAL and alert will fire.



Smsalert sample7.png



All ready. We can "force" the alert to execute and test it. To force the alert, go to the agent alert view, and click on the round green icon:



Smsalert sample8.png



An SMS should be appear in my mobile phone, just as the following photo. I get a "N/A" data because when you force the alert, no real data is received by the module.



Smsalert sample9.png



1.10 Correlation

The Alert Correlation allows to use more than one module to generate a Pandora FMS reaction. These modules could be from the same agent or from different ones.

The management of Correlated Alerts is done at Administration>Manage Alerts >Correlation.

1.10.1 Creating Correlated Alerts

To create a correlated alert click on the "Create" button that is at Administration>Manage Alerts >Correlation.



File:Correlativa.jpg



Once you have clicked on "Create" it will appear this:



600px



Next are detailed the fields that you should fill in:

  • Name: Field to put the correlated alerts name
  • Asigned to: Combo in which you have to select the agent to assign the alert to.
  • Description: Describes the template function, and it is useful to identify the template between others in the Alert General View.
  • Condition:In this section are shown the conditions that the correlated alert should fulfill.

To add conditions you should select one group and one agent. One these have been chosen all the alerts that this agent has will appear.

To add the alert click on the "+" symbol that is on the right of the selected alert.



600px



When two alerts have been chosen, either from the same agent or from different ones, then a combo will appear where you could choose the logical operator that will be used to verify the conditions. The operators that could be selected are the following ones:


  • AND: The two conditions must be fulfilled.
  • NAND:Neither of them must be fulfilled.
  • NOR: At least one condition is not fulfilled.
  • NXOR: Or at least one of the two condition is fulfilled at the same time of neither of them are fulfilled.
  • OR: Debe cumplirse al menos una de las condiciones pudiendo cumplirse las dos.
  • XOR: One or other is fulfilled but not both of them .



600px



Each time that a new condition is added you should choose the logical operator. The checking that Pandora does is a sequential one. In the example that is showed above the two first condition or the third one instead should be fulfield.



600px



Finally the alert will be this:



600px



Once all the conditions have been selected, click on "Next".

One screen like this will appear:



600px



The fields that are established here are the same fields that are used when a "common" alert template is defined.

Once you have filled in all the fields click on the "Finish" button.

1.10.2 Editing Correlated Alerts

It is possible to edit the correlated alerts that have been created from Administration>Manage Alerts >Correlation,



600px



To edit a correlated alert you only need to press on the alert name.

1.10.3 Deactivating Correlated Alerts

Once an alert has been created, it is possible to deactivate it by pressing on the light bulb that is on the right of the alert name.



600px



The alerts that are activated are in blue and the ones that are not activated are in yellow.

1.10.4 Deleting Correlated Alerts

It is possible to delete the correlated alerts that have been created from Administration>Manage Alerts >Correlation



600px



To delete a correlated alert you only need to press on the red cross that is on the right of the alert.

1.11 Cascade Protection

The cascade protection is a Pandora FMS functionality that allows to avoid a "rain" of alerts when a group of agents could not been reached due to a connection fail. This kind of things happens when an intermediate device such as a router or a switch is down and all the devices that comes after it simply finish to be reachables from Pandora FMS. Probably the devices are working correctly, but as Pandora FMS can not see them through ping, then it consider them as downs.



Recursive cascade protection ilustration.png



The cascade protection is activated from the agent configuration menu. Pressing the "cascade protection" box and it is deactivated unchecking this box.



Down1.jpg



When the cascade protection is activated in an agent, then the alerts with father CRITICAL state are checked either this would be simple or correlated. In this way if any father has a critical alert fired, the the alerts configured in the agent will not be fired. These alerts will be fired if the agent father does not have any module in a CRITICAL state or if they are fired by the father with an state lower than CRITICAL. It is understood that the agent will launch the alerts if the required conditions are fullfied.

So as the cascade protection works well, it is convenient to configure in all the fathers an alert with a CRITICAL state that check if the device is down. Besides, in order to avoid that an alert from an agent defined as a father and the other alerts have the CRITICAL state.

1.11.1 Examples

You will have the following monitors:

- ROUTER: a ICMP check and a SNMP check using a Standard OID to get the ATM port status. Also may have a Latency check for your parent/provider router.

- WEB SERVER: you have several internal checks running with the Pandora FMS agent: CPU usage, MEM usage and process check of your Apache. You have also a latency check for a 4-step navigation HTTP check.

- DATABASE SERVER: you have several internal checks running with the Pandora FMS agent: CPU usage, MEM usage and process check of your Database. Also a few database integrity checks. You also check remote connectivity to database using a plugin-defined test to login, make a query and exit, timing the answer.

Now you define several SINGLE alerts:

-ROUTER: ICMP Check / CRITICAL -> Action, send MAIL. SNMP Check / CRITICAL -> Action, send MAIL. Latency > 200ms / WARNING -> Action, none, just compound.

-WEB SERVER

CPU / WARNING -> Action, none, just compound. MEM / WARNING -> Action, none, just compound. PROCESS / CRITICAL -> Action, send MAIL. HTTP LATENCY / WARNING -> Action, none, just compound.

-DATABASE SERVER

CPU / WARNING -> Action, none, just compound. MEM / WARNING -> Action, none, just compound. PROCESS / CRITICAL -> Action, send MAIL. SQL LATENCY / WARNING > Action, send MAIL.

You define ROUTER as parent for DATABASE and WEB servers. You enable the Cascade Protection in both agents (Database and Web).

You now define one correlation alert assigned to DATABASE:

Router ICMP Check NOT Fired AND Router SNMP Check NOT Fired AND WEB Server Process NOT Fired AND Database Server Process Critical

THEN Send MAIL: "Service DOWN: Database Failure" You now define one correlation alert assigned to DATABASE:

Router ICMP Check NOT Fired AND Router SNMP Check NOT Fired AND WEB Server Process Fired AND Database Server Process NOT Fired THEN Send MAIL: "Service DOWN: WebServer Failure"

And more complex alerts like: Router ICMP Check NOT Fired AND Router SNMP Check NOT Fired

AND

WEB Server HTTP Latency NOT Fired

AND

DATABASE Server SQL Latency Fired

AND

DATABASE Server CPU NOT fired

AND

DATABASE Server MEM Fired

THEN

Send MAIL: Database is getting exhausted. Please check it ASAP.

1.12 Special days list

From version 5.0 of Pandora FMS has special days feature. It allows to define holidays and special working days for alert template. Days defined in special days list are treated as the same day of the week you selected.

1.12.1 Creating a special day

The new special days are created clicking on the Create button at Special days list, in the Manage Alerts menu, from the Administration menu.



Creating special day1.png



Once clicked on Create a screen as follows appears.



Creating special day2.png



Next are the fields that you should fill in:

  • Date: Special days date. The data format is YYYY-MM-DD. If you want to define same day every year, you can use '*' for YYYY.
  • Same day of the week: Select a day. The above date is treated as the same as selected day.
  • Description: Special day description.

For example, assume that is a holiday, May 03, 2012. When you define '2012-05-03' with 'Sunday', that day is treated as the same as Sunday.

Once you have filled the fields, click on the Create button.


Template warning.png

To enable special days list, "Use special days list" should be set on the alert template (step 2)

 


1.12.2 Editing a special day

It is possible to edit the special days created from option Special days list at the Manage Alerts menu of Administration menu.



Editing special day1.png



To edit a special day just click on the date.



Editing special day2.png



Once these changes have been done, click on the "Update" button.

1.12.3 Deleting a special day

In order to delete a special day, click on the red cross located at the right hand side of the day.



Deleting special day.png



Go back to Pandora FMS documentation index