Pandora: Documentation en: SNMP traps Monitoring

From Pandora FMS Wiki
Revision as of 16:00, 18 June 2012 by Slerena (talk | contribs) (Data visualization)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Go back to Pandora FMS documentation index

1 Operation with SNMP traps

1.1 Introduction

Pandora FMS has a trap reception console that allows to display traps sent by monitored objects and add alerts to such traps. SNMP traps are recieved through the operating system demon that Pandora FMS's SNMP server starts when Pandora's server starts. This server, usually stores traps in a logfile /var/log/pandora/pandora_snmptrap.log.

Traps are usually recieved in "raw" format, that means, with numeric OID's, unless a MIB installed in the Operating System is capable of solving them. Pandora FMS Enterprise's SNMP console, enables rule creation for renaming numeric OID's to alphanumeric OID's or simple descriptive text strings (e.g.: Interface is down) in order to make working with TRAPS more intuitive. Pandora FMS allows to load Trap MIB's of any manufacturer in order to automatically define such rules.

1.2 Access to trap reception console

To go to the trap reception console go to Operation > SNMP console where is the list of traps that have been received. There is an icon( an eye) that allows to display all the trap information, same as with the events. Here we could see the detailed info of a SNMP trap.

Snmp trap sample.png

For each trap following columns are displayed:


Green box if trap is validated or red one if not so.

SNMP Agent

Agent which sent the trap. An address of the agent is obtained from the agent-addr field of the protocol data unit (PDU) if sent trap is SNMPv1. Otherwise, a source address of the packet is used.


Sent trap's OID. If sent trap is SNMPv1, the enterprise field of the PDU is used. Otherwise, the snmpTrapOID filed is used.


Value field of sent trap. A trap can only send a data in this field. Not used in SNMPv2 Trap.

Custom OID, Custom Value

Customized fields sent in the trap. They could be very complex data, that have an specific logic depending on the device that send the trap. A trap can sent several data in these field.

Time Stamp

Time elapsed since trap reception.


Yellow box if any alert was launched with this trap or grey one if no alert was launched.


Field for deleting or validating the trap.

On top of that, traps have different colors depending on the trap type.

  • Blue: Manteinance traps.
  • Lila: Information traps.
  • Green: Normal traps.
  • Yellow: Warning traps.
  • Red: Critical traps.

1.2.1 Trap Filtering

In the upper part of the trap console option “Toogle Filter” is displayed. Clicking on that option trap filtering fields appear or dissappear.


It is feasible in the trap console to filter by following fields:

  • Agent: Combobox where Pandora agentes are displayed.
  • OID: Combobox where OIDs are displayed.
  • Alert: Combobox to select either triggered or non-triggered alerts.
  • Search value: Combobox to freely input any text.
  • Severity: Combobox where the different trap types are displayed: Maintance, Information, Severity, Warning y Critical.
  • Status: Combobox to select between alert SNMP validated, not validated or all.
  • Free search: search by any alphanumeric field in the trap.
  • Type: type filter in SNMP alerts. Could be: Cold start, Warm start, Link down, Link up, Authentication failure or Other.

On top of these search fields there is the option “Block size of pagination”, that allows to define the ammount of traps to be displayed per page.

1.2.2 Trap Validation

In order to effectively manage the traps, it is possible to validate them, so the administrator can discriminate between the traps she's already seen and the pending ones.

In order to validate a trap, click on the green circle at the left of the trap.


It is also possible to validate multiple traps marking them all and clicking on the green “validate” button.


1.2.3 Trap Deletion

It is possible to delete traps once they have been treated.

To delete a trap, click on the red cross at the left of the trap.


1.3 SNMP Trap Alerts

It's possible to associate an alert to a trap, so Pandora FMS warns us on arrival of a specific trap. To manage alerts associated to traps, go to Administration>Manage SNMP Console. SNMP traps have got nothing to do with the rest of the system alerts, even if both reuse the actions' system.

1.3.1 Alert creation

To add an alert associated to a trap go to Administration>Manage SNMP Console and click on “Create”.

The SNMP alert traps have several fields that could be used to "search" data in the SNMP trap. The fields that could be used, both separately and in combination are:

  • Description: Combox for an alert's description.
  • OID: main Trap OID.It's a regulara expresion. If a regular expression is not used, it'll look for the exact string. If you look for a piece of the OID, a regular expression should be used, so if we want to search, for example in a larger OID, we could use the regular expression .**
  • Custom Value/OID: This search in the trap "Value" fields, and also in the fields "Custom OID" and "Custom Value", that is, in the rest of the TRAP fields. Here it works, the same, the substring "Testing TRAP" with the regular expression "Testing.*TRAP.*"
  • SNMP Agent: IP of the agent that send the trap. Same way, we can use a regular expression or a substring.
  • Trap type: Filter by trap type: Cold start, Warm start, Link down, Link up, Authentication failure o Other.
  • Single value: Filter by trap value. In this example is .666
  • Custom OID/Data #1: It's a regular expression. It is possible to use selectors in order to select a part of the regular expression that will be load in _snmp_f1_ macro. You can use this macro in Field #1 (Alias, name), Field #2 (Single Line) and Field #3 (Full Text).
  • Custom OID/Data #2: It's a regular expression. It is possible to use selectors in order to select a part of the regular expression that will be load in _snmp_f2_ macro. You can use this macro in Field #1 (Alias, name), Field #2 (Single Line) and Field #3 (Full Text).
  • Custom OID/Data #3: It's a regular expression. It is possible to use selectors in order to select a part of the regular expression that will be load in _snmp_f3_ macro. You can use this macro in Field #1 (Alias, name), Field #2 (Single Line) and Field #3 (Full Text).
  • Custom OID/Data #4: It's a regular expression. It is possible to use selectors in order to select a part of the regular expression that will be load in _snmp_f4_ macro. You can use this macro in Field #1 (Alias, name), Field #2 (Single Line) and Field #3 (Full Text).
  • Custom OID/Data #5: It's a regular expression. It is possible to use selectors in order to select a part of the regular expression that will be load in _snmp_f5_ macro. You can use this macro in Field #1 (Alias, name), Field #2 (Single Line) and Field #3 (Full Text).
  • Custom OID/Data #6: It's a regular expression. It is possible to use selectors in order to select a part of the regular expression that will be load in _snmp_f6_ macro. You can use this macro in Field #1 (Alias, name), Field #2 (Single Line) and Field #3 (Full Text).
  • Field 1: Field 1 to set the parameter of the alarm's command.
  • Field 2: Field 2 to set the parameter of the alarm's command.
  • Field 3: Field 3 to set the parameter of the alarm's command.
  • Min. Number of Alerts: Field to define the minimal amount of traps that have to arrive for the alarm to be triggered.
  • Max. Number of Alerts: Field to define the maximal amount of times the action will be executed.
  • Time Threshold: Field for defining the time to elapse before resetting the alarm counter. This counter is the one used for field Min. Number of alerts.
  • Priority: Combobox for establishing the alarm priority.
  • Alert Action: Combobox for selecting the action that will execute the alert.If you select an event, the normal event of alert creation will be not created.

Snmp trap alert.png

In this situation, when an alert is defined using an event and the field 1 to generate the information, the result would be an event like the following:

Snmp trap alert event.png

Once the fields are filled in, click on “Create”

1.3.2 Alert Edition

To edit an alert associated with a trap, go to Administration>Manage SNMP Console, select the alert you want to edit and click on the tool icon.


A page with the alert's configuration is shown. Fields to be changed are changed, then click on Update

1.3.3 Alert Deletion

To delete an alert associated with a trap go to Administration>Manage SNMP Console, select the chosen alert, then click on the red X.


1.4 Customising SNMP Traps

In order to make the operator understand better the traps sent by the monitored devices, it is possible to either load the manufacturer's MIBs to Pandora FMS or edit the traps as it likes.


These are Pandora FMS Enterprise features


1.4.1 Renaming/Customising Traps

"Editing traps" is a process where the operator is allowed to "customise" the appearance a trap has in the console.

If you take a look at the following screenshot, it could happen that all of your traps are difficult to distinguish due to the mess received as the data they contain. Could we rename them, so we could identify them as "Blue box" instead of a huge amount of senseless encrypted data?

To edit a trap, go to Operation > SNMP Console and click on the OID field of the chosen trap.


A page like the following will appear:


This way, when the SNMP Console finds traps with OID ".", it will display them as "Blue box sample" and it will be way more readable by the human eye. It's content (included the original OID) won't be changed at all.

Keep in mind that all the older traps won't change their appearance. This feature will start working only with new incoming traps in the system from now on.

From the same place, new traps can be created from zero with the "create trap" option. For this, you only have to go to the trap editor, as shown in this screenshot. In that section we can alter or remove a trap definition.

Snmp trap editor.png

Finally, this is how a user redefined trap would be seen:

Snmp custom view.png

1.4.2 Load the manufacturer's MIBs


These are Pandora FMS Enterprise features


This option is useful to upload trap MIBS (only) and do the internal translating Pandora FMS database bigger, so when a trap comes, it will be automatically translated by its description.

To upload the manufacturer MIBs, click on "Examine", choose the file that should be with txt extension and click on “Upload MIB”.


Once it has been uploaded, the system will incorporate it to its trap library.

1.5 Alerts with complex SNMP traps

The alerts described previously are used only where the trap is correctly defined, is always the same, and it doesn't have a relevant data to recover.

However, in certain situations, we might find a trap with the following structure:

OID: . Value: 666 Custom data: = STRING: "ID-00342" . = STRING: "Automated check" . = STRING: "NIC Offline" . = STRING: "4897584AH/345"

This is a "complex" trap which, besides a OID and a value, contains complex data, based in many other OIDs and values. A trap can contain, in its complex part, a completely random structure, based in OID/value pairs (counters, numeric, alphanumeric, dates...).

This trap would be seen in the trap console this way:

Compex trap def3.png

If we pay close attention to the extended info (Custom data), there are some pieces of useful data. For instance, in the first field, with the OID ending in 2005.1, that looks like an identifier. The third field with OID ending in 2005.3 looks like an error message. Fields 2 and 4 don't look pretty useful, since they seem to be an unknown code for us.

Let's think we can create an event from a trap, placing specific parts of the trap data in the text. For instance, let's suppose we want to build an event containing the following information:

Chassis Alert: <error message> in device <identifier>

The challenge is to make an alert which does a "match" in those fields, obtaining the piece of data and using it later to create a message in the alert. We can do this with Pandora FMS, using advanced regular expressions, using selectors. You can find more info about regular expressions here: [1].

The selectors, using (), allow us to "copy" information, using a search expression.

The regular expression to obtain fields 1 and 3 would be the following:

.*. \= STRING\: \"([0-9\-\_A-Za-z]+)\".*. \= STRING\: \"([\sA-Za-z]+)\".*

Once we've got the data fields, we must use them in the alert. With this purpose, the special macros _snmp_f1_, _snmp_f2_ and _snmp_f3_ are used. Using these macros doesn't have any sense out of SNMP trap alerts.

To build the message, we would use the following string:

Chassis Alert: _snmp_f2_ in device _snmp_f1_

In resume, this is how the complete alert would be created:

Compex trap def1.png

And this is how it would be seen in the event viewer.

Compex trap def2.png

In order to successfully make this kind of alerts, it is needed a good knowledge of regular expressions, since a simple blank space, a symbol or another character in the wrong place could make it work wrong. Always remember the SNMP alerts implies the use of regular expressions. The way to establish an easier alert, using regular expressions, would be the following:

Compex trap def4.png

This alert would be "Compatible" with the previous one in a way that, for the same trap, two different events could be displayed:

Compex trap def5.png

1.6 Trap association to the rest of Pandora alerts / SNMP Agent trap forwarding

The alerts defined on traps are completely independant from Pandora's alert engine, so correlations of kind “trigger an alarma if temperature reaches 29 degrees and the trap for secondary power supply is on” cannot be established. This kind of alerts can neither be displayed (since they are -eventually- not associated to any Pandora FMS module, and thus, trap console monitoring cannot be related to elements such as reports or maps.

Special ModuleModule SNMPTrap,with the trap sent from the SNMP console: Snmptrap agent.png

In order to achieve this, a method called "Agent SNMP Trap Forwarding" exists. This (server wide) option forwards the trap to a special agent's module named "SNMPTrap" as a text string, if and only if, the trap's origin IP address is defined as agent IP. Whenever this occurs, the trap arrives as a text line to the agent inside that module, which is a module that's only defined at arrival of the first trap.

Text alerts can be specified on that module, being these completely standard, just as any other module. This enables for customization of SNMP monitoring in order for certain traps from certain origins to be treated as yet another module, and thereby integrate it in the rest of the monitoring, including alert correlation.

SNMPTrap data sample': Snmptrapforward2.png

This is a Enterprise feature, and could be activated in the main setup screen, as shown here:

Configuration option to activate the trap forward to the agents:
Snmptrap agent forwardsetup.png

If this setting is changed, Pandora FMS server needs to be restarted to enable it.

Another solution is mounting an alert on the trap to activate an agent's module. For example, be the trap a "1" written in certain logfile, and there is an agent reading that file ready to run when it finds a "1". This way, the module will be triggered when the desired trap arrives and the correlation can be established basing on the arrived trap.

1.7 Trap Filtering in the Server

Some systems gets a high number of traps. From this ones, we are interested only at monitoring a tiny percentage. From the Pandora FMS 3.2. version it's possible to filter the traps that the server gets in order to avoid loading the application in an unnecessary way.

From Administration>Manage SNMP Console>SNMP Filters, it's possible to define different filters. One trap that goes with any of them, it will be only automatically ruled out for the server.

Snmp filter editor.png

The filter is applied as a regular expression over the corresponding entry of the trap in the SNMP log (by default /var/log/pandora/pandora_snmptrap.log), that has the following format:

SNMPv1 Trap:


SNMPv2 Trap:



  •  %y: Current year.
  •  %m: Current month (numerical).
  •  %l: Day of the current month.
  •  %h: Current hour.
  •  %j: Current minute.
  •  %k: Current second.
  •  %a: Origin address (only traps from version 1).
  •  %b: Origin address (source address of the packet)
  •  %N: OID.
  •  %w: Kind of trap (numerical).
  •  %W: Trap description.
  •  %q: Trap's Sub-kind (numerical)
  •  %v: variable list in tab separated format (custom OID).

Note that if trap is SNMPv2, trap OID is contained in the %v as 2nd parameter, and 3rd and following are parameters actual part of the variable list (custom OID).

For example, to filter all the sent traps by host we could define the following filter:

Snmp filter example.png

1.8 External SNMP Trap Handler

The SNMP console is only to get traps, so it only processes TRAP as individual item, but one trap can have lot of information. Sometimes, it happens that the only monitoring we could do is based on traps. For doing it, we could choose to "post processing" the information that is got in one trap through an external script, that works as a plugin.

To process the data of one trap in detail, you can send all the information of one trap to an script, as a result of an alert. I have used this trap for the example. It's the trap view as it would be in the Pandora's SNMP console log:

2010-08-26 12:01:46 pandora . . 233 . = STRING: AIX_Software_Failure . = STRING: 08 25 2010 08:23:43:697685 . = STRING: 1: A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED. . = STRING: 8 . = STRING: An application may not work properly . = STRING: An application may not work properly . = INTEGER: 4 . = OID: .

Snmp plugin1.png

Snmp plugin2.png

In the screenshots you can see how an special alert would be created. It executes an script with the complete contents of the trap (_data_) and how the SNMP alert kind is created. In this case it has been mapped for the Specific OID (. but it could have been more general, for example (. to call the script when there would be any kinds of traps like these (., I suppose it would be part of the AIX specific MIB)

An script that processes these data is executed. It also "analyzes" the trap to write data in Pandora FMS directly, creating an XML and putting it at /var/spool/pandora/data_in as data, as it would come from an agent. A basic script for this case would be, for example, to generate complex information, so we already have enough information in this trap, that is:

  • Origin IP
  • Main Event (Cold start)
  • Secondary Events (descriptives): AIX_Software_Failure, 1: A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly, An application may not work properly.

When designing an script that "parses" each of these data, for example "" and that store at /var/spool/pandora/data_in in the XML with a generic name plus one random number,

The generated XML should be like this:

<?xml version='1.0' encoding='ISO-8859-1'?>
<agent_data description='' group='' os_name='aix' os_version='' interval='300' version='3.1(Build 100608)' timestamp='2010/08/26 12:20:26' agent_name=''>
      <data><value><![CDATA[A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC.]]></value></data>
      <data><value><![CDATA[Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly, An application may not work properly.]]></value></data>

The application is endless, but, any script should be customized, so it would have a very dynamic structure. In lot of systems the information that is get is not only text, but also numerical, so it could feed numerical information modules in order to could represent graphics, etc. Please, consider that all data is always asynchronous.

1.8.1 Practical Sample: ESX Monitoring using traps

One of the most problematic things to monitor is "distributed" infraestructure, more if each version changes it's implementation to gather information, like VmWare ESX. In this small chapter we try to explain how to monitor ESX systems using an external SNMP Trap handler.

ESX traps are like this:

. = STRING: "host" . = STRING: "c7000-06-01.tsm.inet" . = "" . = STRING: "Green" . = STRING: "Yellow" . = STRING: "Host cpu usage - Metric Usage = 1%"

. = STRING: "host" . = STRING: "dl360-00.tsm.inet" . = "" . = STRING: "Yellow". = STRING: "Green" . = STRING: "Host memory usage - Metric Usage = 84%"

. = STRING: "host" . = "" . = "" . = STRING: "Red" . = STRING: "Green" . = STRING: "Datastore usage on disk - Metric Storage space actually used = 55%"

As you can see, traps could be used to gather information from CPU, Disk or Memory. The global idea behind the trap handler is to write a small script able to "understand" the trap and create a XML simulating a software agent. So for each technology you should write a trap handler, but all the process is common. The process to understand this is explained in four steps.

  1. Create the handler script. You can base your work on the script provided below.
  2. Create an alert command.
  3. Create an alert action using previous command, sometimes with custom options for each "destination" agent you want (if you have several farms of ESX; you will like to have data in different agents.
  4. Create a SNMP Trap alert which maps the enterprise OID (information trap contains for all kind of this specific technology) and / or the source trap IP address.

Let's see the first step: Creating the trap handler script: Trap handler:

# (c) Sancho Lerena 2010 <[email protected]>
# Specific Pandora FMS trap collector for ESX 

use POSIX qw(setsid strftime);

sub show_help {
	print "\nSpecific Pandora FMS trap collector for ESX\n";
	print "(c) Sancho Lerena 2010 <[email protected]>\n";
	print "Usage:\n\n";
	print " <destination_agent_name> <TRAP DATA>\n\n";

sub writexml {
	my ($hostname, $xmlmessage ) = @_;
	my $file = "/var/spool/pandora/data_in/$hostname.".rand(1000).".data";

	open (FILE, ">> $file") or die "[FATAL] Cannot write to XML '$file'";
	print FILE $xmlmessage;
	close (FILE);

if ($#ARGV == -1){

$chunk = "";

# First parameter is always destination host for virtual server
$target_host = $ARGV[0];

foreach $argnum (1 .. $#ARGV) {
	if ($chunk ne ""){
		$chunk .= " ";
	$chunk .= $ARGV[$argnum];

my $hostname = "";
my $now = strftime ("%Y-%m-%d %H:%M:%S", localtime());
my $xmldata = "<agent_data agent_name='$target_host' timestamp='$now' version='1.0' os='Other' os_version='ESX_Collectordime ' interval='9999999999'>";

if ($chunk =~ m/. \= STRING\: ([A-Za-z0-9\-\.]*)\s\.1/){
	$hostname = "_".$1;

if ($chunk =~ m/Host cpu usage \- Metric Usage \= ([0-9]*)\z/){
	$value = $1;
	$module_name = "CPU_OCUPADA$hostname";

if ($chunk =~ m/Host memory usage \- Metric Usage = ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "MEMORIA_OCUPADA$hostname";

if ($chunk =~ m/Datastore usage on disk \- Metric Storage space actually used \= ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "DISCO_OCUPADO$hostname";

$xmldata .= "<module><name>$module_name</name><type>async_data</type><data>$value</data></module>\n";

$xmldata .= "</agent_data>\n";
writexml ($target_host, $xmldata); Step 2: Create the alert command

In this example, I've put the command script in /tmp, put on a safer place, and be sure it's executable (chmod 755):

Trap handler step2.png Step 3: Create the alert action

Create specific action to send all information to a specific agent traps. In this case information will be sent an agent named WINN1247VSR. The above command accepts as parameters the name of the agent that will go all the information (ESX Virtual Center), and "chunk" of data from the TRAP, which can be unlimited and includes all the information you send the trap.

Trap handler step3.png Step 4: Create the SNMP alert

Set alert traps using the action you have just created.

Trap handler step4.png

In order to process all the traps of ESX Tecnology, this will find, using the specific OID . to map ESX traps. We may also filter by source IP for each VirtualCenter, by filtering for IP address origin (send in the trap). Data visualization

This is a sample on how information will see. With this data, you can manage as standard modules.

Trap handler step5.png

Go back to Pandora FMS documentation index