Pandora: Documentation en: SNMP traps Monitoring

From Pandora FMS Wiki
Jump to: navigation, search

Go back to Pandora FMS documentation index

1 Operations with SNMP Traps

1.1 Introduction

Network devices that support SNMP, such as switches, routers, servers, printers, or APs can send alarms (or SNMP traps) when certain events take place, such as interface failure, too hight CPU or network load, a UPS changing status or a filled up disk partition. Each device has its own "collection" of possible events, which is reflected in a collection, called MIB, in this case, different from the MIB used to query the device.

Trap-example.png

Pandora FMS features a TRAP Reception Console which allows TRAPs that were sent by monitored objects to be displayed and add alerts to said traps. SNMP traps are received by the operating system's daemon which starts the SNMP Server of Pandora FMS at the same time of the server startup. This server stores received traps in a log file under

/var/log/pandora /pandora_snmptrap.log

Traps are usually received in RAW format. Regarding numerical OIDs, that means that unless an MIB is installed on the Operating System that is capable of resolving them, Pandora FMS Enterprise SNMP Console enables rule creation for renaming numeric OIDs to alphanumeric ones or simple descriptive text strings (e.g. 'interface is down'), in order to make working with TRAPs more intuitive. Pandora FMS allows loading trap MIBs of any sort in order to automatically define such rules.

First, modify the following parameter in /etc/pandora/pandora_server.conf to enable the SNMP Console:

snmpconsole 1

If you want the traps to be translated (either the variable bindings or the Enterprise string), please enable the following options too (only Enterprise):

translate_variable_bindings 1
translate_enterprise_strings 1

Also, set up the /etc/snmp/snmptrapd.conf file with the appropriate parameters. For instance:

authCommunity log public
disableAuthorization yes

As for this configuration, traps will be created for the "public" community and will not need any authorization.

1.1.1 SNMPv3

SNMPv3 traps are rejected unless the sending user is added to /etc/snmp/snmptrapd.conf using the createUser directive. For example:

disableAuthorization yes
createUser -e 0x0102030405 snmpv3user SHA mypassword AES

Template warning.png

The engineID must be specified with the -e option. Otherwise, only SNMPv3 INFORMs will be received.

 


1.2 Access to TRAP Reception Console

To use the trap reception console, go to Monitoring > SNMP > SNMP Console, where you may take a look at the list of TRAPs which have been received so far. There is an eye-shaped icon which displays all the trap information. You can learn any detailed information regarding SNMP traps there.


Traps.JPG


For each trap, the following columns will be displayed:

Status

A green box appears if the trap is validated. It turns red if not.

SNMP Agent

The Agent which has sent the trap.

OID

The OID of the sent trap. A trap can only send one piece of data in this field.

Value

The value field of the sent trap. A trap can only send one piece of data in this field.

Custom OID, Custom Value

The customized fields, sent within the trap. They sometimes consist of very complex data which bears a specific logic, which sends the trap. A trap is able to send several types of data in this field.

Time Stamp

The elapsed time since the trap reception.

Alert

A yellow box appears if any alert was launched by this trap. It is gray if no alert was launched.

Action

The field for deleting or validating the trap.

Traps also bear the following colors, depending on the type:

  • Blue: Maintenance traps.
  • Purple: Information traps.
  • Green: 'normal' traps.
  • Yellow: 'warning' traps.
  • Red: 'critical' traps.

In the upper part of the trap console, the option named 'Toggle Filter' is displayed. By clicking on that option, the trap's filtering field options appear or disappear.


Alert.png


1.2.1 TRAP Validation

In order to effectively manage the traps it is possible to validate them, so the Administrator is able to distinguish between pending traps and the ones already seen.

To validate a trap, click on the green circle to the left of the trap. It is also possible to validate multiple traps by marking them and clicking on the "validate" button. The procedure is similar to that of events in Pandora FMS.


Traps2.JPG


1.2.2 TRAP Deletion

It is possible to delete traps once they have been edited, either individually or by multiple selection and the "Delete"action. To prevent traps from accumulating, there is a default setting option that automatically deletes traps older than 10 days.


Traps3.JPG


1.3 SNMP Trap Alerts

1.3.1 Introduction

Pandora FMS also has an alert system for the SNMP traps that it receives. They are mainly based on filtering rules, searching for matches in all possible fields according to rules that are set up to trigger the alert. Before reading on, bear in mind you can find out more about Pandora FMS alerts [1]

1.3.2 Alert Creation

SNMP traps alerts have several fields that will be used to look for matches in SNMP traps received in the console. You may optionally use any fields you want to create more general or more specific rules as required:


AlertSNMP1.png


  • Description: A combo box for an alert description.
  • Enterprise String: The main OID of the trap. It will look for the presence of the string. For example, if you are looking for a piece of the OID, you may use 1.21.34.2.3 and every OID that contains that one will be filtered, as if it were *1.21.34.2.3.* But there is NO need to use the * character.
  • Custom Value / OID: This element will search within the trap's 'value', 'custom OID', 'custom value' and in the rest of the TRAP fields. Regular expressions are supported here. For example, if you have a trap that sends the "Testing TRAP 225" string, you can search for any trap with the subchain "Testing TRAP" through the regular expression "Testing. *TRAP.*"
  • SNMP Agent: The IP of the agent which has sent the trap. You may use a regular expression or a substring here too.
  • Trap type: The filter by trap type. The accepted values are 'cold start', 'warm start', 'link down', 'link up', 'authentication failure' or 'other'. Most of the generated traps are usually "Other" type. If you do not specify anything, it will look for any type of trap.
  • Single value: It filters by trap value. In this example, it is .666. Keep in mind this refers only to the MAIN OID value, not to any additional OIDs within the custom data section.
  • Variable bindings/Data #1-20: These are regular expressions which try to match the binding variables from '1' to '20'. If there is a match, the alert is triggered. The value of the variable is stored in the corresponding '_snmp_fx_' macro (e.g. _snmp_f1_, _snmp_f2_, etc.). Although only twenty binding variables are able to search for matches, the _snmp_fx_ macros are set for all of them (_snmp_f11_, _snmp_f12_, etc.).


AlertSNMP2.png


  • Field 1: Field to set the Field 1 alarm command parameter. This is the field that will be used in case of choosing to generate an event, or the destination mail in case of choosing an email action (if you wish to overwrite the default email in the action). To fully understand how custom fields work in actions/alerts templates, read the documentation chapter that explains the alerts in Pandora FMS [2]
  • Field 2: Field to set the command parameter of the Field 2 alarm. In the case of sending an email, it will be the subject of the message. If left blank, it would use what it had defined in the action.
  • Field 3: Field to set the command parameter of the Field 3 alarm. In the case of sending an email, it would be the text of the message. If left blank, it would use what it had defined in the action.
  • Min. Number of Alerts: The field to define the minimum amount of traps that must be received to trigger the alarm.
  • Max. Number of Alerts: Field where the maximum number of times the action will be executed in the given interval (or time threshold) is.
  • Time Threshold: The field for determining the time to elapse before resetting the alarm counter. This counter is the one which gets used for the field named 'min. number of alerts'.
  • Priority: Combo where the alarm priority is set. The priorities of the alerts are different and have nothing to do with the priority of the traps, nor with the Pandora FMS events.
  • Alert Action: The combo box for selecting the action that will execute the alert. If you select an event, the normal event of an alert creation will not be created.
  • Position: The alerts with a lower position are evaluated first. If several alerts with the same position match a trap, all alerts matching the same position will be triggered. Although lower position alerts may match the trap, they will not be triggered.

1.3.3 Alert Field Macros

The following macros can be used in any of the alert fields:

  • _data_: Full trap
  • _agent_: Agent name
  • _address_: IP Address
  • _timestamp_: Trap date
  • _snmp_oid_: Trap OID
  • _snmp_value_: Trap OID value

1.3.4 Trap Alert Example

Let us suppose you get a trap like the next one:

Trap sample for alert.jpg

In this case, you have a main OID (. 1.3.6.1.4.1.2789.2005) that identifies a trap that can contain CPU overheating messages (you do not know if anything else), but what you do know is that in two variables it shows that the CPU heats up, and the temperature of that CPU at that time in variables 1 and 2 respectively. Since you want to identify only the CPU heating traps, you match the Heat alert string in the first variable of the trap (remember that you have up to 20 for searching).

Defining the first part of the trap is simple, just use the main OID to make the first and most important pre-filter:

Trap alert definition 1.jpg

The second part of the trap definition is the one containing the essential part. In the first variable of the trap, look for the "Heat alert" string. If traps arrive with the main OID but they do not contain that text string in the first variable, the alert will not be triggered.

Trap alert definition 2.jpg

And finally, by choosing a "Pandora Event" type alert, conform the message using the variables that contain the value of variables 1 and 2 of the trap received:

Trap alert definition 3.jpg

This way, when the alert goes off, the generated event will look like this:

Event result of alert.jpg

1.4 Working with environments with many traps

1.4.1 TRAP-Storm Protection

There are a couple of parameters in the server which are conceived to protect the system against the arrival of a Trap Storm, coming from a single location. Use the following settings in the 'pandora_server.conf' file for this:

  • snmp_storm_protection: The max. number of processed SNMP traps by the same source IP in a given interval (see below).
  • snmp_storm_timeout:: The interval in seconds for protection against an SNMP Trap Storm. During this interval, the system will only process 'snmp_storm_protection' type traps from the same source (IP).

Trap storm protection combined with trap filtering (see below) allows that if you receive hundreds of thousands of traps per day, you work with only a few thousand traps to delete redundant or unhelpful traps.

1.4.2 TRAP Filtering in the Server

Some systems receive a large number of traps from which only a small percentage is relevant to monitor. With Pandora FMS, it is possible to filter the traps that the server receives in order to avoid unnecessary application loading.

From Administration>Manage SNMP Console>SNMP Filters you can define different filters. A trap that matches any of them will be automatically discarded by the server:

Snmp filter editor new.png

The filter is applied as a regular expression on the trap entry in the SNMP log (default /var/log/pandora/pandora_snmptrap. log), which has the following fixed format:

%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n

Them being:

  •  %y: current year.
  •  %m: current month (numerical).
  •  %l: current day of the month.
  •  %h: current hour.
  •  %j: current minute.
  •  %k: current second.
  •  %a: Origin address (traps version 1 only).
  •  %N: OID.
  •  %w: trap type (numerical).
  •  %W: trap description.
  •  %q: trap sub-type (numerical).
  •  %v: List of variables separated by tab (custom OID).

For example, to filter all the traps from 192.168.50.20 host set the following filter:

Snmp filter example.png

Since more than one filter can be created simultaneously, the search will take into account those traps that meet all filtering conditions.

1.4.3 SNMP Trap stats

This view allows you to see trap statistics, both by origin (IP) and by OID, this allows an effective filter management, by identifying the IPs that generate more traps and the OIDs that are more repeated. The system displays trap statistics for the last 30 days.

Captura de pantalla 2014-09-23 a la(s) 17.59.56.png

Captura de pantalla 2014-09-23 a la(s) 19.11.40.png

1.5 Customizing SNMP Traps

In order to give the operator a better understanding of the traps sent by the monitored devices, it is possible to either load the manufacturer's MIBs into Pandora FMS or to edit the traps to your liking.


Info.png

The functions mentioned so far are all features of the Pandora FMS Enterprise Version.

 


1.5.1 Renaming and Customizing Traps

The process is called "edit a trap", where it is allowed to "customize" the appearance of a trap in the console. If you look at the next capture, all your traps may be difficult to distinguish because of the cryptic information they contain.

To edit a trap, go to Operation > SNMP Console. Click the edit icon for the trap you want to customize:


Traps7.JPG


That way, when you find traps with the ". 1.3.6.6.1.4.1.2789.2005" OID, you will see them as "Blue box sample". And it will be easier to tell them apart. Its content (including the original OID) will remain the same.

"Custom OID" is a Perl compatible regular expression that will be matched against the part of the trap string that contains variable bindings. Generally there is no need to translate a trap.

Template warning.png

"Custom OID" is not meant to be the whole variable binding string, which can be larger than its supported maximum length, but rather a regular expression that matches one or more variables.

 


Keep in mind that all previous traps will not change their appearance, this will start working with the new traps that enter the system from that moment onwards.

Finally, this is what a user-defined trap would look like:


Traps8.JPG


1.5.2 Loading Manufacturer MIBs

This option is suitable for uploading trap MIBS (exclusively) and enlarging Pandora FMS internal translation database, so that when a trap arrives, it will be automatically translated by its description.

To upload the Manufacturer's MIBs, click on "Browse", choose the file that must have a txt extension and click on "Upload MIB".

Cima6.png

Once uploaded, the system will add it to its trap library.

1.6 Associated Alerts to complex SNMP Traps

The previously described alerts will only going be used where the trap is appropriately defined. It is always the same and it does not bear any relevant data to recover.

In certain situations, you might however find a trap which presents the following structure:

OID: .1.3.6.1.4.1.2789.2005
Value: 666
Custom data: 1.3.6.1.4.1.2789.2005.1 = STRING: "ID-00342" .1.3.6.1.4.1.2789.2005.2 = STRING: "Automated check"  
.1.3.6.1.4.1.2789.2005.3 = STRING: "NIC Offline" .1.3.6.1.4.1.2789.2005.4 = STRING: "4897584AH/345"

This is a 'complex' trap which contains complex data alongside an OID and a value, based on many other OIDs and values. In its complex part, a trap can contain a completely randomized structure which is based on OIDs or value pairs (e.g. counters, numerical, alphanumerical, dates, etc.).

Within the trap console, this trap would appear as in the picture below:


Traps9.JPG


If you carefully read the extended information (Custom data), there might be some pieces of useful data. In the first field, containing the '2005.1' OID ending looks like an identifier. The third field, containing the '2005.3' OID ending looks like an error message. Fields 2 and 4 do not look useful, since they seem to be of unknown code.

Let us assume for a moment you could create an event from a trap, moving specific parts from the trap data into the text. And let us suppose you intend to build an event which contains the following information:

Chassis Alert: <error message> in device <identifier>

The challenge is to make an alert which generates a 'match' in those fields, obtaining the piece of data and using it to create a message in the alert later. This task can be performed using Pandora FMS and advanced regular expressions and selectors. You may find more information about regular expressions here.

The selectors use brackets (), allowing you to 'copy' information by using a research expression.

The regular expression to obtain the identifier would be like the following one:

.*.1.3.6.1.4.1.2789.2005.1 \= STRING\: \"([0-9\-\_A-Za-z]+)\"

The regular expression to obtain the error message would be something like this:

.*.1.3.6.1.4.1.2789.2005.3 \= STRING\: \"([\sA-Za-z]+)\".*

Once you have obtained the data fields, use them in the alert. For this purpose, the '_snmp_f1_', '_snmp_f2_' and '_snmp_f3_' special macros are used. However, using these macros outside any SNMP trap alert does not make any sense.

To build the message, use the following string:

Chassis Alert: _snmp_f2_ in device _snmp_f1_

The picture below shows how the complete alert is created.



Compex trap def1.png




In order to successfully create this type of alert, an extensive knowledge of regular expressions is required, since a simple blank space, a symbol or another character in the wrong location could make it malfunction. Keep in mind that SNMP alerts imply the use of regular expressions. An easier way to establish an alert by regular expressions would be the following one:



Compex trap def4.png




1.6.1 Additional Example

This additional example uses an email alert to send information about the interface name each time you receive a specific trap. You will receive an email containing a device name, IP and interface name as received in the trap.

This is a trap, received from a switch:



Another trap alert sample.png




This is the received mail.



Trap email.png



This is the trap definition.



Another snmp trap sample.png





Another snmp trap sample2.png



1.7 TRAP Association to the Rest of Pandora FMS Alerts and SNMP Agent Trap Forwarding

The alerts defined on traps are completely independent from Pandora FMS Alert Engine, so their correlations trigger an alert if the temperature reaches 29 degrees and the trap for secondary power supply is on. This type of alerts cannot be displayed since they are -possibly- not associated to any Pandora FMS modules. Therefore, trap console monitoring cannot be related to elements such as reports or maps.



Special Module SNMP Trap, with the trap sent from the SNMP console: Snmptrap agent.png

In order to achieve this, a method called 'Agent SNMP Trap Forwarding' has been created. This server-wide option forwards the trap to a special agent's module named 'SNMPTrap' as a text string, but only if the trap's source IP address is defined as the agent's IP. Whenever this occurs, the trap arrives as a text line to the agent within that module, which is one that is only defined on the arrival of the first trap.

Text alerts can be specified within that module, these being completely standard, just as any other module. This enables SNMP Monitoring customization in order for certain traps from certain origins to be treated as yet another type of module, which are thereby integrated into the rest of the monitoring, including Alert Correlation.



SNMPTrap data sample': Snmptrapforward2.png

This is an Enterprise feature which could be activated on the main setup screen as shown below.



Configuration option to activate trap forwarding to the agents:
Snmptrap agent forwardsetup.png

If this setting is modified, the Pandora FMS Server must be restarted to enable it.

Another solution is to create an alert on the trap to activate an agent's module. If the trap has '1' in a certain log file, there is an agent reading that file, ready to run it when it finds this '1'. That way, the module will be triggered if the desired trap arrives and the correlation can be established, based on the arrived trap.

1.8 External SNMP Trap Handler

The SNMP console is made for the sole purpose of obtaining traps. It only processes TRAPs as individual items. One trap is able to contain a lot of information. Sometimes, you may have a situation where the only monitoring that can be carried out is based on traps. Therefore, you might choose to post-process the obtained information of one trap by an external script working as a plugin.

To process the data of one trap in detail, send all contained information to a script as an alert result. Here the trap shown below has been used for the example. It is a trap view that looks the same as the one in Pandora FMS SNMP console log:

2010-08-26 12:01:46 pandora 10.201.246.2 .1.3.6.1.4.1.1722 .1.3.6.1.4.1.1722.2.10.1.1.1 233 .1.3.6.1.4.1.1722.2.10.1.1.3 = STRING: 
AIX_Software_Failure .1.3.6.1.4.1.1722.2.10.1.1.2 = STRING: 08 25 2010 08:23:43:697685 .1.3.6.1.4.1.1722.2.10.1.1.8 = STRING: 1: A 
software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource 
SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED. .1.3.6.1.4.1.1722.2.10.1.1.6 = STRING: 8 
.1.3.6.1.4.1.1722.2.10.1.1.11 = STRING: An application may not work properly .1.3.6.1.4.1.1722.2.10.1.1.10 = STRING: An application 
may not work properly .1.3.6.1.4.1.1722.2.10.1.1.12 = INTEGER: 4 .1.3.6.1.6.3.1.1.4.3.0 = OID: .1.3.6.1.4.1.1722



Snmp plugin1.png



Snmp plugin2.png


On the screen shots below, you may see how a special alert is created. It executes a script which contains the complete content of the trap (_data_) and shows how an SNMP type of alert is created. In such a case, it has been mapped for the specific OID (.1.3.6.1.4.1.1722.2.10.1.1.1), although it could have been more general, e.g. (.1.3.6.1.4.1.1722) to call the script when there are any type of traps like these (.1.3.6.1.4.1.1722, it should be part of the AIX specific MIB).

A script which processes this data is executed. It also 'analyzes' the trap to write data directly to Pandora FMS by creating an XML file, moving it to '/var/spool/pandora/data_in' as data, as if it had come from an agent. A basic script for this case would be one to generate complex information. There is already enough information regarding this trap, which comprises the following:

  • The source IP
  • The main event (cold start)
  • The secondary events (descriptives): AIX_Software_Failure, 1: A software error PERM with label CORE_DUMP, identifier C69F5C9B which took place on Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC. The cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, an application may not work properly

When designing a script which 'parses' each of these data, e.g. 'miscript.pl' and stores it under '/var/spool/pandora/data_in' in the XML file along with a generic name and one random number, e.g. 'snmp_gateway.31415.data'.

The generated XML is supposed to look like the one on the picture below.


<?xml version='1.0' encoding='ISO-8859-1'?>
<agent_data description='' group='' os_name='aix' os_version='' interval='300' version='3.1(Build 100608)' timestamp='2010/08/26 12:20:26' agent_name='10.201.246.2'>
  <module>
    <name><![CDATA[Critical_Event]]></name>
    <description><![CDATA[]]></description>
    <type>async_proc</type>
    <data><![CDATA[1]]></data>
  </module>
<module>
    <name><![CDATA[events]]></name>
    <description><![CDATA[]]></description>
    <type>generic_string</type>
    <datalist>
      <data><value><![CDATA[AIX_Software_Failure]]></value></data>
      <data><value><![CDATA[A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC.]]></value></data>
      <data><value><![CDATA[Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly, An application may not work properly.]]></value></data>
    </datalist>
  </module>
</agent_data>

The application is endless, but any script should be customized and it would have a very dynamic structure. Under a lot of systems, the information which gets received does not solely consist of text. It is also numerical and therefore able to feed numerical information to the modules in order to represent e.g. graphics, etc. Note that the data generated in XML should always be asynchronous.

1.8.1 Practical Example: ESX Monitoring using Traps

One of the most problematic things to monitor is the monitoring of distributed infrastructure. It gets even harder if each version changes its implementation to gather information (like VMware or ESX). This chapter will explain how to monitor ESX systems by using an external SNMP Trap Handler.

ESX traps consist of the following data:

.1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = STRING: "c7000-06-01.tsm.inet"      .1.3.6.1.4.1.6876.4.3.303 = ""  
.1.3.6.1.4.1.6876.4.3.304 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.305 = STRING: "Yellow"    .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host 
cpu usage - Metric Usage = 1%"
.1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = STRING: "dl360-00.tsm.inet" .1.3.6.1.4.1.6876.4.3.303 = ""  
.1.3.6.1.4.1.6876.4.3.304 = STRING: "Yellow".1.3.6.1.4.1.6876.4.3.305 = STRING: "Green"     .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host 
memory usage - Metric Usage = 84%"
.1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = ""  .1.3.6.1.4.1.6876.4.3.303 = ""  .1.3.6.1.4.1.6876.4.3.304 = 
STRING: "Red"       .1.3.6.1.4.1.6876.4.3.305 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.306 = STRING: "Datastore usage on disk - Metric 
Storage space actually used = 55%"

As you can see, traps could be used to gather information from the CPU, the hard drive or the memory. The general idea behind the trap handler is to write a small script which is able to 'understand' the trap and to create an XML file which emulates a software agent. It is recommended to write a trap handler for each technology. The process of doing so is pretty common and explained in four steps:

  1. Create the handler script. You may base your work on the script which is provided below.
  2. Create an alert command.
  3. Create an alert action using the previous command. You can add some custom options for each 'destination' agent you want (if you possess several farms of ESX, you will probably wish to host the data on different agents).
  4. Create an SNMP trap-alert which maps the enterprise OID (information trap for all kinds of this specific technology and / or the source trap IP address).

Now let us go to the next point: how to create the trap-handler script.

1.8.1.1 Step 1 - The Trap Handler Script (esx_trap_manager.pl)

#!/usr/bin/perl
# (c) Sancho Lerena 2010 <[email protected]>
# Specific Pandora FMS trap collector for ESX 

use POSIX qw(setsid strftime);

sub show_help {
	print "\nSpecific Pandora FMS trap collector for ESX\n";
	print "(c) Sancho Lerena 2010 <[email protected]>\n";
	print "Usage:\n\n";
	print "   esx_trap_manager.pl <destination_agent_name> <TRAP DATA>\n\n";
	exit;
}

sub writexml {
	my ($hostname, $xmlmessage ) = @_;
	my $file = "/var/spool/pandora/data_in/$hostname.".rand(1000).".data";

	open (FILE, ">> $file") or die "[FATAL] Cannot write to XML '$file'";
	print FILE $xmlmessage;
	close (FILE);
}

if ($#ARGV == -1){
	show_help();
}

$chunk = "";

# The first parameter is always the destination host for the virtual server.
$target_host = $ARGV[0];

foreach $argnum (1 .. $#ARGV) {
	if ($chunk ne ""){
		$chunk .= " ";
	}
	$chunk .= $ARGV[$argnum];
}

my $hostname = "";
my $now = strftime ("%Y-%m-%d %H:%M:%S", localtime());
my $xmldata = "<agent_data agent_name='$target_host' timestamp='$now' version='1.0' os='Other' os_version='ESX_Collectordime ' interval='9999999999'>";

if ($chunk =~ m/.1.3.6.1.4.1.6876.4.3.302 \= STRING\: ([A-Za-z0-9\-\.]*)\s\.1/){
	$hostname = "_".$1;
}

if ($chunk =~ m/Host cpu usage \- Metric Usage \= ([0-9]*)\z/){
	$value = $1;
	$module_name = "CPU_OCUPADA$hostname";
}

if ($chunk =~ m/Host memory usage \- Metric Usage = ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "MEMORIA_OCUPADA$hostname";
}

if ($chunk =~ m/Datastore usage on disk \- Metric Storage space actually used \= ([0-9\.]*)\z/){
	$value = $1;
	$module_name = "DISCO_OCUPADO$hostname";
}

$xmldata .= "<module><name>$module_name</name><type>async_data</type><data>$value</data></module>\n";

$xmldata .= "</agent_data>\n";
writexml ($target_host, $xmldata);


1.8.1.2 Step 2 - Creating the Alert Command

In this example, the command script has been entered in '/tmp' to place it in a safer place, and to make sure it is executable (chmod 755):



Trap handler step2.png


1.8.1.3 Step 3 - Creating the Alert Action

A specific action to send all information to specific trap agents is going to be created. In this case, all information will be sent to an agent named 'WINN1247VSR'. The above mentioned command accepts the name of the agent as a parameter which will receive all the information (ESX Virtual Center), and 'chunks off' " the data from the trap, which can be unlimited and also includes all the information sent to the trap.



Trap handler step3.png


1.8.1.4 Step 4 - Creating the SNMP Alert

You may set alert traps by using the action you have just created.



Trap handler step4.png


In order to process all the traps the of ESX type by using the specific '.1.3.6.1.4.1.6876.4.3.301' OID to map ESX type traps. There is also the option of filtering by source IP for each virtual center by filtering by source IP address (which is contained in the trap).

1.8.1.5 Data Visualization

This is an example of how the information will look. You may manage them as standard modules with this data.



Trap handler step5.png


1.9 SNMP trap forwarding ( > v5.0 )

With Pandora FMS, it is also possible to forward SNMP traps to an external host by enabling the token named snmp_forward_trap within the server configuration file.

1.9.1 Configuration Example to forward Traps through SNMP v1

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 1
snmp_forward_community public
snmp_forward_secName
snmp_forward_engineid
snmp_forward_authProtocol
snmp_forward_authPassword
snmp_forward_privProtocol
snmp_forward_privPassword
snmp_forward_secLevel

1.9.2 Configuration Example to forward Traps through SNMP v2c

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 2c
snmp_forward_community public
snmp_forward_secName
snmp_forward_engineid
snmp_forward_authProtocol
snmp_forward_authPassword
snmp_forward_privProtocol
snmp_forward_privPassword
snmp_forward_secLevel

1.9.3 Configuration example to forward Traps through SNMP v3

This example is particularly challenging because of the implied knowledge regarding SNMP v3 traps. The remote SNMP agent is considered to be defined under snmp_forward_ip. It contains the following entry in its file under '/etc/snmp/snmptrapd.conf':

createUser -e 0x0102030405 myuser MD5 mypassword DES myotherpassword

The Pandora server configuration file should look like this:

snmp_forward_trap 1
snmp_forward_ip 192.168.1.145
snmp_forward_version 3
snmp_forward_secName myuser
snmp_forward_engineid 0x0102030405
snmp_forward_authProtocol MD5
snmp_forward_authPassword mypassword
snmp_forward_privProtocol DES
snmp_forward_privPassword myotherpassword
snmp_forward_secLevel authNoPriv

If you wish to learn more about it, you can find more information here.

1.10 Independent Management of the snmptrapd daemon

If, for some reason, you would like to manage the snmptrapd daemon independently from Pandora FMS (stop or start it independently from the main Pandora FMS daemon) then take into account several things:

1. The snmpconsole parameter must be active for Pandora FMS server.

2. The logs configured in Pandora FMS server must be the same as the ones in the independent call to snmptrapd.

3. The call to snmptrap must be in a specific format the standard system call cannot be used. That call must be like the following (the -A parameter is very important!):

/usr/sbin/snmptrapd -A -t -On -n -a -Lf /var/log/pandora/pandora_snmptrap.log -p /var/run/pandora_snmptrapd.pid --format1=SNMPv1[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n --format2=SNMPv2[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n

4. The snmptrapd token must be configured in Pandora FMS configuration file:

snmp_trapd manual

5. To enable this feature, complete the following procedure::

  • Change the configuration in /etc/pandora/pandora_server.conf
  • Stop the Pandora FMS server.
  • Check that the snmptrapd process is not running (if it is running, wait until the process stops or kill it)
  • Start snmptrapd manually (with the format specified above).
  • Start Pandora FMS Server.

1.10.1 Trap log file management

The snmptrapd process can be stopped and started without having to stop or start the Pandora FMS server process if the pandora_snmptrap.log.index and pandora_snmptrap.log have not been modified. If those files are modified it is necessary to restart Pandora FMS server. If you need to rotate the trap log files externally, then restart Pandora FMS server, after deleting the two files previously mentioned.

1.11 SNMP trap buffering

If SNMP traps are sent to an external manager through an unreliable connection, some information will be lost. Pandora FMS allows you to instead forward traps from a local snmptrapd to your Pandora FMS Server in a reliable way.

1.11.1 Architecture

Remote snmp trap schema.jpg

  • An SNMP agent sends traps to a local snmptrapd.
  • A local Pandora FMS agent reads traps from snmptrapd's log file and sends them to the designated Pandora FMS Server inside an XML data file, saving it in the XML buffer and retrying it at a later time if necessary.
  • The Data Server reads traps from XML data files and dumps them to a plain text file.
  • The SNMP Console processes traps from the plain text file.

Template warning.png

Having the SNMP Console directly process traps from snmptrapd's log file is more efficient. This setup should only be used if direct connectivity or reliability are a concern.

 


1.11.2 Prerequisites

  • A local snmptrapd that receives traps.
  • A local Pandora FMS Agent.
  • A Pandora FMS installation.

1.11.3 Configuration

1.11.3.1 snmptrapd

Edit the /etc/snmp/snmptrapd.conf file and make sure it logs into a file in a format compatible with Pandora FMS (change the name of the log file if necessary):

[snmp] logOption f /var/log/snmptrapd.log
format1 SNMPv1[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n
format2 SNMPv2[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n

1.11.3.2 Pandora FMS Agent

Use the grep_snmptrapd plugin that comes with the Pandora FMS agent to read data from snmptrapd's log file.

Edit the local agent's configuration file, /etc/pandora/pandora_agent.conf, and add the following line, adjusting the path of snmptrad's log file if necessary:

module_plugin grep_snmptrapd /var/log/snmptrapd.log

1.11.3.3 Pandora FMS Server

Tell the SNMP Console to process traps from an external log file that will be written by the Data Server.

Edit the server's configuration file, /etc/pandora/pandora_server.conf and:

  • Make sure the SNMP Console is enabled:
snmpconsole 1
  • Make sure the Data Server is enabled:
dataserver 1
  • Configure an external SNMP log file. If it does not exist, the SNMP Console will create it:
snmp_extlog /var/log/pandora/pandora_snmptrap.ext.log

Template warning.png

snmp_extlog can be any file writable by the Pandora FMS Server, but it must be different from snmp_logfile (defined in /etc/pandora/pandora_agent.conf too).

 


1.12 Trap Generator

With this tool, you can generate custom traps that you can later see in the SNMP console.

Generador de traps.png


In order to be able to correctly configure the trap generator, fill in the following fields:

  • Host Address: This is the destination IP to which the trap will be sent.
  • Community: Where the password of the SNMP community you are trying to access with the trap generator must be set.
  • Enterprise String: The OID (Object Identifier) of the trap must be configured here. For example: 1.3.6.1.2.1.2.2.1.8
  • Value: The value desired to be given to the trap and that will then appear as Data.
  • SNMP Agent: Insert the agent where you will simulate the trap, entering the IP of the same one.
  • SNMP Type: Choose an SNMP type among the following options:
    • Cold Start: Indicates that the agent has been started or restarted.
    • Warm Start: Indicates that the agent configuration has been modified.
    • Link down: Indicates that the communication interface is out of service (inactive).
    • Link up: Indicates that a communication interface has been activated.
    • Authentication failure: Indicates that the agent received a request from an unauthorized NMS (community-controlled)
    • EGP neighbor loss: Indicates that on the systems where routers were using the EGP protocol, a nearby host is out of service.
    • Enterprise: This category includes all the new traps. Including traps from suppliers.


Go back to Pandora FMS documentation index