Differences

This shows you the differences between two versions of the page.

Link to this comparison view

en:documentation:03_monitoring:08_snmp_traps_monitoring [2021/06/08 23:35]
jimmy.olano [Introduction] Error de migración.
en:documentation:03_monitoring:08_snmp_traps_monitoring [2021/09/16 09:17]
Line 1: Line 1:
-====== SNMP-Traps Monitoring ====== 
-{{indexmenu_n>8}} 
- 
-[[en:documentation:start|Go back to Pandora FMS documentation index]] 
- 
- 
-===== Operations with SNMP Traps ===== 
- 
-==== Introduction ==== 
- 
-Network devices that support SNMP, such as switches, routers, servers, printers, or APs can send alarms (or **SNMP traps**) when certain events take place, such as interface failure, too high CPU or network load, a UPS changing status or a filled up disk partition. Each device has its own "collection" of possible events, which is reflected in a collection, called MIB, in this case, different from the MIB used to query the device. 
- 
-{{  :wiki:trap-example.png  }} 
- 
-Traps are sent only when something takes place in an asynchronous (not reapetitive in time) by a device to an SNMP trap receptor. Pandora FMS includes a TRAP Reception Console which allows TRAPs that were sent by monitored objects to be displayed and add alerts to said traps. **SNMP traps** are received by the operating system's daemon which starts the SNMP Server of Pandora FMS at the same time of the server startup. This server stores received traps in a log file in 
-<code> 
-/var/log/pandora /pandora_snmptrap.log 
- 
-</code> 
- 
-Traps are usually received in RAW format. That means they contain numerical OIDs, unless there is an MIB installed on the Operating System that is capable of resolving them. Pandora FMS Enterprise SNMP Console enables rule creation for renaming numeric OIDs to alphanumeric ones or simple descriptive text strings (e.g., 'interface is down'), in order to make working with TRAPs more intuitive. Pandora FMS allows loading trap MIBs of any sort in order to work with them intuitively. Pandora FMs also allows loading trap MIBs from any manufacturer to define automatically those rules, you may get more information through the video tutorial "[[https://www.youtube.com/watch?v=Oaxf7vi7eME|Loading MIBs in Pandora FMS]]". 
- 
-In order to work with SNMP traps, first modify the following parameter in ''/etc/pandora/pandora_server.conf'' to enable the SNMP Console: 
-<code> 
- 
-snmpconsole 1 
- 
-</code> 
- 
-If you want the traps to be translated (either the variable bindings or the Enterprise string), enable the following parameters (only Enterprise): 
- 
-<code> 
- translate_variable_bindings 1 
- translate_enterprise_strings 1 
- 
-</code> 
- 
-Set up the ''/etc/snmp/snmptrapd.conf'' file with the appropriate parameters too. For instance: 
-<code> 
- authCommunity log public 
- disableAuthorization yes 
- 
-</code> 
- 
-As for this configuration, traps will be created for the **public** community and they will not need any authorization. 
- 
- 
-=== SNMPv3 === 
-SNMPv3 traps are rejected unless the sending user is added to ''/etc/snmp/snmptrapd.conf'' using the [[http://www.net-snmp.org/docs/man/snmptrapd.conf.html|''createUser'']] directive. For example: 
- 
-<code> 
- disableAuthorization yes 
- createUser -e 0x0102030405 snmpv3user SHA mypassword AES 
-</code> 
- 
-<WRAP center round important 60%> 
-The engineID must be specified with the ''-e'' option. Otherwise, only SNMPv3 INFORMs will be received. 
-</WRAP> 
- 
-==== Access to TRAP Reception Console ==== 
-To use the trap reception console, go to **Monitoring > SNMP > SNMP Console**, where you may take a look at the list of TRAPs which have been received so far. There is an eye-shaped icon which displays all the trap information. You can learn any detailed information regarding SNMP traps there. 
- 
- 
-{{ wiki:Traps.jpg?850 }} 
- 
- 
-For each trap, the following columns will be displayed: 
- 
-**Status**  
- 
-A green box appears if the trap is validated. It turns red if not. 
- 
-**SNMP Agent**  
- 
-The Agent which has sent the trap. 
- 
-**OID**  
- 
-The OID of the sent trap. A trap can only send one piece of data in this field. 
- 
-**Value**  
- 
-The value field of the sent trap. A trap can only send one piece of data in this field.  
- 
-**Custom OID, Custom Value**  
- 
-The customized fields, sent within the trap. They sometimes consist of very complex data which bears a specific logic, which sends the trap. A trap is able to send several types of data in this field. 
- 
-**Time Stamp**  
- 
-The elapsed time since the trap reception. 
- 
-**Alert**  
- 
-A yellow box appears if any alert was launched by this trap. It is grey if no alert was launched. 
- 
-**Action** 
- 
-The field for deleting or validating the trap. 
- 
-**Color** 
- 
-Traps also bear the following colors (seen as filling color for the trap line), depending on the trap type: 
- 
-  * **Blue:** maintenance traps. 
-  * **Purple:** information traps. 
-  * **Green:** ''normal'' traps. 
-  * **Yellow:** ''warning'' traps. 
-  * **Red: ** ''critical'' traps. 
- 
-At the top of the trap console, there is the option named **Toggle Filter** displayed. By clicking on that option, the trap's filtering field options appear or disappear. 
- 
- 
-{{ wiki:alert.png?850 }} 
- 
- 
-=== TRAP Validation === 
-In order to effectively manage traps, it is possible to validate them, so the Administrator is able to distinguish between pending traps and the ones already seen.  
- 
-To validate a trap, click on the green circle to the left of the trap.  
- 
- 
-{{ wiki:Traps2.JPG?800 }} 
- 
- 
-It is also possible to validate multiple traps by checking them and clicking on **validate**. 
- 
-=== TRAP Deletion === 
-It is possible to delete traps once they have been edited, either individually or by multiple selection and **Delete** action.  
- 
- 
-{{ wiki:Traps3.jpg }} 
- 
- 
-To prevent traps from pilling up, there is a default setting option that automatically deletes traps older than 10 days. 
- 
-==== SNMP Trap Alerts ==== 
-=== Introduction === 
-Pandora FMS also has an alert system for received SNMP traps. They are mainly based on filtering rules, searching for matches in all possible fields according to rules that are set up to trigger the alert. Before reading on, bear in mind you can find out more about Pandora FMS alerts (([[en:documentation:04_using:01_alerts]])) 
- 
-=== Adding an alert === 
-SNMP trap alerts have several fields that will be used to look for matches in SNMP traps received in the console. You may optionally use any fields you want to create more general or more specific rules as required: 
- 
-{{ wiki:alertSNMP1.png?500 }} 
- 
- 
-**Enterprise String** 
- 
- The main OID of the trap. It will look for the presence of the string. For example, if you are looking for a piece of the OID, you may use ''1.21.34.2.3'' and every OID that contains that one will be filtered, as if it were ''*1.21.34.2.3*'' But there is NO need to use the * character. 
- 
-**Custom Value/OID** 
- 
- This element will search within the trap's **value**, **custom OID**, **custom value** and in the rest of the TRAP fields. Regular expressions are supported here. For example, if you have a trap that sends the "Testing TRAP 225" string, you can search for any trap with the subchain "Testing TRAP" through the regular expression "Testing. *TRAP.*" 
- 
-**SNMP Agent** 
- 
- The IP of the agent which has sent the trap. You may use a regular expression or a substring here too. 
- 
-**Trap type** 
- 
- The filter by trap type. Most of the generated traps are usually **Other** type. If nothing is specified, it will look for any type of trap. 
- 
-{{ wiki:Trap type.png?360 }} 
- 
-**Single value** 
- 
- It filters by trap value. Keep in mind this refers only to the MAIN OID value, not to any additional OIDs. 
- 
-**Variable bindings/Data #1-20'** 
- 
- These are regular expressions which try to match the binding variables from 1 to 20. If there is a match, the alert is triggered. The value of the variable is stored in the corresponding ''_snmp_fx_'' macro (e.g. ''_snmp_f1_'', ''_snmp_f2_'', etc.). Although only twenty binding variables are able to search for matches, the ''_snmp_fx_'' macros are set for all of them (''_snmp_f11_'', ''_snmp_f12_'', etc.). 
- 
-{{ wiki:alertSNMP2.png?600 }} 
- 
-**Field 1** 
- 
- Field to set the ''Field 1'' alarm command parameter. This is the field that will be used in case of choosing to generate an event, or the destination mail in case of choosing an ''eMail'' action (if you wish to overwrite the default email in the action). To fully understand how custom fields work in actions/alerts templates, read the documentation chapter that explains the alerts in Pandora FMS [[http://wiki.pandorafms.com/index.php?title = en:documentation:start:Alerts|here]]. 
- 
-**Field 2** 
- 
-  Field to set the command parameter of the ''Field 2'' alarm. In case of sending an email, it will be the subject of the message. If left blank, it would use what it had defined in the action. 
- 
-**Field 3** 
- 
-  Field to set the command parameter of the ''Field 3'' alarm. In case of sending an email, it would be the text of the message. If left blank, it would use what it had defined in the action. 
- 
-**Min. Number of Alerts** 
- 
- The field to define the minimum amount of traps that must be received to trigger the alarm. 
- 
-**Max. Number of Alerts** 
- 
- Field where the maximum number of times the action will be executed in the given interval (or **time threshold**) is. 
- 
-**Time Threshold** 
- 
- The field for determining the time to elapse before resetting the alarm counter. This counter is the one which gets used for the field named **min. number of alerts**. 
- 
-**Priority** 
- 
- Combo where the alarm priority is set. The priorities of the alerts are different and have nothing to do with the priority of the traps, nor with the Pandora FMS events. 
- 
-**Alert Action** 
- 
- The combo box for selecting the action that will execute the alert. If you select an event, the normal event of an alert creation will not be created. 
- 
-**Position** 
- 
- The alerts with a lower position are evaluated first. If several alerts with the same position match a trap, all alerts matching the same position will be triggered. Although lower position alerts may match the trap, they will not be triggered. 
- 
-=== Alert Field Macros === 
-The following macros can be used in any of the alert //fields//: 
- 
-  * ''_data_'' Full trap 
-  * ''_agent_'': Agent name 
-  * ''_address_'': IP Address 
-  * ''_timestamp_'': Trap date 
-  * ''_snmp_oid_'': Trap OID 
-  * ''_snmp_value_'': Trap OID value 
- 
-=== Trap Alert Example === 
-{{ wiki:trap_sample_for_alert.jpg }} 
- 
-In this case, you have a main OID (''. 1.3.6.1.4.1.2789.2005'') that identifies a trap that can contain CPU overheating messages (or even other kind of messages). The manufacturer know there are two variables fot this type of alert, the first one identified as overheating and the second one with the temperature's value; both text strings. 
- 
-Defining the first part of the trap is simple, just use the main OID to make the first and most important pre-filter: 
- 
-{{ wiki:trap_alert_definition_1.jpg }} 
- 
-The second part of the trap definition is the one containing the essential part. In the first variable of the trap, look for the ''Heat alert'' string. 
- 
-{{ wiki:trap_alert_definition_2.jpg }} 
- 
-And finally, in **Field 1** use the variables that contain the value of variables 1 and 2 of the trap received (''_snmp_f1_'' and ''_snmp_f2_'') together with a descriptive text: 
- 
-{{ wiki:trap_alert_definition_3.jpg }} 
- 
-That way, when the alert goes off, the generated event will look like this: 
- 
-{{ wiki:event_result_of_alert.jpg }} 
- 
-==== Working with environments with many traps ==== 
-=== TRAP-Storm Protection === 
-There are a couple of parameters in the server which are conceived to protect the system against the arrival of a Trap Storm, coming from a single location. Use the following settings in the ''pandora_server.conf'' file for this: 
- 
-  * ''snmp_storm_protection''> The max. number of processed SNMP traps by the same source IP in a given interval (see below). 
-  * ''snmp_storm_timeout''> The interval in seconds for protection against an SNMP Trap Storm. During this interval, the system will only process 'snmp_storm_protection' type traps from the same source (IP). 
-  * ''snmp_storm_silence_period''> If it is greater than 0 each time the storm protection is triggered for a particular source, the current time will be added plus the silence time. Until this time passes, no new traps will be registered for the specific source. 
- 
-When this protection fires, it is reflected in an event on the console: 
- 
-{{ wiki:Storm silence snmp.png?400 }} 
- 
-Trap storm protection combined with trap filtering (see below) allows that if you receive hundreds of thousands of traps per day, you work with only a few thousand traps to delete redundant or unhelpful traps. 
- 
-=== TRAP Filtering in the Server === 
-Some systems receive a large number of traps from which only a small percentage is relevant to monitor. With Pandora FMS, it is possible to filter the traps that the server receives in order to avoid unnecessary application loading. From **Monitoring** > **SNMP** > **SNMP Filters** you can define different filters.  
- 
-{{ wiki:Monitoring-snmp-snmp_filters.png?600 }} 
- 
-Add a description and as many filters as you need with **+**: 
- 
-{{ wiki:Snmp_filter_editor_new.png?700 }} 
- 
-A trap that matches any of them will automatically be discarded by the server. The filter is applied as a regular expression on the trap entry in the SNMP log (default ''/var/log/pandora/pandora_snmptrap.log''), which has the following fixed format: 
- 
-<code> 
-%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n 
-</code> 
- 
-Being:  
- 
-  * ''%y''> current year. 
-  * ''%m''> current month (numerical). 
-  * ''%l''> current day of the month. 
-  * ''%h''> current hour. 
-  * ''%j''> current minute. 
-  * ''%k''> current second. 
-  * ''%a''> Origin address (traps version 1 only). 
-  * ''%N''> OID. 
-  * ''%w''> trap type (numerical). 
-  * ''%W''> trap description. 
-  * ''%q''> trap sub-type (numerical). 
-  * ''%v''> List of variables separated by tab (custom OID). 
- 
-For example, to filter all the traps from 192.168.50.20 host, set the following filter: 
- 
-{{ wiki:Snmp_filter_example.png?400 }} 
- 
-Since more than one filter can be created simultaneously, the search will take into account those traps that meet all filtering conditions. 
- 
-=== SNMP Trap stats === 
-This view allows you to see trap statistics, both by origin (IP) and by OID, this allows an effective filter management, by identifying the IPs that generate more traps and the OIDs that are more repeated. The system displays trap statistics for the last 30 days. 
- 
-{{ wiki:Captura_de_pantalla_2014-09-23_a_la(s)_17.59.56.png?750 }} 
- 
- 
-{{ wiki:Captura_de_pantalla_2014-09-23_a_la(s)_19.11.40.png?750 }} 
- 
- 
-==== Customizing SNMP Traps ==== 
-<WRAP center round tip 60%> 
-{{wiki:icono-modulo-enterprise.png |Enterprise version.}}The following features are exclusive to the Enterprise version. 
-</WRAP> 
- 
-In order to give the operator a better understanding of the traps sent by the monitored devices, it is possible to either load the manufacturer's MIBs into Pandora FMS or to edit the traps to your liking. 
- 
-=== Renaming and customizing traps === 
-"Edit a trap", "customizes" the appearance of a trap in the console. To edit a trap, go to **Operation** > **SNMP Console** > **SNMP trap editor**. 
- 
-{{ wiki:Monitoring-snmp-snmp_trap_editor.png?600 }} 
- 
-Click on the editing icon of the trap you wish to customize: 
- 
-{{ wiki:Traps5.JPG?800 }} 
- 
-A window similar to this one will appear: 
- 
-{{ wiki:Traps7.JPG?500 }} 
- 
-That way, when you find traps with the ''. 1.3.6.6.1.4.1.2789.2005'' OID, you will see them as "Blue box sample". And it will be easier to tell them apart. Its content (including the original OID) will remain the same. 
- 
-**Custom OID** is a Perl compatible regular expression that will be matched against the part of the trap string that contains variable bindings. Generally there is no need to translate a trap. 
- 
-<WRAP center round important 60%> 
-"Custom OID" is not meant to be the whole variable binding string, which can be larger than its supported maximum length, but rather a regular expression that matches one or more variables. 
-</WRAP> 
- 
-Keep in mind that all previous traps will not change their appearance, this will start working with the new traps that enter the system from that moment onwards. 
- 
-Finally, this is what a user-defined trap would look like: 
- 
- 
-{{ wiki:Traps8.JPG?780 }} 
- 
- 
-=== Loading Manufacturer MIBs === 
-This option is suitable for uploading trap MIBS (exclusively) and enlarging Pandora FMS internal translation database, so that when a trap arrives, it will be automatically translated by its description.Go to **Monitoring** > **SNMP** > **MIB uploader**. 
- 
-{{ wiki:Monitoring-snmp-snmp-mib_uploader.png?600 }} 
- 
-To upload a manufacturer's MIBs, click on Upload file (s) choose the file and click **Go**. 
- 
-{{ wiki:cima6.png?500 }} 
- 
-Once  uploaded, the system will add it to its trap library. 
- 
-==== Alerts with complex SNMP traps ==== 
-The previously described alerts will only be used where the trap is appropriately defined. It is always the same and it does not bear any relevant data to recover. 
- 
-In certain situations, you might however find a trap which presents the following structure: 
- 
-<code>  
- OID: .1.3.6.1.4.1.2789.2005 
- Value: 666 
- Custom data: 1.3.6.1.4.1.2789.2005.1 = STRING: "ID-00342" .1.3.6.1.4.1.2789.2005.2 = STRING: "Automated check"   
- .1.3.6.1.4.1.2789.2005.3 = STRING: "NIC Offline" .1.3.6.1.4.1.2789.2005.4 = STRING: "4897584AH/345" 
-</code> 
- 
-This trap contains complex data alongside an OID and a value, based on many other OIDs and values. In its complex part, a trap can contain a completely randomized structure which is based on OIDs or value pairs (e.g., counters, numerical, alphanumerical, dates, etc.). 
- 
-Within the trap console, this trap would appear as in the picture below: 
- 
-{{ wiki:Traps9.jpg?800 |Click to zoom in}} 
- 
-If you carefully read the extended information (**Variable bindings**), there might be some pieces of useful data. In the first field, containing the ''2005.1'' OID ending looks like an identifier. The third field, containing the ''2005.3'' OID ending looks like an error message. Fields 2 and 4 do not look useful, since they seem to be of unknown code. 
- 
-Let us assume for a moment you could create an event from a trap, moving specific parts from the trap data into the text. And let us suppose you intend to build an event which contains the following information: 
- 
-  Chassis Alert: <error message> in device <identifier> 
- 
-The challenge is to make an alert which looks for 'matches' in those fields, obtains the piece of data and uses it to create a message in the alert later.  
-This task can be performed using Pandora FMS and advanced regular expressions and selectors. You may find more information about regular expressions [[http://en.wikipedia.org/wiki/Regular_expression#Expressive_power_and_compactness|here.]] 
- 
-The selectors use ''('' and '')'', allowing you to "copy" information by using a research expression. 
- 
-The regular expression to obtain the identifier would be like the following one: 
- 
-<code> 
-.*.1.3.6.1.4.1.2789.2005.1 \= STRING\: \"([0-9\-\_A-Za-z]+)\" 
-</code> 
- 
- 
-The regular expression to obtain the error message would be something like this: 
- 
-<code> 
-.*.1.3.6.1.4.1.2789.2005.3 \= STRING\: \"([\sA-Za-z]+)\".* 
-</code> 
- 
- 
-Once you have obtained the data fields, use them in the alert. For this purpose, the ''_snmp_f1_'', ''_snmp_f2_'' and ''_snmp_f//n//_'' special macros are used. However, using these macros outside any SNMP trap alert does not make any sense. 
- 
-To build the message, use the following string: 
- 
-  Chassis Alert: _snmp_f2_ in device _snmp_f1_ 
- 
-The picture below shows how the complete alert is created. 
- 
-{{ wiki:Compex_trap_def1.png?640 }} 
- 
- 
-In order to successfully create this type of alert, an extensive knowledge of regular expressions is required, since a simple blank space, a symbol or another character in the wrong location could make it malfunction.  
- 
-<WRAP center round tip 60%> 
-Keep in mind that SNMP alerts imply the use of regular expressions. 
-</WRAP>  
- 
-An easier way to establish an alert by regular expressions would be the following one:  
- 
-{{ wiki:Compex_trap_def4.png?500 }} 
- 
- 
-=== Additional Example === 
-This additional example uses an **eMail** alert to send information about the interface name each time you receive a specific trap. You will receive an email containing a device name, IP and interface name as received in the trap. 
- 
-This is a trap received from a switch: 
- 
-{{ wiki:Another_trap_alert_sample.png?800 |Click to zoom in}} 
- 
- 
-This is how the email should be received. 
- 
-{{ wiki:trap_email.png?700 }} 
- 
-This is how the trap is defined. 
- 
-{{ wiki:another_snmp_trap_sample.png?630 }} 
- 
-{{ wiki:another_snmp_trap_sample2.png?600 }} 
- 
-==== Trap association to the rest of Pandora FMS alerts and SNMP agent trap forwarding ==== 
-The alerts defined on traps are completely independent from Pandora FMS Alert Engine, so their correlations trigger an alert if the temperature reaches 29 degrees and the trap for secondary power supply is on. This type of alerts cannot be displayed since they are -possibly- not associated to any Pandora FMS modules. Therefore, trap console monitoring cannot be related to elements such as reports or maps. 
- 
- 
-//Special Module SNMP Trap, with the trap sent from the SNMP console//: 
-{{ wiki:Snmptrap_agent.png }} 
- 
-In order to achieve this, a method called **Agent SNMP Trap Forwarding** has been created. This server-wide option forwards the trap to a special agent's module named **SNMPTrap** as a text string, but only if the trap's source IP address is defined as the agent's IP. Whenever this occurs, the trap arrives as a text line to the agent within that module, which is one that is only defined on the arrival of the first trap. 
- 
-Text alerts can be specified within that module, these being completely standard, just as any other module. This enables SNMP Monitoring customization in order for certain traps from certain origins to be treated as yet another type of module, which are thereby integrated into the rest of the monitoring, including Alert Correlation. 
- 
- 
-//SNMPTrap data sample'//: 
-{{ wiki:Snmptrapforward2.png }} 
- 
- 
-This is an **Enterprise** feature which could be activated on **Setup** > **Setup** > **Enterprise** with the following option: 
- 
- 
-//Configuration option to activate trap forwarding to the agents//: 
-{{ wiki:Snmptrap_agent_forwardsetup.png }} 
- 
-If this setting is modified, the Pandora FMS Server must be restarted to enable it. 
- 
-Another solution is to create an alert on the trap to activate an agent's module. If the trap has '1' in  a certain log file, there is an agent reading that file, ready to run it when it finds this "1". That way, the module will be triggered if the desired trap arrives and the correlation can be established, based on the arrived trap. 
- 
-==== External SNMP trap handler ==== 
-The SNMP console is made for the sole purpose of obtaining traps. It only processes TRAPs as individual items. One trap is able to contain a lot of information. 
-**Sometimes,  you may have a situation where the only monitoring that can be carried out is based on traps**. Therefore, you might choose to post-process the obtained information of one trap by an external script working as a plugin. 
- 
-To process the data of one trap in detail, send all contained information to a script as an alert result. Here the trap shown below has been used for the example. It is a trap view that looks the same as the one in Pandora FMS SNMP console log: 
- 
-<code> 
- 2010-08-26 12:01:46 pandora 10.201.246.2 .1.3.6.1.4.1.1722 .1.3.6.1.4.1.1722.2.10.1.1.1 233 .1.3.6.1.4.1.1722.2.10.1.1.3 = STRING:  
- AIX_Software_Failure .1.3.6.1.4.1.1722.2.10.1.1.2 = STRING: 08 25 2010 08:23:43:697685 .1.3.6.1.4.1.1722.2.10.1.1.8 = STRING: 1: A  
- software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource  
- SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED. .1.3.6.1.4.1.1722.2.10.1.1.6 = STRING: 8  
- .1.3.6.1.4.1.1722.2.10.1.1.11 = STRING: An application may not work properly .1.3.6.1.4.1.1722.2.10.1.1.10 = STRING: An application  
- may not work properly .1.3.6.1.4.1.1722.2.10.1.1.12 = INTEGER: 4 .1.3.6.1.6.3.1.1.4.3.0 = OID: .1.3.6.1.4.1.1722 
-</code> 
- 
-{{ wiki:SNMP2.JPG?600 }} 
- 
- 
- 
-{{ wiki:SNMP3.JPG?550 }} 
- 
- 
- 
-On the screenshots below, you may see how a special alert is created. It executes a script which contains the complete content of the trap (''_data_'') and shows how an SNMP type of alert is created. In such a case, it has been mapped for the specific OID (''.1.3.6.1.4.1.1722.2.10.1.1.1''), although it could have been more general, e.g. (''.1.3.6.1.4.1.1722'') to call the script when there are any type of traps like these. 
- 
-A script which processes this data is executed. It also 'analyzes' the trap to write data directly to Pandora FMS by creating an XML file, moving it to ''/var/spool/pandora/data_in'' as data, as if it had come from an agent. A basic script for this case would be one to generate complex information. There is already enough information regarding this trap, which comprises the following: 
- 
-  * The source IP. 
-  * The main event (cold start). 
-  * The secondary events (descriptives): AIX_Software_Failure, 1: A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC. Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly, An application may not work properly. 
- 
-When designing a script which "parses" each of these data, e.g. "miscript.pl" and stores it under ''/var/spool/pandora/data_in'' in the XML file along with a generic name and one random number, e.g. ''snmp_gateway.31415.data''. 
- 
-The generated XML is supposed to look like the one on the picture below. 
- 
- 
-<code> 
-<?xml version='1.0' encoding='ISO-8859-1'?> 
-<agent_data description='' group='' os_name='aix' os_version='' interval='300' version='3.1(Build 100608)' timestamp='2010/08/26 12:20:26' agent_name='10.201.246.2'> 
-  <module> 
-    <name><![CDATA[Critical_Event]]></name> 
-    <description><![CDATA[]]></description> 
-    <type>async_proc</type> 
-    <data><![CDATA[1]]></data> 
-  </module> 
-<module> 
-    <name><![CDATA[events]]></name> 
-    <description><![CDATA[]]></description> 
-    <type>generic_string</type> 
-    <datalist> 
-      <data><value><![CDATA[AIX_Software_Failure]]></value></data> 
-      <data><value><![CDATA[A software error PERM with label CORE_DUMP, identifier C69F5C9B occurred at Wed Aug 2 5 10:22:28 DFT 2010 on dvs02 for resource SYSPROC.]]></value></data> 
-      <data><value><![CDATA[Cause is SOFTWARE PROGRAM ABNORMALLY TERMINATED, An application may not work properly, An application may not work properly.]]></value></data> 
-    </datalist> 
-  </module> 
-</agent_data> 
-</code> 
- 
- 
-The applications are endless, but any script should be customized and it would have a very dynamic structure. Under a lot of systems, the information which gets received does not solely consist of text. It is also numerical and therefore able to feed numerical information to the modules in order to represent e.g. graphics, etc. Note that **the data generated in XML should always be asynchronous**. 
- 
-=== Practical example: ESX monitoring using traps === 
-One of the most problematic things to monitor is the monitoring of distributed infrastructure. It gets even harder if each version changes its implementation to gather information (like VMware or ESX). This chapter will explain how to monitor ESX systems by using an external SNMP Trap Handler. 
- 
-ESX traps consist of the following data: 
- 
-<code> 
- .1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = STRING: "c7000-06-01.tsm.inet"      .1.3.6.1.4.1.6876.4.3.303 = ""   
- .1.3.6.1.4.1.6876.4.3.304 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.305 = STRING: "Yellow"    .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host  
- cpu usage - Metric Usage = 1%" 
-</code> 
- 
-<code> 
- .1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = STRING: "dl360-00.tsm.inet" .1.3.6.1.4.1.6876.4.3.303 = ""   
- .1.3.6.1.4.1.6876.4.3.304 = STRING: "Yellow".1.3.6.1.4.1.6876.4.3.305 = STRING: "Green"     .1.3.6.1.4.1.6876.4.3.306 = STRING: "Host  
- memory usage - Metric Usage = 84%" 
-</code> 
- 
-<code> 
- .1.3.6.1.4.1.6876.4.3.301 = STRING: "host"  .1.3.6.1.4.1.6876.4.3.302 = ""  .1.3.6.1.4.1.6876.4.3.303 = ""  .1.3.6.1.4.1.6876.4.3.304 =  
- STRING: "Red"       .1.3.6.1.4.1.6876.4.3.305 = STRING: "Green" .1.3.6.1.4.1.6876.4.3.306 = STRING: "Datastore usage on disk - Metric  
- Storage space actually used = 55%" 
-</code> 
- 
- 
-As you can see, traps could be used to gather information from the CPU, the hard drive or the memory. The general idea behind the trap handler is to write a small script which is able to 'understand' the trap and to create an XML file which emulates a software agent. **It is recommended to write a trap handler for each technology but the process is common for all of them** and explained in four steps: 
- 
-  - Create the handler script. You may base your work on the script which is provided below. 
-  - Create an alert command. 
-  - Create an alert action using the previous command. You can add some custom options for each 'destination' agent you want (if you possess several farms of ESX, you will probably wish to host the data on different agents). 
-  - Create an SNMP trap-alert which maps the enterprise OID (information trap for all kinds of this specific technology and / or the source trap IP address). 
- 
-== Step 1 - The Trap Handler Script (esx_trap_manager.pl) == 
- 
-<code> 
-#!/usr/bin/perl 
-# (c) Sancho Lerena 2010 <[email protected]> 
-# Specific Pandora FMS trap collector for ESX  
- 
-use POSIX qw(setsid strftime); 
- 
-sub show_help { 
- print "\nSpecific Pandora FMS trap collector for ESX\n"; 
- print "(c) Sancho Lerena 2010 <[email protected]>\n"; 
- print "Usage:\n\n"; 
- print "   esx_trap_manager.pl <destination_agent_name> <TRAP DATA>\n\n"; 
- exit; 
-} 
- 
-sub writexml { 
- my ($hostname, $xmlmessage ) = @_; 
- my $file = "/var/spool/pandora/data_in/$hostname.".rand(1000).".data"; 
- 
- open (FILE, ">> $file") or die "[FATAL] Cannot write to XML '$file'"; 
- print FILE $xmlmessage; 
- close (FILE); 
-} 
- 
-if ($#ARGV == -1){ 
- show_help(); 
-} 
- 
-$chunk = ""; 
- 
-# The first parameter is always the destination host for the virtual server. 
-$target_host = $ARGV[0]; 
- 
-foreach $argnum (1 .. $#ARGV) { 
- if ($chunk ne ""){ 
- $chunk .= " "; 
- } 
- $chunk .= $ARGV[$argnum]; 
-} 
- 
-my $hostname = ""; 
-my $now = strftime ("%Y-%m-%d %H:%M:%S", localtime()); 
-my $xmldata = "<agent_data agent_name='$target_host' timestamp='$now' version='1.0' os='Other' os_version='ESX_Collectordime ' interval='9999999999'>"; 
- 
-if ($chunk =~ m/.1.3.6.1.4.1.6876.4.3.302 \= STRING\: ([A-Za-z0-9\-\.]*)\s\.1/){ 
- $hostname = "_".$1; 
-} 
- 
-if ($chunk =~ m/Host cpu usage \- Metric Usage \= ([0-9]*)\z/){ 
- $value = $1; 
- $module_name = "CPU_OCUPADA$hostname"; 
-} 
- 
-if ($chunk =~ m/Host memory usage \- Metric Usage = ([0-9\.]*)\z/){ 
- $value = $1; 
- $module_name = "MEMORIA_OCUPADA$hostname"; 
-} 
- 
-if ($chunk =~ m/Datastore usage on disk \- Metric Storage space actually used \= ([0-9\.]*)\z/){ 
- $value = $1; 
- $module_name = "DISCO_OCUPADO$hostname"; 
-} 
- 
-$xmldata .= "<module><name>$module_name</name><type>async_data</type><data>$value</data></module>\n"; 
- 
-$xmldata .= "</agent_data>\n"; 
-writexml ($target_host, $xmldata); 
-</code> 
- 
- 
-== Step 2 - Creating the Alert Command == 
-In this example, the command script has been entered in ''/tmp'', place it in a safer place, and make sure it is executable (''chmod 755''): 
- 
-{{ wiki:SNMP3.jpg?750 }} 
- 
- 
-== Step 3 - Creating the Alert Action == 
-Create a specific action to send all information to specific trap agents. In this case, all information will be sent to an agent named ''WINN1247VSR''. The above mentioned command accepts the name of the agent as a parameter which will receive all the information (''ESX Virtual Center''), and "pieces" of trap data, which can be unlimited and also includes all the information sent to the trap. 
- 
-{{ wiki:SNMP4.jpg?700 }} 
- 
- 
-== Step 4 - Creating the SNMP Alert == 
-You may set alert traps by using the action you just created to process all the traps the of ESX technology by using the specific ''.1.3.6.1.4.1.6876.4.3.301'' OID to map ESX traps. 
- 
-{{ wiki:SNMP5.jpg?500 }} 
- 
- 
-There is also the option of filtering by source IP for each virtual center by filtering by source IP address (sent in the trap). 
- 
-== Data display == 
-This is an example of how the information will look. You may manage them as standard modules with this data. 
- 
- 
-{{ wiki:SNMP6.jpg?650 }} 
- 
- 
- 
-{{ wiki:SNMP7.jpg?650 }} 
- 
- 
- 
-==== SNMP trap forwarding ( > v5.0 ) ==== 
-With Pandora FMS, it is also possible to forward SNMP traps to an external host by enabling the token named [[en:documentation:start:Configuration#.28.3E.3D_5.X.29_snmp_forward_trap|**snmp_forward_trap**]] within the Pandora FMS configuration file. 
- 
-=== Configuration Example to forward Traps through SNMP v1 === 
- 
-<code> 
- snmp_forward_trap 1 
- snmp_forward_ip 192.168.1.145 
- snmp_forward_version 1 
- snmp_forward_community public 
- snmp_forward_secName 
- snmp_forward_engineid 
- snmp_forward_authProtocol 
- snmp_forward_authPassword 
- snmp_forward_privProtocol 
- snmp_forward_privPassword 
- snmp_forward_secLevel 
-</code> 
- 
- 
-=== Configuration Example to forward Traps through SNMP v2c === 
- 
-<code> 
- snmp_forward_trap 1 
- snmp_forward_ip 192.168.1.145 
- snmp_forward_version 2c 
- snmp_forward_community public 
- snmp_forward_secName 
- snmp_forward_engineid 
- snmp_forward_authProtocol 
- snmp_forward_authPassword 
- snmp_forward_privProtocol 
- snmp_forward_privPassword 
- snmp_forward_secLevel 
-</code> 
- 
-=== Configuration example to forward Traps through SNMP v3 === 
-This example is particularly challenging because of the implied knowledge regarding SNMP v3 traps. If the remote SNMP agent were defined in [[en:documentation:start:Configuration#.28.3E.3D_5.X.29_snmp_forward_ip|**snmp_forward_ip**]]. It would contains the following entry in its ''/etc/snmp/snmptrapd.conf'' configuration file: 
- 
-  createUser -e 0x0102030405 myuser MD5 mypassword DES myotherpassword 
- 
-The Pandora FMS server configuration file should look like this: 
- 
-<code> 
- snmp_forward_trap 1 
- snmp_forward_ip 192.168.1.145 
- snmp_forward_version 3 
- snmp_forward_secName myuser 
- snmp_forward_engineid 0x0102030405 
- snmp_forward_authProtocol MD5 
- snmp_forward_authPassword mypassword 
- snmp_forward_privProtocol DES 
- snmp_forward_privPassword myotherpassword 
- snmp_forward_secLevel authNoPriv 
-</code> 
- 
-More information [[http://net-snmp.sourceforge.net/wiki/index.php/TUT:snmptrap_SNMPv3#SNMPv3_TRAPs|**here**.]] 
- 
-==== Snmptrapd daemon independent management ==== 
-If, for some reason, you would like to manage the **snmptrapd** daemon independently from Pandora FMS (stop or start it independently from the main Pandora FMS daemon) then take into account several things: 
- 
-1. The ''snmpconsole'' [[en:documentation:start:Configuration#snmpconsole|el parameter]] must be active for Pandora FMS server.  
- 
-2. Logs configured in Pandora FMS server must be the same as the ones in the independent call to **snmptrapd**. 
- 
-3. The call to **snmptrap** must be in a specific format, **the standard system call cannot be used**. That call must be like the following (the ''-A'' parameter is very important!): 
- 
-<code> 
-/usr/sbin/snmptrapd -A -t -On -n -a -Lf /var/log/pandora/pandora_snmptrap.log -p /var/run/pandora_snmptrapd.pid --format1=SNMPv1[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n --format2=SNMPv2[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n 
-</code> 
- 
- 
-4. The snmptrapd token must be configured in Pandora FMS configuration file: 
- 
-  snmp_trapd manual 
- 
-5. Once this feature is enabled, complete the following procedure:: 
- 
-  * Change the configuration in ''/etc/pandora/pandora_server.conf''. 
-  * Stop the Pandora FMS server. 
-  * Check that the **snmptrapd** process is not running (if it is running, wait until the process stops or "kill" it) 
-  * Start **snmptrapd** manually (with the format specified above). 
-  * Start Pandora FMS Server. 
- 
-=== Trap log file management === 
-The snmptrapd process can be stopped and started without having to stop or start the Pandora FMS server process if the  ''pandora_snmptrap.log.index'' and ''pandora_snmptrap.log'' have not been modified. If those files are modified it is necessary to restart Pandora FMS server. If you need to rotate the trap log files externally, then restart Pandora FMS server, after deleting the two files previously mentioned. 
- 
-==== SNMP trap buffering ==== 
-If SNMP traps are sent to an external manager through an unreliable connection, some information will be lost. Pandora FMS allows you to instead forward traps from a local **snmptrapd** to your Pandora FMS Server in a reliable way. 
- 
-=== Architecture === 
-{{ wiki: Remote snmp trap schema.jpg }} 
- 
-  * An SNMP agent sends traps to a local **snmptrapd**. 
-  * A local Pandora FMS agent reads traps from **snmptrapd** log file and sends them to the designated Pandora FMS Server inside an XML data file, saving it in the XML buffer and retrying it at a later time if necessary.  
-  * The Data Server reads traps from XML data files and dumps them to a plain text file. 
-  * The SNMP Console processes traps from the plain text file. 
- 
-<WRAP center round important 60%> 
-Having the SNMP Console directly process traps from **snmptrapd** log file is more efficient. This setup should only be used if direct connectivity or reliability are a concern. 
-</WRAP> 
- 
-=== Prerequisites === 
-  * A local **snmptrapd** that receives traps. 
-  * A local Pandora FMS Agent. 
-  * A Pandora FMS installation. 
- 
-=== Configuration === 
-== snmptrapd == 
-Edit the ''/etc/snmp/snmptrapd.conf'' file and make sure it logs into a file in a format compatible with Pandora FMS (change the name of the log file if necessary): 
- 
-<code> 
-[snmp] logOption f /var/log/snmptrapd.log 
-format1 SNMPv1[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%a[**]%N[**]%w[**]%W[**]%q[**]%v\n 
-format2 SNMPv2[**]%4y-%02.2m-%l[**]%02.2h:%02.2j:%02.2k[**]%b[**]%v\n 
-</code> 
- 
- 
-== Pandora FMS Agent == 
-Use the **grep_snmptrapd** plugin that comes with the Pandora FMS agent to read data from **snmptrapd** log file. 
- 
-Edit the local agent's configuration file, ///etc/pandora/pandora_agent.conf//, and add the following line, adjusting the path of //snmptrad's// log file if necessary: 
- 
-  module_plugin grep_snmptrapd /var/log/snmptrapd.log 
- 
-== Pandora FMS Server == 
-Tell the SNMP Console to process traps from an external log file that will be written by the Data Server. 
- 
-Edit the server's configuration file, ''/etc/pandora/pandora_server.conf'' and: 
- 
-  * Make sure the SNMP Console is enabled: 
- 
-  snmpconsole 1 
- 
-  * Make sure the Data Server is enabled: 
- 
-  dataserver 1 
- 
-  * Configure an external SNMP log file. If it does not exist, the SNMP Console will create it: 
- 
-  snmp_extlog /var/log/pandora/pandora_snmptrap.ext.log 
- 
-<WRAP center round important 60%> 
-**snmp_extlog** can be any file writable by the Pandora FMS Server, but it must be different from **snmp_logfile** (defined in ''/etc/pandora/pandora_agent.conf'' too). 
-</WRAP> 
- 
-==== Trap Generator ==== 
-With this tool, you can generate custom traps that you can later see in the SNMP console. 
- 
-{{ wiki:Generador_de_traps.png }} 
- 
- 
-In order to be able to correctly configure the trap generator, fill in the following fields: 
- 
-**Host Address** 
- 
- This is the destination IP to which the trap will be sent. 
-  
-**Community** 
- 
- Where the password of the SNMP community you are trying to access with the trap generator must be set. 
- 
-**Enterprise String** 
- 
-  The OID (Object Identifier) of the trap must be configured here. For example: ''1.3.6.1.2.1.2.2.1.8'' 
- 
-**Value** 
- 
- The value desired to be given to the trap and that will then appear as Data. 
- 
-**SNMP Agent** 
- 
- Insert the agent where you will simulate the trap, entering the IP of the same one.  
- 
-**SNMP Type** 
- 
- Choose an SNMP type among the following options: 
- 
-  * **Cold Start**: It indicates that the agent has been started or restarted. 
-  * **Warm Start**: It indicates that the agent configuration has been modified. 
-  * **Link down**: It indicates that the communication interface is out of service (inactive). 
-  * **Link up**: It indicates that a communication interface has been activated. 
-  * **Authentication failure**: It indicates that the agent received a request from an unauthorized NMS (community-controlled) 
-  * **EGP neighbor loss**: It indicates that on the systems where routers were using the EGP protocol, a nearby host is out of service. 
-  * **Enterprise**: This category includes all the new traps. Including traps from suppliers. 
- 
- 
-[[en:documentation:start|Go back to Pandora FMS documentation index]] 
  
ºº