Pandora FMS: What do you know about sending additional information in email alerts?

When monitoring the software components of a server, sometimes you may need to find out some additional information about the performance of the metrics you monitor.

Sending additional information in email alerts

This is all about putting data into context, because 25 is not the same if it is 25% of a CPU, or 25% of a CPU with other three more CPUs, all of them at 100%, or 25% of a CPU of a computer about to be blocked by excess of I/O transactions that has been overloaded for hours, which has risen the system’s temperature to the maximum.

Isn’t it true that “25” can mean many things? Let’s dig a little deeper into it

Continuing with the previous case, suppose that you wish to measure the CPU usage in one of the servers you manage, and where you defined thresholds that force notifications by email to computer managers. 

This threshold (CRIT for CPU > 80) would be reflected in the module configuration as follows:

The alert is defined using the template “Critical condition” and I use the action of sending custom email to send to a specific mailbox.

Okay, so far it’s kind of basic. 

But you may like to be able to send some additional information about the problem, something to collect only when needed, real-time information that is useful for you to find a solution automatically.

But…. Can Pandora FMS do this?

Well, you may include information from other modules in the email using alert macros that reference other modules. 

You may check the full list of macros in our documentation, but we will give you a quick summary of some interesting ones:

moduledata_X_: It will show the data of the  “X” module of the same agent as that of the module that triggers the alert. 

That way, if you have a module called LoadAVG, calling the macro _moduledata_LoadAVG_ you may dump the value of the system load. 

Likewise, to show a graph of the last 24 hours of that module, you should use the macro _modulegraph_LoadAVG_24_ 

*This last thing about the graph is still experimental, until version 772 it will not be available to everyone.

Showing procedures

Perhaps you would like to include textual information about what to do in case your module is set into CRITICAL, as some kind of instructions or procedure. To that end the macro _alert_critical_instructions_ will return the text that you configured in said module for this case.

Showing previous data

You may also be interested in showing the previous data on this monitor before the alert was triggered; to that end you may use the macro  _prevdata_

Showing agent informational custom fields

Imagine that you have a field in the agent that collects data, which tells you the name of the system administrator and their phone number, as custom agent fields with ID 7 and ID 9. 

To that end we would use the macro_agentcustomfield_n_ as macro_agentcustomfield_7_ and macro_agentcustomfield_9_ and thus be able to add to the email the name and phone of the person in charge of the system.

A full example, using macros in an email

Let’s now see an example!

If you wanted to send a somewhat more complete email, edit the email action you are using to add new macros:

Using dynamic information

What if you want to go further? 

What if you want to execute a command every time something happens and collect the value to send it in the email? 

That way, you do not need to have a constant additional metric when monitoring so as not to fill the database with information that is only useful in critical situations.

In order to submit additional information, it will be necessary to create a previous module in your software agent so that it collects the information only when necessary. 

With that purpose in mind, enter the following structure within your module:

module_begin 

module_name MODULE_NAME

module_type async_string 

module_precondition > X COMMAND_TO_RUN_AS_CONDITION

module_exec FINAL_COMMAND_TO_BE_EXECUTED

module_end

• The module will have to be asynchronous and with the type of data that you will collect. 

This is due to the need for this module not to go into an unknown state, thus giving a false perception of failure, when it does not meet the module’s precondition.

• A condition expression will have to be incorporated so that you may run the module normally. 

For that purpose, run a previous command from which to obtain the numerical information to obtain the data. Following the example of the CPU described above (which is higher than 80%) it would be something like:

• module_precondition > 80 uptime | awk -F “,” ‘{print $3}’ | awk ‘{print $3}’ | tr d “\n”
• Finally, use the command to obtain such additional information in order to attach it to your email.

In the previous case, it would be convenient to know the processes that are using more CPU (top 5 processes), so you may enter the following case:

• module_exec top -n1 | head -12

So in the previous example there would be a module as follows:

module_begin 

module_name TOP5_CPU_Proc_Usage

module_type async_string 

module_precondition > 80 uptime | awk -F “,” ‘{print $3}’ | awk ‘{print $3}’ | tr d “\n”

module_exec top -n1 | head -12

module_end

Once you create the module that will only be executed when necessary, enter this information in your email.

 For that purpose use the following macro within the email:

_moduledata_TOP5_CPU_Proc_Usage_ 
This is only a way of using it to send the additional information, you may use it for other cases such as for example if the disk usage of a server is reaching the limit, you may show the five heaviest files saved on the server using the command  “du -Sah/ | sort -rh | head -5”

Do you want to know more?