Table of Contents
Monitoring with Software Agents
Monitoring with Software Agents
Monitoring with software agents
Software agents are in execution in OS where they get information from. Each of the checks performed on the system, such as CPU usage, free memory or disk space correspond to a module. So for each module a single data is collected in each execution.
the software agent's own directives are useful to retrieve certain data directly from the operating system (e.g. CPU usage, memory, events, etc.), executing the operating system's own commands following instructions from predefined scripts. It is also possible to execute those commands directly as well as any other software as long as data are returned in a standard way.
Pandora FMS Dataserver processes and stores in the database all the information generated by software agents, whcih send their data through and XML file.
Logical outline of an agent/physical agent.
If versions prior to 7 NG are executed, check software agent naming at the end of this article.
Agent Configuration
All the configuration and monitoring parameters of the software agents can be found in their configuration file pandora_agent.conf. This is stored locally in the machine where the software agent is installed, so any modification to be made in the agent must be reflected in this file. You have a detailed description of all agent configuration tokens in the chapter “PandoraFMS Agent Configuration” 1) while here we will only focus on the advanced uses of some of them.
Local configuration
In the software agent's configuration file, modules are defined with the following text basic structure:
module_begin module_name <your module name> module_type generic_data module_exec <your command> module_description <your description> module_end
module_name
: The module name.module_exec
: Command to be executed.module_description
: Description of the task to monitor.
Example 1
module_begin module_name Files in var spool module_type generic_data module_exec ls /var/spool | wc -l module_description Number of files incoming dir module_end
In the *nix environment, the command ls lists directory files and is executed with the line module_exec to deliver the value to the wc command, which will count the amount of words received for the same number of files. The value returned by this last execution will be the data that the module will obtain and will be displayed in the monitoring.
For the Software Agent on MS Windows® and module_name
instruction, if you want or need to use extended ASCII characters (áéíóú
, for example) you must use an external plugin or script. See the Software Agent plugin section.
Example 2
module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'
The vmstat command reports virtual memory statistics. In this examples there are two additional commands to “refine” the desired information. It is recommended to first launch the command manually and analyze the output.
$> vmstat 1 2 | tail -1 | awk '{ print $13 }'
If the result satifies the requirement, it will be possible to add it to the configuration file. Later on the value returned by the execution through the software agent will be stored in the XML as module data.
Example 3
Any command or software can be executed through module_exec while the output supports the values accepted by Pandora FMS (numeric, alphanumeric or boolean), so it is possible to indicate custom scripts:
module_exec myScript.pl --h 127.0.0.1 -v cpu
Again, the agent will execute the shell and will retrieve the result, as if it was executed by an operator:
$> myScript.pl --h 127.0.0.1 -v cpu
Remote Configuration
On the Enterprise version, there is a remote Agent Configuration feature which allows centralized configuration and file management from the server console. This allows centralized management of all our software agents without the need to physically access the systems where they are installed.
The configuration consists of two files. Their file names are <md5>.conf and <md5>.md5, where <md5> is the agent's name hash code.
Those files are stored in:
/var/spool/pandora/data_in/conf
and
/var/spool/pandora/data_in/md5
folders respectively.
The console is in charge of keeping said files synchronized in Pandora FMS server and the local ones accordingly, where each software agent is installed.
To enable remote configuration, enable the corresponding parameter in the agent's local configuration file first. From this moment on, all the changes must be made from Pandora FMS console:
remote_config 1
Once the agent's remote configuration is enabled, any changes made locally in the configuration file will be overwritten by the configuration stored in the console. If you want to prevent this from happening, stop the agent, modify the configuration file, disable the remote configuration remote_config
to zero and launch the agent again.
Custom Fields
Custom fields are an easy way to add additional agent information. Create custom fields by clicking on ►Resources → Custom fields.
You can include links in custom fields using the following tags:
[url]< link >[/url]
or
[url=< link >]< web name >[/url]
.
Display up front and Enabled combo fields are disabled by default:
- By activating the field Display up front, custom field information will be displayed in the agent's general view as shown below. In addition, enable this token to send Custom Fields information to the Metaconsole and be able to see it in the agent view and work in Custom Field View with this data.
- Enabled combo: This parameter allows you to activate the configuration of selectable parameters from a drop-down list. Once activated, a new field will appear in the configuration window of the corresponding custom field to enter the combo values separated by commas.
If the ♦Enabled combo parameter is enabled, ♦Password type will be disabled.
Custom fields can also be retrieved from the agent configuration file, using the following configuration token:
custom_field1_name Model custom_field1_value i386
Common Configuration Parameters
Most important parameters for basic agent configuration (more details in Pandora FMS Software Agents):
- server_ip: IP address of Pandora FMS Server.
- server_path: Path of the 'incoming' folder for the Pandora FMS server,
/var/spool/pandora/data_in
by default. - temporal: Software agent's temporal folder,
/tmp
by default. - logfilefile: Software agent's log file,
/var/log/pandora/pandora_agent.log
by default. - interval: Agent's execution interval,
300
by default. - agent_name: Agent name,
hostname
by default. - remote_config: Activation of remote configuration. It is disabled (
0
) by default. Only for Enterprise version. - debug: When activated (value
1
) it sets thedebug
mode, in this way, a copy of the XML is created and sent to the server and stored in the temporary directory for analysis. In addition, on MS Windows® systems, a.debug
file is created in the Agent installation path with a detailed log of the execution of each Module. In Enterprise versions, the activedebug
mode disables remote configuration from the Console to the Software Agent.
The active debug
mode is not designed for prolonged use. It is a mode for debugging errors for short periods of time. It is important to remember to disable it as soon as debugging is finished.
An example in a *nix environment:
server_ip 192.168.1.1 server_path /var/spool/pandora/data_in temporal /tmp logfile /var/log/pandora/pandora_agent.log interval 300 debug 0 agent_name box01 server_port 41121 transfer_mode tentacle remote_config 1
An example in a MS Windows® environment:
server_ip 192.168.1.1 server_path /var/spool/pandora/data_in temporal "%ProgramFiles%\pandora_agent\temp" logfile "%ProgramFiles%\pandora_agent\pandora_agent.log" interval 300 debug 0 agent_name box02 server_port 41121 transfer_mode tentacle remote_config 1
Password protected groups
By default, when an agent sends data for the first time to the Pandora FMS server, it is automatically added to the group that has been defined in the agent configuration file. This means that, in practice, anyone can add an agent to a group if they know the group name. This could be a problem if several clients share their Pandora FMS instance or if you want to control what is in each group.
We can optionally configure a password for a group from the Pandora FMS Console. An agent will not be added to a group unless the correct password has been specified in the agent configuration file.
Example
To set a password for a group, navigate to the group editor and click on edit, enter the group password and save your changes:
To add a new agent to this group, edit your configuration file and add the following configuration option:
group_password <password>
Do not forget to restart the agent to make the changes effective. The agent should be created correctly in the Pandora FMS console.
Modules in agents and software agents
Types of Modules
The following are the possible types of modules in software agents depending on the type of data returned:
- generic_data: Numerical and floating point data.
- generic_data_inc: A kind of increasing numerical data. Stores the difference between the previous and current data divided by the elapsed time in seconds, showing the rate per second. This type of data is used to count “number of times per second” of something, such as log entries/sec, receivedbytes/sec , incoming connections/sec , etc.
- generic_data_inc_abs: Type of absolute increasing numerical data. It stores the difference between the previous and current data, without dividing it between the elapsed seconds, so the value will correspond to the total increase between the two executions, and not to the increase per second. This type of data is used to count the number of times something happens, such as log entries, total received bytes, number of incoming connections, and so on.
- generic_proc: Boolean type of data, where a value of 0 means False or incorrect, and values above zero mean True or correct. The generic_proc types have the critical (0) and correct (1 or higher) states preconfigured.
- generic_data_string: Kinds of alphanumeric data (text).
- async_data: It is a kind of asynchronous numeric data. It is the same as 'generic_data' but for asynchronous data which is only updated if there is a change. The asynchronous kind of data do not have a defined periodicity when data can be obtained.
- async_string: This is a kind of asynchronous alphanumeric data. It is the same as 'generic_string' but for asynchronous data which are only updated if there is a change. It is the kind of data that you are recommended to use if you want to monitor searches in logs or event viewers. New data can be obtained at any moment or not for several days.
- async_proc: It is a kind of asynchronous boolean data. It is the same as 'generic _proc' but for asynchronous data which are only updated if there is a change.
- Image module: They are based on a text string type module (generic_data_string or async_string). If the data in the module is a base64 image, in other words, part of the string contains “data:image”, it will be identified as an image and, on the views that it appears, it will enable a link to open a window to display the image. Also on its historical data the strings that build/generate the images will be saved and displayed.
Intervals in local modules
The local modules (or software agent modules) all have the interval of their agent as a “base”. However, they can take values that are multiple from that base if the module_interval parameter is modifies by multiplying with an integer higher than zero; for example:
module_interval 2
If an agent has an interval of 300. The interval's module will be 300×2 (600).
Module Creation Interface
Feature exclusive for the Enterprise version; the according Software agent's remote configuration must be enabled.
Console local module creation is done through a form where, inn addition to the common configuration of any module (thresholds, type, group, etc. it has a text box where to specify the configuration data to be set in the Software agent configuration file.
- By clicking on Load basic (template), the content of Data configuration will be deleted with a basic template that you must modify according to your monitoring needs.
- Once modified, by clicking on Check (syntax) it will verify that the template's syntax stays correct, however the rest of the commands will not be checked.
When a module is loaded from a local component, it may have macros. If it has macros, the configuration box will stay hidden and a field will appear for each macro, see more information in Templates and components
Conditional Monitoring
Post-Conditions
Pandora FMS software agent supports the execution of commands and scripts as post-conditions. This means that actions could be performed depending on the value obtained in the execution of the module.
Example 1 With the module_condition parameter, a value or range of values and the execution to be carried out must be indicated in case the obtained data meets the terms (CPU usage under 200%):
module_begin module_name CPU_Usage_Condition module_type generic_data module_exec get_cpu_usage.pl module_condition < 20 add_processes.sh module_end
Example 2 You can specify multiple conditions for the same module, in a range and with a minimum threshold (mathematically, one or none of both options is carried out):
module_begin module_name CPU_Usage_Condition module_type generic_data module_exec get_cpu_usage.pl module_condition (90, 100) remove_processes.sh module_condition < 20 add_processes.sh module_end
Example 3 Similar to the previous example, but both conditions can be executed or one or none (try with selected values: if it is 5, 15 or 30):
module_begin module_name CPU_Usage_Condition module_type generic_data module_exec get_cpu_usage.pl module_condition < 10 start_new_server.sh module_condition < 20 add_processes.sh module_end
Pre-Conditions
The module_precondition parameter defines a precondition to evaluate before a module execution. Depending on the result of this precondition, the software agent will execute the module or not.
Example 1
According to CPU usage, if the active processes are more than ten, obtaining the CPU usage percentage and reporting to Pandora FMS server:
module_begin module_name CPU_Usage module_type generic_data module_precondition> 10 number_active_processes.sh module_exec get_cpu_usage.pl module_end
Example 2
You can define multiple preconditions for the same module and all of them must be met:
module_begin module_name CPU_Usage module_type generic_data module_precondition> 10 number_active_processes.sh module_precondition> 1 important_service_enabled.sh module_exec get_cpu_usage.pl module_end
In this case, the module is executed only if there are more than ten active processes and if at least one of them is an important process.
Intensive Monitoring
There are certain specially important modules, such as critical running processes or services. Intensive monitoring enables more controlled monitoring of these particular cases.
It consists of warning in a shorter interval that a problem has arisen without reducing the agent's general interval.
Software agent configuration:
- Interval: agent sampling time in seconds. This is the general range for all local modules. Required parameter.
- Intensive_interval: time in which you will be notified of a problem on the especially critical modules. Optional parameter.
Module configuration:
- module_intensive_condition = 0: if the module obtains as a result the value indicated in this parameter (in this case 0), it will be notified in the intensive interval defined in the agent.
Example
The sshd service is very important since it is used to connect by shell remotely, we need to monitor its working:
intensive_interval 10 interval 300
module_begin module_name SSH Daemon module_type generic_data module exec ps aux | grep sshd | grep -v grep | wc -l module_intensive_condition = 0 module_end
If the service fails, you will be notified in the next 10 seconds. If the service is up, you will be notified in the next 5 minutes, like normally (normal interval, 300 seconds).
Programmed Monitoring
The software agent supports the definition of programmed modules which are executed in the defined instances. The syntax used is the same as crontab. An example of module definition to execute it all Mondays from 12 to 15 hours:
module_begin module_name crontab module_type generic_data module_exec script.sh module_crontab * 12-15 * * 1 module_end
To execute it in minute 10 of each hour:
module_begin module_name crontab module_type generic_data module_exec script.sh module_crontab 10 * * * * module_end
Note that if you use an interval that causes the module not to report data, this module will go into “unknown” status. Use asynchronous modules for these cases.
Remote Checks with Software Agents
When Pandora FMS main server does not have access to carry out remote checks (generally for security reasons), a software agent is able to take its place for such reasons and can even be distributed in broker agents.
ICMP Checks
ICMP or ping checks are very useful to know whether a machine is connected to a network or not. In this way, a single software agent could easily monitor the status of all machines.
UNIX
Using the system commands (all parameters in the “command line” module_exec
):
module_begin module_name Ping module_type generic_proc module_exec ping -c 1 192.168.100.54>/dev/null 2>&1; if [ $? -eq 0 ]; then echo 1; else echo 0; fi module_end
Note: Replace 192.168.100.54
by the IP address to be monitored.
MS Windows®
The parameters must be specified in module_ping_count
(number of packets, 1 by default) and module_ping_timeout
(time limit in seconds, 1
by default); example:
module_begin module_name Ping module_type generic_proc module_ping 192.168.100.54 module_ping_count 2 module_ping_timeout 5 module_end
Note: module_advanced_options
allows advanced options for ping.exe
.
TCP Checks
TCP checks are useful to verify whether a port of a host stay open and allow to find out whether an application connects or not to the network.
UNIX
With the nmap command and its configuration parameters in the command line, to an IP address check whether port 80 is open (response waiting time of 5 seconds):
module_begin module_name PortOpen module_type generic_proc module_exec nmap 192.168.100.54 -p 80 | grep open > /dev/null 2>&1; echo $?; if [ $? == 0 ]; then echo 1; else echo 0; fi module_timeout 5 module_end
MS Windows®
Parameters must be specified in:
- module_tcpcheck: Host to be checked
- module_port: Port to be checked
- module_timeout: Timeout for the check
Example:
module_begin module_name TcpCheck module_type generic_proc module_tcpcheck 192.168.100.54 module_port 80 module_timeout 5 module_end
SNMP Checks
SNMP checks are commonly used to monitor network devices to check the interface status, inbound/outbound bytes, etc.
UNIX
If you are using the software agent for UNIX platforms, you may create the module using the snmpget command like this:
module_begin module_name SNMP get module_type generic_data module_exec snmpget 192.168.100.54 -v 1 -c public .1.3.6.1.2.1.2.2.1.1.148 | awk '{print $4}' module_end
This module returns the value for OID .1.3.6.1.2.1.2.2.1.1.148 on the '192.168.100.54' host.
MS Windows®
Parameter coniguration:
- module_snmpversion [1,2c,3]: SNMP version (Default value is '1').
- module_snmp_community <community» SNMP community (Default value is 'public').
- module_snmp_agent <host» The host to monitor.
- module_snmp_oid <oid» OID.
- module_advanced_options: Advanced options for 'snmpget.exe'.
Example that does the same as the previous example:
module_begin module_name SNMP get module_type generic_data module_snmpget module_snmpversion 1 module_snmp_community public module_snmp_agent 192.168.100.54 module_snmp_oid .1.3.6.1.2.1.2.2.1.1.148 module_end
Proxy Mode
To use Pandora FMS agent's proxy mode on Linux or UNIX systems, the agent must -not- be executed by a root user ! You are required to perform a custom installation of the Pandora FMS agent to do so. You may look up all the details about custom installations in the section Custom Agent Installation.
Pandora FMS Software Agents have a Proxy Mode which allows them to act other software agent proxies, redirecting the communication of several agents to the Pandora FMS Server. The software agent with an enabled proxy mode is able to perform monitoring tasks too.
The Proxy Mode was created for local area network where a single computer is exposed to the Internet, where Pandora FMS server is. It is necessary to monitor with software agents the rest of computers of that network; other computers will communicate with the proxy instead of with the server. The proxy mode also supports the Remote configuration and File collection features.
Parameter configuration:
- server_ip: IP of the Pandora FMS Server.
- proxy_mode: Enabled (1) or diabled (0).
- proxy_max_connection: Maximum number of simultaneous connections for the proxy. The default value is '10'.
- proxy_timeout: Proxy timemout. The default value is '1' (in seconds).
- proxy_address: Address in which the proxy listens.
- proxy_port: Port in which the proxy listens.
Example:
server_ip 192.168.100.230 proxy_mode 1 proxy_max_connection 20 proxy_timeout 3
To redirect the connection of a software agent, enter as Pandora FMS server address that of the agent with the Proxy Mode activated.
For example, the software agent in proxy mode has the IP address 192.168.100.24, the rest of the software agents must be configured with:
server_ip 192.168.100.24
Broker Mode
The software agent has a Broker Mode which allows one agent to monitor and manage the configuration as if there were several software agents installed:
When the broker mode is activated in a software agent, a new configuration file is created. From that moment on, the original software agent and the new broker will be managed separately with their independent configuration files, as if they were two completely separate software agents on the same machine.
The main features of the Broker Mode are:
- Sending local data as another agent. Very useful to monitor different software instances as different agents.
- Sending the collected data from the remote checks to other machines as if a software agent had been installed on them.
To create a broker, add a line with the broker_agent <broker_name> parameter. It is possible to create as many broker agents as you wish, just by adding the corresponding broker_agent lines, as follows:
broker_agent dev_1 broker_agent dev_2
Once the brokers are created, the 'dev_1.conf' and 'dev_2.conf' configuration files will be created with the same content as in the original software agent, but with their corresponding name. By adding or deleting modules from 'dev_1.conf' and 'dev_2.conf' configuration files, you can customize the checks performed by the brokers.
On the Pandora FMS web console the brokers appear and will be managed independent agents, which means that if you have a software agent installed with two brokers, you will see three different agents with their modules, configurations, etc. on the web console.
NOTE: Broker agent instances cannot use file collections. If you want to use collections, distribute them and/or use them in the “real” agent that is used as a basis for the broker agent, not in one of its instances.
Modules that save data in memory between executions (module_logevent and module_regexp in MS Windows®) do not work when there are broker agents configured.
Broker mode use Examples
Monitoring a local Database as a different Agent
As an example, there is a software agent installed that monitors the CPU, memory and disk of a computer that in addition executes a database. For independent monitoring, add the line:
broker_agent DBApp
With that you create a broker agent with name DBApp that generates the configuration file dbapp.conf. There add, to monitor the database (number of connected users and number of slow connections):
module_begin module_name Num Users module_type generic_data module_exec get_db_users.pl module_end module_begin module_name Num slows queries module_type generic_data module_exec get_db_slows_queries.pl module_end
Pandora FMS console will show one with the name of the machine and CPU, memory and disk modules, and in addition another called DBApp with the modules Num Users and Num slows queries.
Monitoring Devices Remotely Using Brokers
As an example, there is a software agent installed in a machine with MS Windows®, that monitors CPU, memory and disk. You need to monitor a router with IP 192.168.100.54 without installing an agent on it. For that create a broker using the following parameter:
broker_agent routerFloor5
With that you create the broker agent named as routerFloor5'. Then in the file routerFloor5.conf, modify the lines to store the ping and snmp modules available:
module_begin module_name Ping module_type generic_proc module_ping 192.168.100.54 module_ping_count 2 module_ping_timeout 500 module_end module_begin module_name Eth 1 up module_type generic_data module_snmpget module_snmpversion 1 module_snmp_community public module_snmp_agent 192.168.100.54 module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1 module_end module_begin module_name Eth 2 up module_type generic_data module_snmpget module_snmpversion 1 module_snmp_community public module_snmp_agent 192.168.100.54 module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2 module_end
The web console will show two agents: one is the Windows machine with the CPU, Memory and hard drive modules and the other one is routerFloor5 with the modules named “Ping”, “Eth 1 up” and “Eth 2 up”.
Monitoring inaccessible networks remotely
In some cases, you need to monitor devices remotely where the Pandora FMS Remote Server cannot access them directly.
The software agent in broker mode allows sending XMLs to Pandora FMS server as if they were different devices. For that you may add as many brokers as devices to be monitored, for example:
broker_agent device_1 broker_agent device_2 broker_agent device_3 broker_agent device_4 ...
Once the brokers are created, the monitoring for each device can be customized by modifying the configuration file of each broker as explained for each agent in remote check mode.
Shared Monitoring Load through Brokers
The capacity of Pandora FMS remote server is around 2000 agents. Working with Broker agents you may raise it to 3000 and free the main server from most of the work. In the graph, each of the networks has a software agent with broker mode enabled, there you may create as many brokers as devices you have to monitor. For example, configuration for Broker_Agent_Net_A agent would be:
broker_agent device_1 broker_agent device_2 broker_agent device_3 broker_agent device_4 ...
In addition, for each of the brokers, you would need to add the corresponding modules to monitor the devices as explained before.
Inventory using Software Agents
Pandora FMS Software Agents support inventory features for both hardware and software. The inventory system allows to keep a history of CPU, cards, RAM memory, patches, software, etc, used in the company servers. Furthermore, it is possible to generate alerts if there is a change in the inventory, e.g. if a disk was replaced or an application was uninstalled.
For further information on the subject, please have a look at the section Local Inventory through Software Agents.
UDP remote commands
A software agent is capable of receiving remote requests and executing orders.
Bear in mind that UDP is unsafe by nature (but efficient to send messages without compromising a true response).
To allow Pandora FMS server to send order to Software agents in charge of it, configure:
- udp_server: it enables (1) or disables (0) this feature.
- udp_server_port: listening port of the UDP server in the software agent.
- udp_server_auth_address: IP address of Pandora FMS server.
Restart the software agent to apply changes.
Although it may be set to 0.0.0.0 for it to accept from all sources, said practice is not recommended. If you have serveral Pandora FMS servers and/or use IPv6, you may set different IPs separated by commas. For example, if you have in IPv6 2001:0db8:0000:130F:0000:0000:087C:140B
, its abbreviation is 2001:0db8:0:130F::87C:140B
use both separated by commas.
How to request software agent service restart
Use the udp_client. pl script, present in the Pandora FMS server, and normally located in /usr/share/pandora_server/util. It can be run from the command line or used in an alert, making use of the command that is pre-configured in the “Remote agent control” console.
There is also a default alert action called Restart agent, on this script, using the action REFRESH AGENT.
Then force the alert's execution or force an incorrect status of the module for the alert to fire and thus check configuration.
Custom remote actions
Apart from the Refresh agent command, you can specify new and custom actions. For that, add a line for each command to execute, like the following (replace < order_name >
with appropiate value):
process_< order_name >_start comando
For example, if you want a remote order to start the sshd service:
process_sshd_start /etc/init.d/sshd start
Then create a new alert action at Pandora FMS Console for each remote command you made. You can copy the “Remote agent control” action, which is already prepared to send UDP commands. Set “START PROCESS sshdproc” on Field 1, as seen on the screenchot.
Now, you only need to set a new manual alert with the new alert action on the agent whose sshd service you wish to start. When the alert is forced, the order will be launched and the agent will start the service.
Custom orders can also be created to execute scripts. This allows a huge variety of remote actions to be performed on a remote agent just by clicking a button.
Plugins in software agents
They are characterized by performing complex advanced checks from the software agents, being able to return several modules as a result instead of a single value. Unlike the server plugins, which are executed by Pandora FMS server, agent plugins return their data in an XML, reporting one or several modules at the same time.
Execution on Windows systems
In Windows, all the default plugins are programmed in VBScript. To run them, it is vital to use the appropriate interpreter indicating the full path.
Here are some examples of how to use the default plugins included in the Windows agent:
module_plugin cscript.exe //B "%ProgramFiles%\pandora_agent\util\logevent_log4x.vbs" Aplication System 300 module_plugin cscript.exe //B "%ProgramFiles%\pandora_agent\util\df.vbs" module_plugin cscript.exe //B "%ProgramFiles%\pandora_agent\util\ps.vbs" iexplore.exe myapp.exe
The Windows agent includes several ready-to-use plugins.
Execution on Unix systems
Unix plugins are by default in the directory “/etc/pandora/plugins” of the agent directory, so they are invoked and then the necessary parameters are sent:
module_plugin grep_log /var/log/syslog Syslog . module_plugin pandora_df tmpfs /dev/sda1
The Unix software agent comes with several plugins by default ready to work.
Software agent plugin management from the Console
In Enterprise version, it is possible to manage without directly editing the configuration file. When having remote configuration enabled, a software agent in its administration view will have the plugin editor tab.
This section shows the list of plugins enabled within the agent, and allows deleting, adding and disabling them. Regarding policy plugins, it may be useful to deactivate them because when applying the policy again they will stay disabled.
Plugins managed by this editor may be, in turn, edited from the agent's configuration file.
Example 1
Plugins for the software agent can return a piece of data or a group of data. An example of a plugin that returns a piece of data can be ps. vbs in a Windows environment, which simply checks whether a process is running.
module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" IEXPLORE.EXE
The result will be a module that returns 0 if the process is not active and 1 if it is active:
<module> <name><![CDATA[IEXPLORE.EXE]]></name> <description><![CDATA[Process IEXPLORE.EXE status]]></description> <data><![CDATA[1]]></data> </module>
Example 2
The plugin df. vbs in a Windows environments returns the free space in each storing device with the following order:
module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"
Result:
<module> <name><![CDATA[C:]]></name> <description><![CDATA[Drive C: free space in MB]]></description> <data><![CDATA[805000]]></data> </module> <module> <name><![CDATA[D:]]></name> <description><![CDATA[Drive D: free space in MB]]></description> <data><![CDATA[90000]]></data> </module>
Advanced agent Plugin Management from the Console
Version NG 750 or later.
It is possible to add a token in the configuration of the plugin agent that when enabled allows the option of encapsulating the plugin definitions within the tags module_begin
and module_end
.
This enabled token allows inserting configuration clocks such as module_interval
or module_crontab
, among others.
To enable this token, just go within agent management to agent plugin item and at the top of the configuration, you will find it under the name “Advanced”.
How to create custom software agent plugins
Plugins can be created in any programming language. Just bear in mind the general rules and the specific rules for its development.
Make sure you finish the output of your plugin (if it is a script) with an errorlevel 0
, or the agent will think that the plugin has had an error and was not able to be run.
Shellscript (Linux/Unix) plugin example
#!/bin/bash # Detect if local Mysql is without password # First, do we have a running MySQL? CHECK_MYSQL=`netstat -an | grep LISTEN | grep ":3306 "` if [ ! -z "$CHECK_MYSQL" ] then CHECK_MYSQL_ROOT=`echo "select 1234" | mysql -u root 2> /dev/null | grep 1234` if [ -z "$CHECK_MYSQL_ROOT" ] then echo "<module>" echo "<type>generic_proc</type>" echo "<name>mysql_without_pass</name>" echo "<data>1</data>" echo "<description>MySQL have a password</description>" echo "</module>" else echo "<module>" echo "<type>generic_proc</type>" echo "<name>mysql_without_pass</name>" echo "<data>0</data>" echo "<description>MySQL do not have a password</description>" echo "</module>" fi fi exit 0
VBScript (Windows) plugin example
' df.vbs ' Returns free space for available drives. ' -------------------------------------- Option Explicit On Error Resume Next ' Variables Dim objWMIService, objItem, colItems, argc, argv, i ' Parse command line parameters argc = Wscript.Arguments.Count Set argv = CreateObject("Scripting.Dictionary") For i = 0 To argc - 1 argv.Add Wscript.Arguments(i), i Next ' Get drive information Set objWMIService = GetObject ("winmgmts:\\.\root\cimv2") Set colItems = objWMIService.ExecQuery ("Select * from Win32_LogicalDisk") For Each objItem in colItems If argc = 0 Or argv.Exists(objItem.Name) Then If objItem.FreeSpace <> "" Then Wscript.StdOut.WriteLine "<module>" Wscript.StdOut.WriteLine " <name><![CDATA[" & objItem.Name & "]]></name>" Wscript.StdOut.WriteLine " <description><![CDATA[Drive " & objItem.Name & " free space in MB]]></description>" Wscript.StdOut.WriteLine " <data><![CDATA[" & Int(objItem.FreeSpace /1048576) & "]]></data>" Wscript.StdOut.WriteLine "</module>" Wscript.StdOut.flush End If End If Next
Using Nagios plugins from the agent
Nagios has a large number of plugins that can be used with Pandora FMS. One way to do this is using remote plugins with the Plugin Server, using Nagios compatibility. But in this way, you will only get the statuses, since it does not use the descriptive output that some plugins for Nagios have.
Using the wrapper to use Nagios plugins in the software agent will solve this problem. The wrapper comes by default with the Unix 3.2 agent. An equivalent plugin for Pandora FMS Windows agents can be downloaded from Pandora FMS resource library.
General performance
The wrapper executes the Nagios plugin, using its original parameters and turning the output into useful data for Pandora FMS. It has two types of information:
- Status information: taking into account Nagios error levels: NORMAL (1), CRITICAL (0), WARNING (2), UNKNOWN () and others (4). By default, they will use a proc module, so the NORMAL and CRITICAL values are working “by default”. If you wish to have information about WARNING and other values, you must configure the module thresholds manually.
- Descriptive information: generally string information. It will be placed in the module description field. Usually something like:
<![CDATA["OK: successfully logged in"]]>
Monitoring with KeepAlive
There is a special module in Pandora FMS called keep_alive
used to alert about a software agent not sending information anymore (see previous Remote actions through UDP). This alert takes place when it has not updated its last contact date for twice of its interval, firing and checking the monitor in critical status.
KeepAlive modules can be created themselves from the console (although you may not have remote configuration enabled) and they do not leave any trace in pandora_agent.conf
file.
Creation of a new KeepAlive module:
Performance in “NORMAL” status (green), “NOT INITIALIZED” (blue):
If the agent stops sending data (for this example there is a 1-minute interval), then it will automatically be triggered and change to CRITICAL status (red). To display this type of modules you may go to the left side menu and select Monitoring → Views → Monitor detail (Pandora FMS interface) and in the filter dialog, in field Data type, select KeepAlive and then click Show:
KeepAlive module works like any other module: they can have an alert associated and it may be used for other elements such as reports, maps, etc.
Command screenshot monitoring
Commands that have extensive outputs, such as top or netstat can be captured completely by a module and fully reproduced. The module must be configured as a text type.
In order for it to work like this, it is necessary to configure properly both Pandora FMS console (setup) and the agent that collects this information, making sure that it is untreated text.
In the console, activate the option:
Image monitoring and visualization
This method allows you to define string type modules (generic_data_string
or async_string
) that contains images in text format with base64 encoding, being able to display that image instead of a specific result. This is stored as text information, and displayed in a different way, not as simple data, but by means of reconstructing an image when clicking in the special icon for screenshots:
To capture these images, just type a plugin that sends all the data, generating the necessary XML tags, and running the plugin as such, with the module_plugin directive. Example:
#!/bin/bash echo "<module>" echo "<name>Actual leader</name>" echo "<type>async_string</type>" echo "<data><![CDATA[data:image/jpeg;base64,/9j/4AAQSkZ....]]></data>" echo "</module>"
(The previous data would be generated by a device/application rendering images in base64.)
Save that content in a file in the agent (or distribute it with file collections) and run it as follows:
module_plugin < complete path to the file >
Specific Monitoring for Windows
The software agent for Windows has specific features to make monitoring a lot easier. These features are explained with some examples. Common rules:
If the name of the process contains blank spaces, do not use “ ”
. The name of the process must be the same shown in the Windows task administrator ( taskmngr
), including the extension .exe
; it is important to respect uppercase and lowercase.
Processes monitoring and process watchdog
Process monitoring
The parameter module_proc verifies whether a process with a preset name is running on this machine. The module definition is:
module_begin module_name CMDProcess module_type generic_proc module_proc cmd.exe module_description Process Command line module_end
If you want the software agent to immediately notify you if a process is not working, add the parameter module_async yes. In this case, the module definition would be:
module_begin module_name CMDProcess module_type generic_proc module_proc cmd.exe module_async yes module_description Process Command line module_end
Watchdog Process
The watchdog feature on Pandora FMS Agent for MS Windows® allows immediate response to the failure of a process and restarts it.
Example:
module_begin module_name Notepad module_type generic_data module_proc notepad.exe module_description Notepad module_async yes module_watchdog yes module_user_session yes module_start_command "%SystemRoot%\notepad.exe" module_startdelay 3000 module_retrydelay 2000 module_retries 5 module_end
Each time the notepad.exe process is deactivated and the command:
%SystemRoot%\notepad.exe
will be executed (see common rules at the beginning of the Windows section). The process reactivation will be attempted 5 times with an initial waiting time of 3 seconds and a waiting time between retries of 2 seconds in the user's active session.
Service monitoring and service watchdog
Service monitoring
The module_service parameter verifies whether a specified service is running on the machine. The definition of this module is as follows:
module_begin module_name Service_Dhcp module_type generic_proc module_service Dhcp module_description Service DHCP Client module_end
If you want the software agent to warn you immediately when a service is down, add the parameter module_async yes (see common rules at the beginning of the Windows section):
module_begin module_name Service_Dhcp module_type generic_proc module_service Dhcp module_description Service DHCP Client module_async yes module_end
Service watchdog
It works similarly to the process watchdog. Example:
module_begin module_name ServiceSched module_type generic_proc module_service Schedule module_description Service Task scheduler module_async yes module_watchdog yes module_end
The watchdog definition for services has no need for any extra parameters because they are incorporated in the service definition.
Basic Resource Monitoring
This section describes how to monitor the basic variables of a Windows-based machine.
CPU Monitoring
The parameter module_cpuusage returns the CPU usage percentage. It is possible to monitor the CPU based on its ID with the following module definition:
module_begin module_name CPU_1 module_type generic_data module_cpuusage 1 module_description CPU usage for CPU 1 module_end
It is also possible to monitor the average CPU usage from all systems with the following module:
module_begin module_name CPU Usage module_type generic_data module_cpuusage all module_description CPU Usage for all system module_end
Memory Monitoring
To monitor the memory, you can use two parameters: module_freememory which returns the amount of free memory in the system and module_freepercentmemory which returns the percentage of free memory.
Example module for module_freememory:
module_begin module_name FreeMemory module_type generic_data module_freememory module_description Non-used memory on system module_end
An example module for module_freepercentmemory:
module_begin module_name FreePercentMemory module_type generic_data module_freepercentmemory module_end
Hard drive monitoring
To monitor hard drive space, you may use two parameters: module_freedisk which returns the amount of available space and module_freepercentdisk which returns the percentage of available space. Both parameters require the monitored unit as an input. Do not forget the character :
, for example:
module_begin module_name FreeDisk module_type generic_data module_freedisk C: module_end
Module example for module_freepercentdisk:
module_begin module_name FreePercentDisk module_type generic_data module_freepercentdisk C: module_end
WMI queries
Pandora FMS Software Agent allows you to retrieve information by using WMI queries, which is a source of data widely used to obtain external or system-related information.
The software agent allows you to execute any local WMI query you want using the module_wmiquery parameter. To perform the query, WMI query is defined in the module_wmiquery parameter and the column that contains the information to be monitores with the module_wmicolumn parameter.
For example, getting a list with the installed services:
module_begin module_name Services module_type generic_data_string module_wmiquery Select Name from Win32_Service module_wmicolumn Name module_end
Get the current CPU load using WMI:
module_begin module_name CPU_Load module_type generic_data module_wmiquery SELECT LoadPercentage FROM Win32_Processor module_wmicolumn LoadPercentage module_end
Versions prior to 7 NG
Name of the agents
From Pandora FMS version 7, agents have an alias and a name or (single identifier). An agent configured by default will generate a name (or identifier) based on a pseudorandom hexadecimal string, and an alias (or visible name) based on the machine's hostname.
In previous versions, there was only the “name” of the machine, and the previous system fully supports Pandora FMS most modern versions, but if in the same Pandora FMS installation there are two agents with the same identifier (or names), the data from both data will get mixed or overwritten. That is why from version 7, the possibility of adding agents with different name but same alias was added.
To change this performance, use the following configuration tokens:
pandora_agent pandora_alias
by default, the configuration file does not use any of them, so it gets the machine's hostname as alias and a large random hexadecimal number as identifier or name. The agent's name is not visible (except for the agent's detailed view) and CANNOT be changed. The agent's alias cab ver changed at any time, without worrying about software agent configuration, since the one used for clearly identifying the agent is the agent's “name”.