Pandora: Documentation en: Operations

From Pandora FMS Wiki
Jump to: navigation, search

Go back Pandora FMS 3.0 documentation index

Contents

1 Monitoring with Software Agents

1.1 Agent names

From Pandora FMS version 7 onwards, the agents have an alias and a name (or unique identifier). An agent configured by default will generate a name (or identifier) based on a pseudo-random hexadecimal string, and an alias (or visible name) based on the hostname of the machine.

In previous versions, there was only the "name" of the machine, and the previous system is totally compatible with later Pandora FMS versions, the only issue is that if there are two agents with the same identifier (or names) in the same Pandora FMS installation, the data of both agents will be mixed and/or will be overwritten. That is why we introduced from version 7 onwards the possibility that agents with different names could have the same alias.

The following configuration tokens are used to change this behavior:

pandora_agent
pandora_alias

The configuration file does not use either one of them by default, so it gets the hostname of the machine as an alias and a very large random hexadecimal number as a name or identifier. The agent name is no longer visible (except in the agent detail view) and CANNOT be changed. The Agent Alias can be changed at any time, without having to worry about the configuration of the software agent, since the agent's unique identifier is the agent's "name".

1.2 Agent Configuration

All the configuration and monitoring parameters of the software agents can be found in their configuration file pandora_agent.conf. This is stored locally in the machine where the software agent is installed, so any modification to be made in the agent must be reflected in this file. You have a detailed description of all agent configuration tokens in the chapter "PandoraFMS Agent Configuration" [1] while here we will only focus on the advanced uses of some of them.

1.2.1 Remote Configuration

On the Enterprise version, there is a remote Agent Configuration feature which allows centralized configuration and file management from the server console. This allows centralized management of all our software agents without the need to physically access the systems where they are installed.

The configuration consists of two files. Their file names are <md5>.conf and <md5>.md5, where <md5> is the agent's name hash code. Those files are stored in '/var/spool/pandora/data_in/conf' and '/var/spool/pandora/data_in/md5' folders respectively.

These files stored in the server are modified by Pandora FMS console when the agent configuration is edited remotely.


Sw-agent.png


To enable remote configuration, enable the corresponding parameter in the agent's local configuration file first. From this moment on, all the changes must be made from Pandora FMS console:

remote_config 1

Info.png

Once the agent's remote configuration is enabled, any changes made locally in the configuration file will be overwritten by the configuration stored in the console. If you want to prevent this from happening, stop the agent, modify the configuration file, disable the remote configuration and launch the agent again.

 


1.3 Common Configuration Parameters

In the Pandora FMS Software Agents section you can find a complete explanation on Agent Configuration. In this section, the common parameters used to configure the Software Agents will be explained.

The most common parameters are:

  • server_ip: IP address of Pandora FMS Server.
  • server_path: Path of the 'incoming' folder for the Pandora FMS server (it's '/var/spool/pandora/data_in' by default).
  • temporal: Software agent's temporal folder (it is '/var/spool/pandora/data_out' by default).
  • logfile: Software agent's log file (it is '/var/log/pandora/pandora_agent.log' by default).
  • interval: Agent's execution interval (it is '300' by default).
  • intensive_interval: Intensive module execution interval (it is '300' by default).
  • debug: Enables (1) the debug mode, to save XML data files and analyze them. When it is activated, it does not send XML to the server and for execution, to be able to analyze the XML generated.
  • agent_name: Agent name (hostname is taken by default).
  • remote_config: Activation of remote configuration. It is disabled (0) by default. Only for Enterprise version.

An example of the general parameters for UNIX configuration would be:

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        /var/spool/pandora/data_out 
logfile         /var/log/pandora/pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config   1 

An example of the general parameters for Windows configuration would be:

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        c:\program files\pandora_agent\temp
logfile         c:\program files\pandora_agent\pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config   1

1.4 Custom Fields

Custom fields are an easy way to customize agent information. Create custom fields by clicking on 'Resources' -> 'Custom fields'.


Customfields1.JPG


Custom fields2.png


Info.png

You can include links in custom fields using the [url]link[/url] or [url=link]webname[/url] tags.

 


When creating your own custom fields, you will be able to specify the following parameters:

  • Name: Name of the custom field.
  • Password type: The fields with this parameter activated will be shown as asterisks.
  • Display on front: With this parameter activated, the information of the customizable field will be shown in the general view of the agent, as shown below:


Customfields3.JPG


  • Enabled combo: This parameter allows you to activate the configuration of selectable parameters from a drop-down list. Once activated, a new field will appear in the configuration window of the corresponding custom field to enter the combo values separated by commas.

Template warning.png

If the "Enabled combo" parameter is enabled, "Password type" will be disabled.

 



Customfields2.JPG


These custom fields can also be retrieved from the agent configuration file, using the following configuration token:

custom_field1_name Model
custom_field1_value i386

This example allows you to use a custom agent field defined in the agent's. conf.

1.5 Monitoring with the Software Agent

Software agents are running on the systems from which they collect information. Each of the checks they make on the system, such as CPU usage, free memory or disk space, corresponds to a module. Therefore, each of these modules collects a single data in each execution.

There are also some agent-specific directives that help to collect certain data directly from the operating system (e.g. CPU usage, memory, events, etc.). With these agent-specific directives, there is no need to execute commands. Check the Software Agents Installation Section to obtain more information about them.


Agent-monitoring.png


Pandora FMS software agents use the commands of the operating system in which they are installed to obtain the information for each one of their modules. The Pandora FMS data server processes and stores in the database all the information generated by the software agents, which send you their data in an XML file.


Esquema-3.png
Logical schema of an agent/physical agent


In the agent configuration file, modules are defined by the following structure:

module_begin 
module_name cpu_user
module_type generic_data
module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'
module_description User CPU Usage (%)
module_end

A real case can be:

module_begin 
module_name Files data_in
module_type generic_data
module_exec ls /var/spool/pandora/data_in | wc -l
module_description Number of files incoming dir
module_end

The module_exec line contains the command that will be executed to collect the information. The value returned by that execution will be the data obtained by the module and will be shown in the monitoring. Another command to get information using module_exec would be:

module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'

To collect the information, the agent will execute the command in the shell as if it was done by an operator, so it is always advisable to manually launch the command and analyze the output:

$> vmstat 1 2 | tail -1 | awk '{ print $13 }'

The value returned by the execution will be stored in XML as module data. You can indicate anything in the module_exec line as long as the output is compatible with the values accepted by Pandora FMS (numerical, alphanumeric or boolean), so it is possible to indicate custom scripts:

module_exec myScript.pl --h 127.0.0.1 -v cpu

Again, the agent will execute the command and collect the returned value in the same way as an operator would type in the shell:

$> myScript.pl --h 127.0.0.1 -v cpu

When the software agent is executed for the first time, it sends an XML to the Pandora FMS which creates the agent automatically with all its modules.

1.5.1 Types of Modules

The following are the possible types of modules in software agents depending on the type of data returned:

  • generic_data: Numerical and floating point data.
  • generic_data_inc: A kind of increasing numerical data. Stores the difference between the previous and current data divided by the elapsed time in seconds, showing the rate per second. This type of data is used to count "number of times per second" of something, such as log entries/sec, receivedbytes/sec , incoming connections/sec , etc.
  • generic_data_inc_abs: Type of absolute increasing numerical data. It stores the difference between the previous and current data, without dividing it between the elapsed seconds, so the value will correspond to the total increase between the two executions, and not to the increase per second. This type of data is used to count the number of times something happens, such as log entries, total received bytes, number of incoming connections, and so on.
  • generic_proc: Boolean type of data, where a value of 0 means False or incorrect, and values above zero mean True or correct. The generic_proc types have the critical (0) and correct (1 or higher) states preconfigured.
  • generic_data_string: Kinds of alphanumeric data (text).
  • async_data: It is a kind of asynchronous numeric data. It is the same as 'generic_data' but for asynchronous data which is only updated if there is a change. The asynchronous kind of data do not have a defined periodicity when data can be obtained.
  • async_string: This is a kind of asynchronous alphanumeric data. It is the same as 'generic_string' but for asynchronous data which are only updated if there is a change. It is the kind of data that you are recommended to use if you want to monitor searches in logs or event viewers. New data can be obtained at any moment or not for several days.
  • async_proc: It is a kind of asynchronous boolean data. It is the same as 'generic _proc' but for asynchronous data which are only updated if there is a change.


  • Image module: They are based on a text string type module (generic_data_string or async_string). If the data in the module is a base64 image, in other words, part of the string contains "data:image", it will be identified as an image and, on the views that it appears, it will enable a link to open a window to display the image. Also on its historical data the strings that build/generate the images will be saved and displayed.

The software agent already comes preconfigured to send certain data from the system on which it is installed. These usually are (depending on the version):

  • System CPU
  • Available space on the hard-drive
  • Free memory
  • Monitor of the programs and services states

1.5.2 Intervals in local modules

The local modules (or software agent modules) all have the interval of their agent as a "base". In other words, if an agent has an interval of 5 minutes (300 seconds), all modules will have an interval of 5 minutes by default. You can make a module have a HIGHER interval, but you can never make it lower than the agent's base interval, since the interval of a module is defined as a multiplier of the agent's interval.

If an agent has an interval of 300 and a module of that agent has the following configuration:

module_interval 2

The interval module will be 300x2. The module_interval parameter only supports whole numbers higher than zero.

1.5.3 Module Creation Interface

(Enterprise only)

In the Enterprise version, it is possible to create and manage local modules of the software agents if they have the remote_config 1 parameter. If you do not have the Enterprise version, all these operations must be done directly on the configuration file of the software agent, locally in the system where the agent is installed.

Creating local modules from the console is done using a text box form. This is the location to specify the configuration data, which will be placed into the configuration file of the software agent (besides the common configuration with the remote modules like thresholds, type and group).

Creating a module with the remote config enabled on the agent:

Local module editor.png

In this text box there are two buttons, one to create a basic configuration structure and the other to check that the data is correct. This check is intended for basic parameters, e.g. checking if it begins with 'module_begin', ends with 'module_end' and whether it has a valid type and name. Other parameters may be wrong, but that will not be detected here.

The field name and the type combo are linked to the 'module_name' and 'module_type' parameters of the data configuration. If the module name on the name field is changed, the configuration data name will be changed automatically and vice versa. If a type in the combo is selected, the data configuration type will be changed and if a correct type was written in the configuration data, this type will be selected automatically in the combo.

When a module from a local component is changed, it might have macros. If that is the case, the configuration data box will be hidden and a field for each macro will appear instead. You can find a detailed explanation of this feature in the following section:

Templates and components

1.5.4 Conditional Monitoring

1.5.4.1 Post-Conditions

Pandora FMS software agent supports the execution of commands and scripts as post-conditions. This means that actions could be performed depending on the value obtained in the execution of the module.

With the module_condition parameter, a value or range of values and the execution to be carried out must be indicated in case the module is between these values:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

You can define multiple postconditions for the same module, e.g.:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition (90, 100) remove_processes.sh
module_condition < 20 add_processes.sh
module_end

Some examples:

Execution if the module data is less than '20' :

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

If the script named 'get_cpu_usage.pl' returns '18', the software agent will execute the script 'add_processes.sh', otherwise it will not.

Execution with two postconditions:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 10 start_new_server.sh
module_condition < 20 add_processes.sh
module_end

If the module returns '15', the software agent will only execute the script named 'add_processes.sh' - but if the value is '6', the script will execute both scripts named 'start_new_server.sh' and 'add_processes.sh'.

1.5.4.2 Pre-Conditions

The module_precondition parameter defines a precondition to evaluate before a module execution. Depending on the result of this precondition, the software agent will execute the module or not. The structure of the configuration file is:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

You can define multiple preconditions for the same module:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_precondition = 1 important_service_enabled.sh
module_end

In the example above, there are two preconditions. For the module to be executed all preconditions must be met, in this case two, so the module will only be executed when the number_active_processes.sh script returns a number higher than 10 and the important_service_enabled.sh script returns 1.

Module execution if the precondition is above '10' only:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

In the example above, the software agent executes the 'number_active_processes.sh' script, if the returned value is higher than '10'. If the returned value is lower than '10', the module will not be executed.

1.5.5 Intensive Monitoring

There are certain specially important modules, such as critical running processes or services. Intensive monitoring enables more controlled monitoring of these particular cases.

It consists of warning in a shorter interval that a problem has arosen without reducing the agent's general interval.

The software agent presents these two configuration parameters:

  • Interval: agent sampling time in seconds. This is the general range for all local modules. Required parameter.
  • Intensive_interval: time in which you will be notified of a problem on the especially critical modules. Optional parameter.

At module level, the module_intensive_condition parameter will be used to determine under which condition the status of the module will be notified in the time defined by the intensive_interval.

  • module_intensive_condition = 0: if the module obtains as a result the value indicated in this parameter (in this case 0), it will be notified in the intensive interval defined in the agent.

The following example shows the configuration of a agent about which it is desired to be notified in 10 seconds if the sshd process has stopped working:

intensive_interval 10
interval 300
module_begin
module_name SSH Daemon
module_type generic_data
module exec ps aux | grep sshd | grep -v grep | wc -l
module_intensive_condition = 0
module_end

If the service fails, you will be notified in the next 10 seconds. If the service is up, you will be notified in the next 5 minutes, like normally.

1.5.6 Programmed Monitoring

The software agent supports the definition of programmed modules which are executed in the defined instances. The syntax is the same as crontab. The module structure is the following:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab * 12-15 * * 1
module_end

In this example, the module is executed every Monday from 12 to 15.

If you are required to execute the module every hour and ten minutes, you can use this module definition:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab 10 * * * *
module_end

Info.png

Note that if you use an interval that causes the module not to report data, this module will go into "unknown" status. Use asynchronous modules for these cases.

 


1.6 Specific Monitoring for Windows

The software agent for Windows has specific features to make monitoring a lot easier. These features are explained with some examples:

1.6.1 Monitoring Processes and Process Watchdog

1.6.1.1 Process Monitoring

The parameter module_proc verifies whether a process with a preset name is running on this machine. The module definition is:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_description Process Command line
module_end

If the process name has blanks, do not use «" "». The process name is the same as the one shown in Windows Task Manager (taskmngr) including the .exe extension. It is very important to use the same upper and lower-case letters.

If you want the software agent to immediately notify you if a process is not working, add the parameter module_async yes. In this case, the module definition would be:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_async yes
module_description Process Command line
module_end

1.6.1.2 Watchdog Process

The watchdog feature on the Pandora FMS Agent for Windows allows immediate response to the failure of a process and restarts it. It is important to keep in mind that the watchdog only works if the module is of the asynchronous type.

The definition of a module with watchdog enabled would be as follows:

module_begin
module_name Notepad
module_type generic_data
module_proc notepad.exe
module_description Notepad
module_async yes
module_watchdog yes
module_user_session yes
module_start_command c:\windows\notepad.exe
module_startdelay 3000
module_retrydelay 2000
module_retries 5
module_end

In the previous example, the watchdog will be activated each time the notepad.exe process is deactivated and the command c: \windows\notepad. exe will be executed. In addition, if we look at the other watchdog configuration parameters, we will see that the process reactivation will be attempted 5 times with an initial wait time of 3 seconds and a waiting time between retries of 2 seconds. In this example, the notepad.exe process will be launched in the user's session.

1.6.2 Service Monitoring and Service Watchdog

1.6.2.1 Service Monitoring

The module_service parameter verifies whether a specified service is running on the machine. The definition of this module is as follows:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_end

If the service name has blanks, do not use «" "». To find the service name, look for the Service Name field under the Windows Service Manager. It is very important to use the same upper and lower-case letters.

If you want the software agent to warn you immediately when a service is down, add the parameter module_async yes. The module definition should be as follows:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_async yes
module_end

1.6.2.2 Service Watchdog

There is a watchdog mode for services which allows you to detect and restart a downed service almost in real time. A module definition example using watchdog would be the following:

module_begin
module_name ServiceSched
module_type generic_proc
module_service Schedule
module_description Service Task scheduler
module_async yes
module_watchdog yes
module_end

The watchdog definition for services has no need for any extra parameters because they are incorporated in the service definition.

1.6.3 Basic Resource Monitoring

This section describes how to monitor the basic variables of a Windows-based machine.

1.6.3.1 CPU Monitoring

The parameter module_cpuusage returns the CPU usage percentage.

It is possible to monitor the CPU based on its ID with a module definition like the following:

module_begin
module_name CPU_1
module_type generic_data
module_cpuusage 1
module_description CPU usage for CPU 1
module_end

It is also possible to monitor the average CPU usage from all systems with the following module:

module_begin
module_name CPU Usage
module_type generic_data
module_cpuusage all
module_description CPU Usage for all system
module_end

1.6.3.2 Memory Monitoring

To monitor the memory, you can use two parameters: module_freememory which returns the amount of free memory in the system and module_freepercentmemory which returns the percentage of free memory.

An example for a module using the module_freememory parameter would be:

module_begin
module_name FreeMemory
module_type generic_data
module_freememory
module_description Non-used memory on system
module_end

An example for a module using the module_freepercentmemory parameter would be:

module_begin
module_name FreePercentMemory
module_type generic_data
module_freepercentmemory
module_end

1.6.3.3 Hard Drive Monitoring

To monitor hard drive space, you may use two parameters: module_freedisk which returns the amount of available space and module_freepercentdisk which returns the percentage of available space. Both parameters require the monitored unit as an input. Do not forget the character « : ».

A module that uses the module_freedisk parameter is defined in this way:

module_begin
module_name FreeDisk
module_type generic_data
module_freedisk C:
module_end

A module example that uses the module_freepercentdisk parameter is defined in this way:

module_begin
module_name FreePercentDisk
module_type generic_data
module_freepercentdisk C:
module_end

1.6.3.4 WMI Queries

Pandora FMS Software Agent allows you to retrieve information by using WMI queries, which is a source of data widely used to obtain external or system-related information.

The software agent allows you to execute any local WMI query you want using the module_wmiquery parameter. To perform the query, WMI query is defined in the module_wmiquery parameter and the column that contains the information to be monitores with the module_wmicolumn parameter.

You will get a list with the installed services:

module_begin
module_name Services
module_type generic_data_string
module_wmiquery Select Name from Win32_Service
module_wmicolumn Name
module_end

You will also be able to get the current CPU load using WMI:

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

1.7 Remote Checks with Software Agents

A remote check performed by the agent makes it easy to monitor complex networks that have special, security-related requirements.

This way of working is usually used when remote checks have to be launched on systems that the main Pandora FMS server does not have access to, for which it is possible to install a software agent, run remote checks from that point and distribute them in broker agents.

This section explains how to use this feature of the software agents.

1.7.1 ICMP Checks

ICMP or ping checks are very useful to know whether a machine is connected to a network or not. In this way, a single software agent could easily monitor the status of all machines.

UNIX

By using the UNIX software agent, you are able to use the system commands to create a module which performs the ping check. An example module definition would be:

module_begin
module_name Ping
module_type generic_proc
module_exec ping -c 1 192.168.100.54 >/dev/null 2>&1; if [ $? -eq 0 ]; then echo 1; else echo 0; fi
module_end

In this example module, a ping check is performed on the '192.168.100.54' host. To check a different host, change the IP.

Windows

The software agent for Windows platforms supports specific configuration parameters to configure ping checks:

  • module_ping_count x: Number of ECHO_REQUEST packages to be sent (Default value is '1').
  • module_ping_timeout x: Timeout in seconds (Default value is '1').
  • module_advanced_options: Advanced options for 'ping.exe'.

A module configuration example could be:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 5
module_end

In this example, the same check is going to be performed as the previous one, but now the Software Agent for Windows platforms is going to be used.

1.7.2 TCP Checks

TCP checks are useful to verify whether a port of a host happens to be open or not. That may be interesting in case you want to know whether an application is connected to the network or not.

UNIX

You will able to perform the TCP checks through the following module by using the software agent for UNIX platforms:

module_begin
module_name PortOpen
module_type generic_proc
module_exec nmap 192.168.100.54 -p 80 | grep open > /dev/null 2>&1; echo $?; if [ $? == 0 ]; then echo 1; else echo 0; fi
module_timeout 5
module_end

This module will help you to check whether port 80 of the '192.168.100.54' host is open or not.


Windows

If you want to use the software agent for Windows, here you have some parameters to configure the module. They are:

  • module_tcpcheck: Host to be checked
  • module_port: Port to be checked
  • module_timeout: Timeout for the check

A module definition example is this:

module_begin
module_name TcpCheck
module_type generic_proc
module_tcpcheck 192.168.100.54
module_port 80
module_timeout 5
module_end

This module is the equivalent for the Windows software agent to perform the check on port 80 of the '192.168.100.54' host.

1.7.3 SNMP Checks

SNMP checks are commonly used to monitor network devices to check the interface status, inbound/outbound bytes, etc.

UNIX

If you are using the software agent for UNIX platforms, you may create the module using the 'snmpget' command like this:

module_begin
module_name SNMP get
module_type generic_data
module_exec snmpget 192.168.100.54 -v 1 -c public .1.3.6.1.2.1.2.2.1.1.148 | awk '{print $4}'
module_end

This module returns the value for OID .1.3.6.1.2.1.2.2.1.1.148 on the '192.168.100.54' host.

Windows

For Windows software agent, these are the parameters:

  • module_snmpversion [1,2c,3]: SNMP version (Default value is '1').
  • module_snmp_community <community>: SNMP community (Default value is 'public').
  • module_snmp_agent <host>: The host to monitor.
  • module_snmp_oid <oid>: OID.
  • module_advanced_options: Advanced options for 'snmpget.exe'.

A module example could be:

module_begin
module_name SNMP get
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.148
module_end

This module would be the Windows platform equivalent for the previous check performed through the software agent for Unix.

1.8 Proxy Mode

Template warning.png

To use Pandora FMS agent's proxy mode on Linux or UNIX systems, the agent must -not- be executed by a root user ! You are required to perform a custom installation of the Pandora FMS agent to do so. You may look up all the details about custom installations in the section Custom Agent Installation.

 


Pandora FMS Software Agents have a Proxy Mode which allows them to act other software agent proxies, redirecting the communication of several agents to the Pandora FMS Server. The software agent with an enabled proxy mode is able to perform monitoring tasks too.


Proxy-mode.png


The Proxy Mode is very useful if you are dealing with a network in which only one machine can communicate with Pandora FMS Server and you need to monitor by means of software agents the rest of network computers. In this situation, the other computers will communicate with the proxy instead of the server.

In addition data forwarding through XML, the proxy mode supports Remote Configuration and File Collection features.

With all these features, the Proxy Mode offers a transparent operation of software agents in networks with limited connectivity.

To enable the Proxy Mode in a software agent, configure the following parameters:

  • server_ip: IP of the Pandora FMS Server.
  • proxy_mode: Enabled (1) or diabled (0).
  • proxy_max_connection: Maximum number of simultaneous connections for the proxy. The default value is '10'.
  • proxy_timeout: Proxy timemout. The default value is '1' (in seconds).

A configuration example:

server_ip 192.168.100.230
proxy_mode 1
proxy_max_connection 20
proxy_timeout 3

To redirect the connection of a software agent, enter as Pandora FMS server address that of the agent with the Proxy Mode activated. For example:

This proxy mode enabled agent has the IP 192.168.100.24

In the software agent which cannot directly connect to the Pandora FMS Server, configure the server_ip parameter in the following way:

server_ip 192.168.100.24

With this configuration, the software agent with limited communication will use the software agent in Proxy Mode to communicate with Pandora FMS server, keeping all its features such as remote configuration, policies or file collections.

1.9 Broker Mode

The software agent has a Broker Mode which allows one agent to monitor and manage the configuration as if there were several software agents installed:


Modo-broker.png


When the broker mode is activated in a software agent, a new configuration file is created. From that moment on, the original software agent and the new broker will be managed separately with their independent configuration files, as if they were two completely separate software agents on the same machine.

The main features of the Broker Mode are:

  • Sending local data as another agent. Very useful to monitor different software instances as different agents.
  • Sending the collected data from the remote checks to other machines as if a software agent had been installed on them.

To create a broker, add a line with the broker_agent <broker_name> parameter. It is possible to create as many broker agents as you wish, just by adding the corresponding broker_agent lines, as follows:

broker_agent dev_1
broker_agent dev_2

Once the brokers are created, the 'dev_1.conf' and 'dev_2.conf' configuration files will be created with the same content as in the original software agent, but with their corresponding name. By adding or deleting modules from 'dev_1.conf' and 'dev_2.conf' configuration files, you can customize the checks performed by the brokers.

On the Pandora FMS web console the brokers appear and will be managed independent agents, which means that if you have a software agent installed with two brokers, you will see three different agents with their modules, configurations, etc. on the web console.

NOTE: Broker agent instances cannot use file collections. If you want to use collections, distribute them and/or use them in the "real" agent that is used as a basis for the broker agent, not in one of its instances.

NOTE: Modules that save data within the memory between executions (module_logevent and module_regexp on Windows) will not work when broker agents are enabled.

1.9.1 Broker mode use Examples

1.9.1.1 Monitoring a local Database as a different Agent

The objective here is monitoring basic parameters (CPU, memory and hard drive) and monitoring an installed database separately.

To perform this monitoring, the following structure will be used:

  • Installed Software Agent: monitoring CPU, memory and disk.
  • Broker for the Database: monitoring internal status of the database.

The first step is installing the software agent on the machine to monitor the CPU, memory and hard drive parameters. Then the following line is added in the software agent configuration:

broker_agent DBApp

Secondly, a broker agent called 'DBApp' is created by adding this line, so a configuration file named 'dbapp.conf' will appear. Finally, the modules to perform the checks for the database are added in this configuration file:

module_begin
module_name Num Users
module_type generic_data
module_exec get_db_users.pl
module_end

module_begin
module_name Num slows queries
module_type generic_data
module_exec get_db_slows_queries.pl
module_end

By doing this, two agents will appear in the Pandora FMS web console: One bearing the name of the machine with the 'CPU', 'Memory' and hard drive modules and another one called 'DBApp' with the 'Num Users' and 'Num slows queries' modules.

1.9.1.2 Monitoring Devices Remotely Using Brokers

For this example, a software agent has been installed on a Windows machine, monitoring CPU, memory and hard drive. It is also required to monitor a router with the IP '192.168.100.54' without installing an agent on it. To solve the problem, brokers are used.

A broker will be created using the following parameter:

broker_agent routerFloor5

By adding this line, a broker agent will be created with the name 'routerFloor5'. The software agent was installed on a Windows machine, so the router can be monitored by using the ping and SNMP modules available for Windows software agents. To do that, the 'routerFloor5.conf' file must be modified by adding the following lines:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

In this example, the web console of Pandora FMS shows two agents: One is the Windows machine with the 'CPU', 'Memory' and 'hard drive' modules and the other one is 'routerFloor5' bearing the modules named 'Ping', 'Eth 1 up' and 'Eth 2 up'.

1.9.1.3 Monitoring inaccessible networks remotely

In some cases, you need to monitor devices remotely where the Pandora FMS Remote Server cannot access them directly.


Broker example no access.png


In this example, some devices from one of the company facilities from the headquarters must be remotely monitored. The Pandora FMS Server is connected to the other company facilities in the headquarters using a VPN. Due to some restrictions, the Pandora FMS Remote Server cannot access the machines directly. To monitor the company's branches, the Broker Mode, which allows a software agent to send XML files to the Pandora FMS Server as if there were several different devices, is used.

One can add as many brokers as devices to be monitored to the configuration file of the software agent. A configuration example could be:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

Once the brokers are created, the monitoring for each device can be customized by modifying the configuration file of each broker. For example, the configuration for the Windows machine called 'device_1' is:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

With this configuration, it is possible to configure remotely and send the files to Pandora FMS server in spite of the communication restrictions between the company's headquarters.

1.9.1.4 Shared Monitoring Load through Brokers

Broker mode is very useful for sharing and distributing the monitoring load within several network points.


Broker scalation example.png


In this example, the architecture has several networks named from A to Z with 1000 devices each. The capacity of the Pandora FMS Remote Server is about 2000 agents, so it is decided to use broker mode enabled software agents to share and distribute the monitoring load. These broker mode enabled software agents will monitor remotely all network devices and send the data in XML format to the Pandora FMS Central Server.

For each network, there is a broker mode enabled agent. On it, brokers will be created until the number of devices to be monitored is reached. An example configuration for the 'Broker_Agent_Net_A' software agent could be the following:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

In addition, for each broker, the modules to monitor the devices are added. Example: The broker 'device_1' (which is a router) could have this configuration:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

This is another example configuration for the device_2 broker, which monitors a Windows machine with the following modules:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

Using broker mode enabled software agents, makes sharing the load to collect data from thousands of devices easier.

1.10 Inventory using Software Agents

Pandora FMS Software Agents support inventory features for both hardware and software. The inventory system allows to keep a history of CPU, cards, RAM memory, patches, software, etc, used in the company servers. Furthermore, it is possible to generate alerts if there is a change in the inventory, e.g. if a disk was replaced or an application was uninstalled.

For further information on the subject, please have a look at the section Local Inventory through Software Agents.

1.11 UDP remote commands

1.11.1 How to ask an on-demand agent for information

Pandora FMS software agent includes the UDP Server feature, which allows to remotely indicate actions to an agent, such as restarting or executing a command.

The basic configuration parameters of the UDP Server in the software agents are:

  • udp_server: it enables (1) or disables (0) this feature.
  • udp_server_port: listening port of the UDP server in the software agent.
  • udp_server_auth_address: IP address where the UDP server accepts requests. You can set it to 0.0.0.0.0 to accept from all sources.

Configuration example :

udp_server 1
udp_server_port 41122
udp_server_auth_address 0.0.0.0

Now, to force the restart of the agent the udp_client. pl script must be used, present in the Pandora FMS server, and normally located in /usr/share/pandora_server/util. It can be run from the command line or used in an alert, making use of the command that is pre-configured in the "Remote agent control" console.

There is also a default alert action called Restart agent, which uses this script. It uses the action REFRESH AGENT on the udp_client. pl script to restart the agent if it has the UDP server listening.


Agent restart action.png


Follow these steps to enable the Software Agent's remote refresh option:

1. In the configuration file, set up the options for the software agent (UNIX or Windows). Be mindful on the authorized IP address (is the Pandora FMS server behind a NAT?), or just type in '0.0.0.0' on that field to allow any IP address to force a refresh of the agent.

2. Restart the software agent for the changes to be applied.

3. Associate the Restart agent alert to the module of some agent (it is necessary that this agent has the IP address correctly configured).

4. Force the execution of the alert or force an incorrect state of the module to trigger the alert.

Now, thanks to this action, it is possible to manually force the alert at any time to refresh the agent software remotely and get the information quickly.

1.11.2 Custom remote commands

Apart from the Refresh agent command, you can specify new and custom actions for the agents to perform under the Pandora FMS server's UDP orders.

If you are interested, make a slight modification in pandora_server.conf, apart from enabling the udp service as well as configure it so that it receives the server's orders:

udp_server 1
udp_server_port 41122
udp_server_auth_address <server IP>

Then add a line for each custom command you want to perform, following this syntax:

process_nameofthecommand_start <command>

For example, if you want to create a custom command to start sshd service, you should add a line like this one:

process_sshdproc_start /etc/init.d/sshd start

Then create a new alert action at Pandora FMS Console for each remote command you made. You can copy the "Remote agent control" action, which is already prepared to send UDP commands. Set "START PROCESS sshdproc" on Field 1, as seen on the screenchot.


Udp process.JPG


Now, you only need to set a new manual alert with the new alert action on the agent whose sshd service you wish to start. When the alert is forced, the order will be launched and the agent will start the service.

Info.png

Custom orders can also be created to execute scripts. This allows a huge variety of remote actions to be performed on a remote agent just by clicking a button.

 


1.12 Plugins in software agents

They are characterized by performing complex advanced checks from the software agents, being able to return several modules as a result instead of a single value. Unlike the server plugins, which are executed by Pandora FMS server, agent plugins return their data in an XML, reporting one or several modules at the same time.

1.12.1 Execution on Windows systems

In Windows, all the default plugins are programmed in VBScript. To run them, it is vital to use the appropriate interpreter indicating the full path.

Here are some examples of how to use the default plugins included in the Windows agent:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\logevent_log4x.vbs" Application System 300
module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"
module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" iexplore.exe myapp.exe

The Windows agent includes several ready-to-use plugins.

1.12.2 Execution on Unix systems

Unix plugins are by default in the directory "/plugin" of the agent directory, in /etc/pandora/plugins, so it would not be necessary to use the full path in its execution if they are in this directory.

Here are some examples of using plugins:

 module_plugin grep_log /var/log/syslog Syslog .
 module_plugin pandora_df tmpfs /dev/sda1

The Unix software agent comes with several plugins by default ready to work.

1.12.3 Plugin use Examples

Plugins for the software agent can return a piece of data or a group of data. An example of a plugin that returns a piece of data can be the Windows ps. vbs plugin, which simply checks whether a process is running. With the next line the plugin is run:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" IEXPLORE.EXE

The result will be a module that returns 0 if the process is not active and 1 if it is active:

<module>
    <name><![CDATA[IEXPLORE.EXE]]></name>
    <description><![CDATA[Process IEXPLORE.EXE status]]></description>
    <data><![CDATA[1]]></data>
</module>

An example of a plugin that returns several data can be the Windows plugin df. vbs. The line to run this plugin would be:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"

The plugin returns one module for each found disk, the result would be:

<module>
    <name><![CDATA[C:]]></name>
    <description><![CDATA[Drive C: free space in MB]]></description>
    <data><![CDATA[8050]]></data>
</module>

<module>
    <name><![CDATA[D:]]></name>
    <description><![CDATA[Drive D: free space in MB]]></description>
    <data><![CDATA[900]]></data>
</module>

1.12.4 Agent Plugin Management from the Console

In the Enterprise version, it is possible to manage software agent plugins from the console without directly editing the configuration file.

If an agent has remote configuration activated, the plugin editor tab is available in its administration view.



Plugin editor tab.png

This section shows the list of active plugins in the agent, and allows you to delete, add or deactivate them. In the case of policy plugins, it may be useful to disable them because they will remain disabled when the policy is re-applied.



Plugin editor.png

The plugins managed in this editor can also be edited from the agent configuration file.



Plugin editor conf.png

1.12.5 How to create your own agent plugins

Plugins can be created in any programming language. They only have to obey to these rules:

  • Regardless of what you want to do, it must be automatic (without user interaction), and it must be done from the shell. You can use any type of scripting language or compiled language, but in that case you must also distribute, in addition to the executable, all libraries (or DLL) that are necessary for the plugin to run.
  • The plugin must return the information through the standard output (simply using echo, printf or the equivalent in the language that will be used for the plugin), and must use the XML format of Pandora FMS agents to return the information.

This is an example of a numerical module in XML:

<module>
<name><![CDATA[Sample_Module]]></name>
<type><![CDATA[generic_data]]></type>
<data><![CDATA[47]]></data>
<description><![CDATA[47]]></description>
</module>

The data contained in the <!CDATA[xxx]]> are used to protect XML from certain information that may contain "difficut" characters for XML, such as <, >, & or %.

Before trying to create a plugin, visit the Pandora FMS plugin library at https://library.pandorafms.com, and if you decide to create your own plugin, send it to the public library, so that others can use it.

Template warning.png

Make sure you finish the output of your plugin (if it is a script) with an error level 0, or the agent will think that the plugin has had an error and was not able to be run.

 


1.12.5.1 Shellscript (Linux/Unix) plugin example

#!/bin/bash
# Detect if local Mysql is without password
# First, do we have a running MySQL?
CHECK_MYSQL=`netstat -an | grep LISTEN | grep ":3306 "`
if [ ! -z "$CHECK_MYSQL" ]
then

        CHECK_MYSQL_ROOT=`echo "select 1234" | mysql -u root 2> /dev/null | grep 1234`
        if [ -z "$CHECK_MYSQL_ROOT" ]
        then
        echo "<module>"
        echo "<type>generic_proc</type>"
        echo "<name>mysql_without_pass</name>"
        echo "<data>1</data>"
        echo "<description>MySQL have a password</description>"
        echo "</module>"
        else
        echo "<module>"
        echo "<type>generic_proc</type>"
        echo "<name>mysql_without_pass</name>"
        echo "<data>0</data>"
        echo "<description>MySQL do not have a password</description>"
        echo "</module>"
        fi
fi

exit 0

1.12.5.2 VBScript (Windows) plugin example

' df.vbs
' Returns free space for available drives.
' --------------------------------------

Option Explicit
On Error Resume Next

' Variables
Dim objWMIService, objItem, colItems, argc, argv, i

' Parse command line parameters
argc = Wscript.Arguments.Count
Set argv = CreateObject("Scripting.Dictionary")
For i = 0 To argc - 1
    argv.Add Wscript.Arguments(i), i
Next

' Get drive information
Set objWMIService = GetObject ("winmgmts:\\.\root\cimv2")
Set colItems = objWMIService.ExecQuery ("Select * from Win32_LogicalDisk")

For Each objItem in colItems
	If argc = 0 Or argv.Exists(objItem.Name) Then
		If objItem.FreeSpace <> "" Then
			Wscript.StdOut.WriteLine "<module>"
			Wscript.StdOut.WriteLine "    <name><![CDATA[" & objItem.Name & "]]></name>"
			Wscript.StdOut.WriteLine "    <description><![CDATA[Drive " & objItem.Name & " free space in MB]]></description>"
			Wscript.StdOut.WriteLine "    <data><![CDATA[" & Int(objItem.FreeSpace /1048576) & "]]></data>"
			Wscript.StdOut.WriteLine "</module>"
            Wscript.StdOut.flush
		End If
	End If
Next

1.12.6 Using Nagios plugins from the agent

Nagios has a large number of plugins that can be used with Pandora FMS. One way to do this is using remote plugins with the Plugin Server, using Nagios compatibility. But in this way, you will only get the statuses, since it does not use the descriptive output that some plugins for Nagios have.

Using the wrapper to use Nagios plugins in the software agent will solve this problem. The wrapper comes by default with the Unix 3.2 agent. An equivalent plugin for Pandora FMS Windows agents can be downloaded from the Pandora FMS resource library, at [2]).

What does the plugin wrapper do for Nagios plugins?

It executes the Nagios plugin, using its original parameters and turning the output into useful data for Pandora FMS. It has two types of information:

  • Status information: NORMAL (1), CRITICAL (0), WARNING (2), UNKNOWN () and others (4). By default, they will use a proc module, so the NORMAL and CRITICAL values are working "by default". If you wish to have information about WARNING and other values, you must configure the module thresholds manually.
  • Descriptive information: generally string information. It will be placed in the module description field. Usually something like "OK: successfully logged in."

1.12.6.1 Example

If you have a pop3 plugin (in /tmp/check_pop3_login) with the run permissions, (which checks whether the pop3 account is working only by connecting to a remote host, sending a user and password and displaying everything is correct), then you can run it from the command line:

/tmp/check_pop3_login  mail.artica.es [email protected] mypass

I will return something like :

OK: successfully logged in.

And if it is not right, it will return something like this:

Critical: unable to log on

Using the wrapper is easy. Just put the wrapper and the name you want into the module before executing the call:

/etc/pandora/plugins/nagios_plugin_wrapper sancho_test /tmp/check_pop3_login  mail.artica.es [email protected] mypass

This will generate a complete XML for the agent plugin.

<module>
<name>sancho_test</name>
<type>generic_proc</type>
<data>0</data>
<description><![CDATA[Critical: unable to log on]]></description>
</module>

Or:

<module>
<name>sancho_test</name>
<type>generic_proc</type>
<data>1</data>
<description><![CDATA[OK: successfully logged in.]]></description>
</module>

The complete entry in pandora_agent. conf will be something like:

module_plugin nagios_plugin_wrapper POP3_artica.es /tmp/check_pop3_login mail.artica.es [email protected] mypass

This will look similar to this in the module (in the fail event):


Sample plugin wrapper.png

1.13 Monitoring with KeepAlive

There is a special module which has a unique type called "keep_alive" and it is used to give information in the absence of contact with the agent. It is useful to know when an agent has stopped sending information and receiving an alert of this fact.

When there is a module, remote or local, that obtains information from the agent, the date of last "contact" with the agent is updated, so that whenever there is data, even if it is only one module of the total, the agent will have updated its date of last contact, which is useful to know if the agent "does not respond". Specifically, an agent is considered "dead" when it has not updated its date in twice the time of its interval, that is, if it has an interval of 5 minutes, and it has been more than 10 minutes since it sent an update, the agent is considered dead.

This is the case when the keepalive module comes into play, triggering and marking the monitor into Critical status.

Configuring this type of modules is very easy, just create a new module type "KeepAlive".:



Keepalive.JPG


Once created, if the agent has data, within its interval, it will always be in "NORMAL" status (green):

Keepalive1.png


If the agent stops sending data (in this example, it had an interval of 1 minute), then it will automatically jump and be set to CRITICAL (red) status:

Keepalive2.png


It should be noted that if you have a remote module, for example, a Ping, in addition to the data reported by the agent, the keepalive module would never be triggered, since you are constantly updating the agent through Ping.

The keepalive module also behaves like any other module, it can be associated with an alert and can be used for any other element: reports, maps, etc.

Info.png

The keepalive module can be created only from the console (even if you do not have remote configuration enabled) and leaves no trace in the pandora_agent.con file.f

 


1.14 Command screenshot monitoring

Commands that have extensive outputs, such as top or netstat can be captured completely by a module and fully displayed. The module must be configured as a text type.

In the next screenshot, the result of one of these modules is shown, in this case the output of netstat -an:


Snapshot 1.png


Template warning.png

In order for it to work like this, it is necessary to configure properly both the Pandora FMS console (setup) and the agent that collects this information, making sure that it is untreated text.

 


To configure the console properly, make sure that in the main setup section you have the "Command line snapshot" property activated:

Command line snapshot setup.png

1.15 Image monitoring and visualization

This method allows you to define string type modules (generic_data_string or async_string) that contain images in text format with base64 encoding, being able to display that image instead of a specific result. This is stored as text information, and displayed in a different way, not as simple data, but by means of reconstructing an image.

This is how you can see the output of a text string with the content "data: image" (image in base64), captured by Pandora FMS, when clicking on the special icon for screenshot:


Snapshot text 1.png


To capture these images, just write a plugin that sends all the data, generating the necessary XML tags, and running the plugin as such, with the module_plugin directive. Let us see the following example plugin, which generates the output of an image, as you have just seen in the previous capture:

#!/bin/bash
echo "<module>"
echo "<name>Last football championship winner</name>"
echo "<type>async_string</type>"
echo "<data><![CDATA[data:image/jpeg;base64,/9j/4AAQSkZ....]]></data>"

The previous data would be generated by a device/application rendering images in base64.

echo "</module>"

Save that content in a file in the agent (or distribute it with file collections) and run it as follows:

module_plugin <complete path to the file>

1.16 Password protected groups

By default, when an agent sends data for the first time to the Pandora FMS server, it is automatically added to the group that has been defined in the agent configuration file. This means that, in practice, anyone can add an agent to a group if they know the group name. This could be a problem if several clients share their Pandora FMS instance or if you want to control what is in each group.

We can optionally configure a password for a group from the Pandora FMS Console. An agent will not be added to a group unless the correct password has been specified in the agent configuration file.

1.16.1 Example

To set a password for a group, navigate to the group editor and click on edit, enter the group password and save your changes:

Passgr1.JPG

To add a new agent to this group, edit your configuration file and add the following configuration option:

 group_password <password>

Do not forget to restart the agent to make the changes effective. The agent should be created correctly in the Pandora FMS console.


Go back to Pandora FMS documentation index