Pandora: Documentation en: Operations

From Pandora FMS Wiki
Revision as of 16:50, 13 December 2012 by Slerena (talk | contribs) (Monitoring with KeepAlive)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Go back Pandora FMS 3.0 documentation index

Contents

1 Monitoring with software agents

1.1 Agent Configuration

By default, Pandora FMS agents have a local configuration, so their maintenance will be managed from the monitored machine, editing their configuration file.

1.1.1 Remote Configuration

On the enterprise version, there is the remote agent configuration feature. That allows, in a centralized way from the server console, their configuration file management.

The configuration is composed by two files, their file names are <md5>.conf and <md5>.md5, where <md5> is the agent's name hash code. Those files are stored in "/var/spool/pandora/data_in/conf" & "/var/spool/pandora/data_in/md5" folders respectively.

Those files are composed by plain text, and could be edited from the console, which regenerate them when there is any change. Comunication is always from the agent to the server, thus that functionality is controlled by the agent.



Xml file transfer


In order to enable remote configure, you should edit the remote_config parameter, setting to 1, from the pandora_agent.conf file. Once done that change, agent's configuration file will be ignored, since detecting a change in the configuration, it will download the new version from from the server. A good way to force the server to get the agent's local configuration is to delete both configuration files from the server.

1.2 Common configuration parameters

At Pandora FMS software agents section you can find a complete explanation about agent configuration. In this section there are explained the common parameters used to configure the software agent.

The most used parameters are:

  • server_ip: IP direction of Pandora FMS Server.
  • server_path: Path of incoming folder for Pandora FMS server. By default /var/spool/pandora/data_in.
  • temporal: Software agent's temporal folder. By default /var/spool/pandora/data_out.
  • logfile: Software agent's log file. By default /var/log/pandora/pandora_agent.log
  • interval: Agent's execution interval. By default 300.
  • intensive_interval: Intensive module execution interval. By default 300.
  • debug: Debug mode enable. By deafult 0 (disabled).
  • agent_name: Agent name. By default hostname is taken.
  • server_port: Tentacle server port. By default 41121.
  • transfer_mode: File transference mode. By default tentacle.
  • remote_config: Remote configuration enable. By default 0 (disabled).

An example of the general parameters from a Unix configuration would be:

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        /var/spool/pandora/data_out 
logfile         /var/log/pandora/pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config    1 

An example of the general parameters from a Windows configuration would be :

server_ip       192.168.1.1 
server_path     /var/spool/pandora/data_in 
temporal        c:\program files\pandora_agent\temp
logfile         c:\program files\pandora_agent\pandora_agent.log 
interval        300 
debug           0 
agent_name      box01 
server_port     41121 
transfer_mode   tentacle 
remote_config   1

1.3 Custom fields

Custom fields are an easy way to personalized agent's information. You can create custom fields at Administration > Manage agents > Manage custom fields.

Custom fields1.png


If you want to create a custom field you have to click on "Create field" button and fill the fields described bellow:

Custom fields2.png


  • Name: Name of the custom field.
  • Display on front: With this field checked the custom field will be displayed in front of the agent like in the screenshoot bellow:


Custom fields3.png


1.4 Monitoring with the software agent

Data collected by the software agents is kept in small information pieces called «modules».Each module keps only a kind of data. Each module value is the value of a supervised variable. Once the agent starts sending information, data will start to consolidate at database and you will have access to them.

Check the Software Agents Installation Section to obtain more information about them.

The Pandora FMS software agents use the Operative system specific commands to obtain information. The Pandora FMS data server keeps and processes data generated by these commands that are send to the server in an XML file. The information that has been returned by these commands is contained in what we call «Modules».


PandoraAgent logical schema.png



In agent configuration file the modules are defined by the following structure:

module_begin 
module_name cpu_user
module_type generic_data
module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'
module_description User CPU Usage (%)
module_end

The parameter that indicate how to get the module information is module_exec. This parameter specifies the command that the agent should execute in order to get the information from the system. An example could be:

module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'

The agent will execute the command as an operator will do to collect the information:

$> vmstat 1 2 | tail -1 | awk '{ print $13 }'

Then the agent gets the value returned by the command and appends it to the XML as the module data. Of course the agent can execute any program or script that returns the monitoring value. For example the field module_exec could be:

module_exec myScript.pl --h 127.0.0.1 -v cpu

Again the agent will execute the command and collects the returned value, in the same way as an operator will type in the shell:

$> myScript.pl --h 127.0.0.1 -v cpu


When the software agent is executed for the first time, it sends an XML to the Pandora FMS data server, that recives it through Tentacle, SSH or FTP in the server entry directory. The data server checks this directory every X time, and when it finds a file, it processes it. At opening this data file, consisting on an XML, it identifies the agent by its name, in a unique way, this is, each agent needs to have a completely unique name, where the capital and small letters are distinguished by Pandora FMS.By default, the server creates automatically all agents from which it receives data and that are not logged in at the DDBB. In the same way, if the agent has been added in the «learning» mode, then the modules that have not been previously defined in the agent will be created automatically by the server.

Template warning.png

Configuration file encoding it's very important and must be the same that value set in encoding configuration parameter. If encoding is set properly we will avoid to receive data with wrong encoding characters

 


1.4.1 Kinds of modules

There are several kinds, mainly classified in two: data with origin at software agents and data with origin at network modules executed by a network server. Those identified as «generic» are modules with origin at software agents and those identified as «remote» are network modules.

generic_data

Kind of numerical data. It is useful to keep numerical data(whole numbers and in floating comma)obtained through one Pandora FMS agent module.

generic_data_inc

Kind of increasing numerical data.It keeps data that are the result of the difference between the last agent data and the current data.Pandora FMS server calculates and keeps the rate by second in an automatic way. All of the modules ended in «inc» are of incremental kind. These kind of data is used to count the "nº of times " of something, for example the entries in a log, bytes/sec, connexions/sec, etc.

generic_proc

Also generically called "monitors". They are a boolean kind of data. Where a value 0 means false or «Bad value», and values higher than 0 means right or « right value».The «Generic Proc» kinds are also called monitors, because they can show if something is right or not without needing to interpret it or stablish alerts on it. They are displayed in the agent view as small lights. Red if it is zero, green if it is higher than zero. All of the modules ended in «proc» are monitors.

generic_data_string

Kind of alphanumeric data (text).

async_data

Kind of asynchronous numeric data. Same as the generic_data but for asynchronous data, that are updated only when there is a change. The asyncrhonous data kinds have not a defined periodicity when we can obtain data.

async_string

Kind of asynchronous alphanumeric data. Same as the generic_string but for asynchronous data, that are only updated when there is a change. Is the kind of data that we should use to monitor searches in logs or event viewers, so we could have one data by second or not having one in many days.

async_proc

Kind of asynchronous boolean data. Same as generic _proc but for asynchronous data, that are only updated when there is a change.

The software agent comes already configured to send certain data from the system where it is installed. These usually are (depending on the version):

  • System CPU
  • Free space at disk
  • Free memory
  • Monitor of the program and/or services state

Depending on the software agent would be for an operative system or another, they use to have more modules or different checkings.

All these information is located in the file pandora_agent.conf. This file is in the directory/etc/pandora/ in GNU/Linux and in the predetermined Windows installation directory (C:/Archivos de Programa/pandora_agent/ o C:/Program Files/pandora_agent/, o similares).

Next we are going to explain the data for some of the modules:

CPU usage percentage at GNU/Linux

# CPU usage percentage (GNU/Linux)
module_begin
module_name cpu_user
module_type generic_data
module_interval 1
module_exec vmstat 1 2 | tail -1 | awk '{ print $13 }'
module_max 100
module_min 0
module_description User CPU Usage (%)
module_end

It could be seen that the kind of module is generic_data, and that it executes a GNU/Linux console command to obtain the result (module_exec). It is known that the maximum is 100 and the minimum 0. The interval (module_interval) represents the number or iterations between the execution of each module, if it is different from 1, the module will only start the execution of agent every these number of times. This is, if the agent execution time is 300 and the module interval is 3, then the module will be executed every 3* 300 = 900 seconds.

CPU usage percentage in Windows:

# CPU usage percentage (Windows)
module_begin
module_name CPUUse
module_type generic_data
module_cpuusage all
module_description CPU#0 usage
module_end

It is possible to check that the module is completely different in Windows and in GNU/Linux. In Windows it is an internal agent command, where module_cpuusage all represents the CPU usage in all CPU. Using module_cpuusage it will calculate the CPU usage only in the CPU #0. The rest of the fields are optionals.

To add one more module,please check the agent configuration and create a valid module block. Once done this, keep the agent configuration file and restart the agent, would be this the UNIX daemon or the Windows service.

1.4.2 Modules creation interface

The creation of local modules from console is done using a form where, besides the common configuration with the remote modules (thresholds, type, group...) there is a text box where specify the configuration data that will be placed into the configuration file of the software agent.



Local module editor.png



Near this text box there are two buttons, one to create a basic configuration structure and the other to check that the data is correct. This check is for the basic parameters, checking that begins with module_begin, ends with module_end, has a valid type and a name. Other parameters could be wrong and it won't be detected.



Local module editor basic.png





Local module editor check wrong.png





Local module editor check right.png



The field name and the type combo are linked the the parameters module_name and module_type of the data configuration. When change the module name on the name field, the configuration data name will be changed automatically and vice versa. In the same way, when select a type in the combo, the data configuration type will be changed and when a correct type were write in the configuration data this type will be selected in the combo automatically.


When a module is charged from a local component, it can has macros. If it has macros, the configuration data will be hidden and will appear a field for each macro. This feature is explained deeply at the section Templates and components

1.4.3 Conditional monitoring

Pandora FMS software agent supports the execution of script as postcondition depending on the module value. Also the sofware agent has a feature to evaluate a precondition before the module execution. In this section both features will be explained with examples.

1.4.3.1 Postconditions

The parameter module_condition defines a postcondition on module execution. It means that this parameters defines the command that will be executed depending on the value returned by the module. The structure in the configuration file is:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

You can define multiple postconditions for the same module. For example:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition (90, 100) remove_processes.sh
module_condition < 20 add_processes.sh
module_end

Some examples:

Execution when the module data is less than 20:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 20 add_processes.sh
module_end

If the script get_cpu_usage.pl returns 18, then the software agent will execute the script add_proceses.sh, otherwise not.

Execution with two preconditions:

module_begin
module_name CPU_Usage_Condition
module_type generic_data
module_exec get_cpu_usage.pl
module_condition < 10 start_new_server.sh
module_condition < 20 add_processes.sh
module_end

If the module returns 15, the software agent will only execute the script add_processes.sh. But if the value is 6 the script will execute both scripts start_new_server.sh y add_processes.sh

1.4.3.2 Preconditions

The parameter module_precontition defines a precondition to evaluate before a module execution, depending an the result of this precondition the software agent will execute the module or not. The structure in the configuration file is:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

You can define multiple preconditions for the same module. For example:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_precondition = 1 important_service_enabled.sh
module_end

Some examples:

Module execution only when the precondition is above 10:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_end

In this example first of all the software agent executes the script number_active_processes.sh, if the returned value is greater than 10 then the module is executed. If the returned value is lower than 10 the module will never be executed.

Module execution only if two preconditions are accomplished:

module_begin
module_name CPU_Usage
module_type generic_data
module_exec get_cpu_usage.pl
module_precondition > 10 number_active_processes.sh
module_precondition = 1 important_service_enabled.sh
module_end

In this example we have two preconditions. All preconditions must be accomplished to execute the module, in this case both. The module will only be executed when the script number_active_processes.sh returns a value greater than 10 and when the script important_service_enabled.sh returns 1.

1.4.4 Intensive monitoring

(Only available in 5.0 version or higher)

For some modules certain values are very important, while others are not. For example, when you are monitoring a system service, you want to be notified as soon as possible if the service goes down, but you don't need to be reminded the service is up every ten seconds.

This is what intensive modules are for. Intensive modules run with an interval of intensive_interval seconds while intensive conditions are met, the rest of the time they run with an interval of interval seconds like the rest of the modules.

For example, if you want to check whether the sshd system service is running every 10 seconds, but want to be notified every 10 minutes if the service is fine, you can configure the agent like this:

intensive_interval 10
interval 600
module_begin
module_name SSH Daemon
module_type generic_data
module exec ps aux | grep sshd | grep -v grep | wc -l
module_intensive_condition = 0
module_end

If the service goes down, you will be notified in the next 10 seconds, but if the service is up, you will be notified in the next 10 minutes.

This can save a lot of space in Pandora FMS's database.

1.4.5 Programmed monitoring

The software agent support the definition of programmed modules that are executed in the instants defined. The syntax is the same as crontab http://es.wikipedia.org/wiki/Cron_(Unix)#Sintaxis. The module structure is the following:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab * 12-15 * * 1
module_end

In this example the module is executed once, every Monday from 12 to 15.

If you want a module that is executed during all the interval we must set the option module_cron_interval to 0 in this way:

module_begin
module_name crontab
module_type generic_data
module_exec script.sh
module_crontab * 12-15 * * 1
module_cron_interval 0
module_end

If we need to execute the module every hour past ten minutes we can use this module definition:

module_begin module_name crontab module_type generic_data module_exec script.sh module_crontab 10 * * * * module_cron_interval 0 module_end

1.4.6 Windows platforms specific monitoring

The software agent for Windows platforms has specific features for this platform to make monitoring easier. Following these feature are explained with some examples.

1.4.6.1 Monitoring processes and process watchdog

1.4.6.1.1 Monitoring de processes

The parameter module_proc chekcs if a process with given name is running in this machine. The module definition is:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_description Process Command line
module_end

If the process name has blanks, don't use «" "». The process name is the same as showed in Windows Task Manager (taskmngr) including the .exe extension, its very important to use the same upper-case and lower-case.

If you want the software agent to notice you immediately when a process is not working you must add the parameter module_async yes, the module definition would be:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_async yes
module_description Process Command line
module_end
1.4.6.1.2 Process watchdog

The watchdog feature on Pandora Agent for Windows, allows to react immediately to the crash of a process, raising it again. It is important to remark that the watchdog only works if the module is asynchronous.

The definition of a module with watchdog would be like this:

module_begin
module_name Notepad
module_type generic_data
module_proc notepad.exe
module_description Notepad
module_async yes
module_watchdog yes
module_start_command c:\windows\notepad.exe
module_startdelay 3000
module_retrydelay 2000
module_retries 5
module_end

In the previous sample, the watchdog will only activate each time notepad.exe process is not running, and will execute the command c:\windows\notepad.exe each time. Other parameters configured for the watchdog execution are to try 5 times, with a warm-up time (delay before try) of 3 seconds, and a time between attempts of 2 seconds.

1.4.6.2 Service monitoring and service watchdog

1.4.6.2.1 Service monitoring

The parameter module_service checks if a specified service is running in the machine. The definition of this module is:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_end

Remember to use «" "» if the name of the service has blanks. To find the service name you can look at the Service Name field in Windows Service Manager. Its important to respect upper-case and lower-case.

As with the processes if we want the software agent to notice us immediately when a service is down we must add the parameter module_async yes. The module definition would be:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_async yes
module_end
1.4.6.2.2 Service Watchdog

As with processes there is a watchdog mode for services that allow you to detect and restart a down service almost in realtime. A module definition example using watchdog is like this:

module_begin
module_name ServiceSched
module_type generic_proc
module_service Schedule
module_description Service Task scheduler
module_async yes
module_watchdog yes
module_end

The watchdog definition for services doesn't need any extra parameter because they are in the service definition.

1.4.6.3 Monitoring basic resources

This section describes how to monitor basic variables of a Windows machine.

1.4.6.3.1 Monitoring CPU

To monitor the CPU you can use the parameter module_cpuusage that returns the percentage cpu usage.

Its possible to monitor the cpu based on its id with the following module definition:

module_begin
module_name CPU_1
module_type generic_data
module_cpuusage 1
module_description CPU usage for CPU 1
module_end

Also you can monitor the average of CPU usage from all system with the module:

module_begin
module_name CPU Usage
module_type generic_data
module_cpuusage all
module_description CPU Usage for all system
module_end
1.4.6.3.2 Monitoring memory

To monitor the memory you can use two parameters module_freememory that returns the amount of free memory in the system and module_freepercentmemory that returns the percentage of free memory.

An example module usgin module_freememory would be:

module_begin
module_name FreeMemory
module_type generic_data
module_freememory
module_description Non-used memory on system
module_end

A module using module_freepercentmemory would be defined in this way:

module_begin
module_name FreePercentMemory
module_type generic_data
module_freepercentmemory
module_end
1.4.6.3.3 Monitoring disk

To monitor the disk space you can use two parameters: module_freedisk that returns the amount of free space and module_freepercentdisk that returns the percentage of free space. Both parameters require the unit to monitor as an input, don't forget the character «":"».

A module that uses module_freedisk is define in this way:

module_begin
module_name FreeDisk
module_type generic_data
module_freedisk C:
module_end

An example moudel that uses the parameter module_freepercentdisk is defined with this structure:

module_begin
module_name FreePercentDisk
module_type generic_data
module_freepercentdisk C:
module_end

1.4.6.4 WMI queries and SQL queries via ODBC

Pandora FMS software agent allows you to extract information using WMI queries and ODBC connexions, very common technologies used to store information related to the system or external to it.

1.4.6.4.1 WMI queries

Using the parameter module_wmiquery the software agent allows you to execute any local WMI query you want. To do the query you must set the parameter module_wmiquery with the query that will be performed and module_wmicolumn with the column which has the information to monitor.

For example, we can get a list with the services installed:

module_begin
module_name Services
module_type generic_data_string
module_wmiquery Select Name from Win32_Service
module_wmicolumn Name
module_end

Using WMI we can also get the CPU load:

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end
1.4.6.4.2 ODBC queries

This module allows you to get generic information from a database using the ODBC interface. This module only works with 32 bits driver for ODBC so if you are using a 64 bits version of Windows you must ensure that you are using the 32 bits driver for ODBC.

This module makes easy the access to data no matter what database server you are using Microsoft SQL Server, Oracle, MySQL o PostgreSQL.

To use ODBC modules you must define an ODBC handler first in the main section of software agent configuration file, an example would be like this:

odbc_WebDSN_username UserNameForDsn
odbc_WebDSN_password Password1234

With this two parameters we create the DSN handler called "WebDSN" that we will use later:

An example performing a query could be:

module_begin
module_name Web_Users
module_type generic_data
module_odbc WebDSN
module_odbc_query SELECT COUNT(*) FROM webdb.users_logged 
module_description Get users logged
module_end

This example performs a query to a database called webdb and counts the number of users logged in the system.

  • NOTE: Nowadays the ODBC module only returns the first row of the query.

You can monitor data from multiple databases creating other ODBC handlers and any modules you need.

1.4.7 Remote checks with software agents

Remote check performed by the agent make easy to monitor clompex network with special requirements for example related to security issues. This sections explains how to use this feature of software agent.

1.4.7.1 ICMP checks

ICMP or ping checks are very useful to know if machine is connected to a network. This way one software agent can monitor the status of the whole network easily.

Unix

Using Unix software agent you can use the system commands to create a module that performs a ping check. The module definition would be:

module_begin
module_name Ping
module_type generic_proc
module_exec ping -c 1 192.168.100.54 >/dev/null 2>&1; if [ $? == 0 ]; then echo 1; else echo 0; fi
module_end

In this example module we perform a ping checl to the host 192.168.100.54, if you want to check another host you only need to change the IP.

Windows

The software agent for Windows platforms has some parameters to configure the ping checks and are the following:

  • module_ping_count x: Number of ECHO_REQUEST packages to send (By default 1).
  • module_ping_timeout x: Timeout in miliseconds (By default 1000).
  • module_advanced_options: Opciones avanzadas para ping.exe.

An example of a module configuration could be:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

In this example we perform the same check that in the previous one, but now we use the software agent for Windows platforms.

1.4.7.2 TCP checks

TCP checks are useful to verify if a port of a host is open. It could be interesting to know if an application is connected to the network or not.

Unix

With the software agent for Unix platforms we could perform the TCP check with the following module:

module_begin
module_name PortOpen
module_type generic_proc
module_exec nmap 192.168.100.54 -p 80 | grep open > /dev/null 2>&1; echo $?; if [ $? == 0 ]; then echo 1; else echo 0; fi
module_timeout 5
module_end

With this module we check if the port 80 of the host 192.168.100.54 its open or closed.


Windows

If we are using the software agent for Windows we have some parameters to configure the TCP check. The parameters are:

  • module_tcpcheck: Host to be checked
  • module_port: Port to be checked
  • module_timeout: Timeout for the check

An example of a module definition would be:

module_begin
module_name TcpCheck
module_type generic_proc
module_tcpcheck 192.168.100.54
module_port 80
module_timeout 5
module_end

This module is the equivalent for the Windows software agent to perform the check on 80 port of host 192.168.100.54

1.4.7.3 SNMP checks

SNMP checks are commonly used to monitor network devices to check interface status, inbound/outbound bytes, etc.

Unix

If you are using the software agent for Unix platforms you can create the module using the command snmpget like the following:

module_begin
module_name SNMP get
module_type generic_data
module_exec snmpget 192.168.100.54 -v 1 -c public .1.3.6.1.2.1.2.2.1.1.148 | awk '{print $4}'
module_end

This module returns the value for OID .1.3.6.1.2.1.2.2.1.1.148 of the host 192.168.100.54.

Windows

For Windows software agent we have the following parameters to configure the module:

  • module_snmpversion [1,2c,3]: SNMP version (By default 1).
  • module_snmp_community <community>: SNMP community (By default public).
  • module_snmp_agent <host>: Host to monitor.
  • module_snmp_oid <oid>: OID.
  • module_advanced_options: Advanced options for snmpget.exe.

An example module could be:

module_begin
module_name SNMP get
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.148
module_end

This module is the equivalent for Windows platforms to do the last check performed using Unix software agent.

1.4.8 Proxy Mode

Pandora FMS software agents have a Proxy Mode that allows them to act as proxies redirecting the the communication of several agentrs to Pandora FMS server. The software agent that has Proxy Mode enabled can perform monitoring task too.



Proxy agent schema.png



The Proxy Mode is very useful when you are dealing with a network in which only one machine can communicate with Pandora FMS server. In this situation all software agents installed in machines without access to Pandora FMS server will send the XML files to the agent working as a proxy and it will redirect all files to Pandora FMS server.

In addition to XML file redirection the Proxy Mode supports Remote Configuration and File Collection features.

To enable the Proxy Mode in a software agent you must configure the parameters:

  • server_ip: IP of Pandora FMS server. With Proxy Mode enabled the IP can't take this value: 127.0.0.1, localhost, 0.0.0.0, o similar.
  • proxy_mode: If is set to 1 enabled Proxy Mode. By default is 0, disabled.
  • proxy_max_connection: Max. number of connections for proxy, by default 10.
  • proxy_timeout: Proxy tiemout, by default 1 second.

An example of configuration could be:

server_ip 192.168.100.230
proxy_mode 1
proxy_max_connection 20
proxy_timeout 3

To reditect an software agent connection you only must to set the IP address of software agent with Proxy Mode enabled as the IP address of Pandora FMS server in the agents with limited access. For example:

Our agent with Proxy Mode enabled has this IP: 192.168.100.24

In the software agent that can't connect directly with Pandora FMS server we configure the parameter server_ip with this IP:

server_ip 192.168.100.24


This configuration allows the software agent with limited access to use the other software agent with Proxy Mode enabled to communicate with Pandora FMS server.

1.4.9 Broker Mode

The software agent has a Broker Mode that allow one agent to monitor and manage the configuration as if several software agents would be installed.



Broker agent schema.png



The software agent with Broker Mode enabled has an auxiliary configuration file, similar to software agent main configuration file, for each broker defined. In this way the software agent behaviour is the same as if several software agent would be installed in the same machine, it means, each broker works in the same way that a software agent does. This feature is very useful to manage several devices remotely only installing one software agent, in this way is possible to monitor several devices with different configuration files but installing only one software agent.

The main features of Broker Mode are:

  • Send local data as other agent. Very useful to monitor different software instances as different agents.
  • Send data collected from remote checks to other machines as if a software agent had been installed on them.

To create a broker you only need to add a line with the parameter broker_ Para crear un broker sólo tiene que añadir una línea con el parámetro broker_agent <nombre_broker> como estas:

broker_agent dev_1
broker_agent dev_2

Once the brokers were created the configuration files dev_1.conf and dev_2.conf will be created with the same content that in the original software agent only the agent name will be changed. Adding or deleting modules from configuration files dev_1.conf and dev_2.conf we can customize the checks performed by the brokers.

Inside Pandora FMS web console the brokers appear and are managed as the other agents. It means that if we have a software agent installed with two brokers, in the web console we will see three different agents with their modules, configurations, etc.

1.4.9.1 Examples of use

1.4.9.1.1 Monitor a local database as a different agent

We want to monitor the basic parameters of a machine (CPU, memory and disk) and also there is a database installed that we want to monitor separately.

To perform this monitoring we will use the following structure:

  • Software agent installed: monitoring CPU, memory and disk.
  • Broker for database: monitoring internal status of database.

To do this we install the software agent in the machine to monitor CPU, memory and disk. In the software agent configuration we add the following line:

broker_agent DBApp

With this line we create a broker agent called DBApp because of that a configuration file named dbapp.conf will appear. In this configuration file we add the modules to perform checks to database:

module_begin
module_name Num Users
module_type generic_data
module_exec get_db_users.pl
module_end

module_begin
module_name Num slows queries
module_type generic_data
module_exec get_db_slows_queries.pl
module_end

With this you will see two agents in Pandora web console one with the name of the machine and with the modules CPU memory and disk and the other called DBApp with the modules Num Users and Num slows queries.

1.4.9.1.2 Monitor devices remotely using brokers

For this example we have a software agent installed in a Windows machine monitoring (CPU, memory and disk) and we need to monitor a router with IP 192.168.100.54 without installing an agent inside. To solve the problem we can use brokers.

We create a borker using the parameter;

broker_agent routerFloor5

With this line we create a boker agent with the name routerFloor5 and because the software agent were installed in a Windows machine we can monitor the router using the ping and snmp modules available for Windows software agents. To do that we modify the file routerFloor5.conf adding the lines:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

In this example the web console of Pandora FMS shows two agents, one is the Windows machine with the modules CPU, memory and disk and the other is routerFloor5 with the modules: Ping, Eth 1 up and Eth 2 up.

1.4.9.1.3 Remote monitoring of network with restricted communication

In some cases is needed to monitor devices remotely but Pandora FMS Remote Server can't access to them directly.



Broker example no access.png



In this example we must monitor remotely the devices of one of the company sites from the headquarters. Pandora FMS server is in the headquarters connected to the other company sites using a VPN, due to some restrictions the Pandora Remote Server can't access the machines remotely. To monitor the company sites we will use the Broker Mode that allows a software agent to send XMLs to Pandora server as if it would be several different devices.

In the configuration file of software agent we add as many brokers as devices to be monitored. A configuratio example could be:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

Once the broker were created we can customize the monitoring for each devices modifying the configuration file of each broker. For example the configuration for the Windows machine called device_1 is:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

With this configuration we can use remote configuration feature and we can send monitoring information to Pandora FMS server despite of the restrictions in communication between the different company sites.

1.4.9.1.4 Share monitoring load through brokers

The Broker Mode is very useful to distribute the monitoring load within several network points.



Broker scalation example.png



In this example our architecture has several networks named from A to Z with 1000 devices each one. The capacity of Pandora FMS Remote Server is about 2000 agents, because of that we decide to use software agents with Broker Mode enabled to distribute the load. These software agents with Broker Mode enabled monitor remotely all devices from the network and send the data in XML format to Pandora FMS central server.

For each network we have an agent with Broker Mode enabled, inside it we create as many brokers as devices to be monitored, configuration for the software agent Broker_Agent_Net_A would be:

broker_agent device_1
broker_agent device_2
broker_agent device_3
broker_agent device_4
...

In addition for each broker we will add the modules to monitor the devices. For example the broker device_1 which is a router would have this configuration:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name Eth 1 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.1
module_end

module_begin
module_name Eth 2 up
module_type generic_data
module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.2
module_end

Another example would be the broker device_2 which monitors a Windows machine with the following modules:

module_begin
module_name Ping
module_type generic_proc
module_ping 192.168.100.54
module_ping_count 2
module_ping_timeout 500
module_end

module_begin
module_name CPU_Load
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Processor
module_wmicolumn LoadPercentage
module_end

module_begin
module_name Mem_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Memory
module_wmicolumn FreeMemory
module_end

module_begin
module_name Disk_Free
module_type generic_data
module_wmiquery SELECT LoadPercentage FROM Win32_Disk
module_wmicolumn FreeSpace
module_end

Using software agents with Broker Mode enabled we can share the load to collect data from thousand of devices easily.

1.4.10 Inventory using software agents

Pandora FMS software agent support inventory features for both hardware and software. The inventory system allows you to get a history of CPU, cards, memory, patches, software, etc … used in the company servers. Furthermore is possible to generate alerts when a change occurs in the inventory, for example if a disk was replaced or if an application was deleted.

You can find more information in section Local Inventory with Software Agents .

1.4.11 How to ask agent for information on demand

Until 3.2 version, you didn't have any way for asking the remote software agent for information, you should wait to the agent to reach its interval limit and wait it to send its information. Windows agent 3.0 has a non very known feature called "UDP Server" which allow to receive communications from outside to ask for information and to force the agent to refresh its cycle, forcing it to send the information to the server.

Now, in 3.2 version we have implemented the same feature: REFRESH AGENT in the Unix agent as well, and we have included a "default" alert template and command to make it easy. You now can setup your agents (Windows and Unix) to receive orders from the console to report data immediately, without waiting for it's interval.

This feature is pretty simple, first you need to setup your agent (windows or Linux) to accept outside connections on a specific UDP port, from a specific IP address (or 0.0.0.0 for anyone). In Windows you can also define other possible things the agent can execute, as a result of a remote command. On Unix the only supported (at this time) operation is "REFRESH AGENT". That will result on an immediate agent execution, skipping its interval.

This is a sample of the UDP server settings in the Unix software agent v3.2 :

udp_server 1
udp_server_port 41122
udp_server_auth_address 0.0.0.0

Enable the server with 1 and disable with 0 in "udp_server" option. Set 0.0.0.0 as source ip address to enable any IP address.

This is a sample of the UDP server settings in the Windows software agent v3.x:

udp_server 1
udp_server_port 41122
udp_server_auth_address 192.168.1.23

Is exactly the same, and in this case, we use the address 192.168.1.23 as the only authorized IP address to refresh this agent.

Pandora FMS server has a small script, which send the order to the agent. In the default command we have created, its fully operational and ready to be used. This script is a small Perl script which acts as a small client to communicate with the simple UDP server embedded in the agent and sent commands passed in command line.




Remote agent command.png



We also provide a default alert template to assign "manual" alerts to an agent, that means, an alert which never will be fired automatically. You will use "manual alerts" to force execution manually using the round button on the agent main view, to force the execution of the alert command.




Alert template manual.png



We have created a default alert action called "Restart agent", which call to the remote agent command. This action pass the REFRESH AGENT command to the command, and uses the main IP address of the agent to reach, using the default port for the UDP server (41122/UDP):




Agent restart action.png



Follow these steps to enable the software agent remote refresh option:

1. Set up the options in the configuration file for the software agent (Unix or Windows). Please take care of the authorized ip address (is Pandora FMS server behind a NAT?), or put 0.0.0.0 to allow any IP address to force refresh the agent.

2. Restart the software agent.

3. You need to setup the IP address on your agent item in the Pandora FMS console. Alert action will use the IP address to connect the software agent running in the remote system.

4. Set an alert to any of the modules of that agent (no matter which), using this screenshot as a sample guide:




Alert restart combinationsettings.png



5. You're ready now to force a refresh for that agent using the main view, clicking in the round green button at the left of the alert you've just defined:



Agent refresh button.png



Anytime you want to get the information from the agent "at once" without waiting for agent interval, just click in the button and wait a few seconds. Agent will be contacted and forced to execute, XML will be transferred to your Pandora FMS server and it will be processed, depending on your system load, it will be processed in 1-5 secs and displayed in the console.

1.4.12 Using software agent plugins

Agent plugins are executed by the software agent, and could report several information (modules) at once. Each plugin works in a different way and you should test how it works before using it. Default instalation of Pandora FMS comes with a bunch of plugins. Of course, Unix agents and Windows agents comes with different plugins, some of them works very similar.

1.4.12.1 On Windows systems

In 3.2 version, windows agent comes with following plugins:

  • df.vbs: Reports disk space in bytes. Report different modules for each harddisk found on system. If you want only to report specific units, just use parameters after the module_plugin call.
  • df_percent.vbs: Very similar to previous one, but will report free space on %. Will generate modules with a name like "DiskFree_C".
  • logevent_log4x.vbs: Read eventlog entries and generate log4x data.
  • ps.vbs: It will require process names, and check if that process are running. For example: if you execute it with "iexplorer.exe mucommand.exe other.exe" will check for three processes and will return different modules for each of them, reporting if proccess is down or alive.

In windows, all default plugins are coded in VBScript. To run it you will need to use the correct interpreter for console VBScript, sometimes referred as Windows Scripting Host

These are examples of usage of previous plugins:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\logevent_log4x.vbs" Aplicacion System 300

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" iexplore.exe myapp.exe

1.4.12.2 On Unix systems

In 3.2 version, generic Unix agent comes with following plugins:

  • files_indir: This plugn receives a target directory, for example "/tmp" and will return two modules, one called "FS_/tmp/" (boolean) returning 1 if contains the same number of files than in the previous execution, and other module "NumFiles_FS_/tmp/" with the number of files in that directory.
  • grep_log: Is a generic log parser, it takes tree arguments: <file_to_process> <module_name> <reg_exp>. It will generate information inside a async_string moduletype called <module_name> using all data which match the <reg_exp> regular expression. See example below on this plugin.
  • pandora_df: Very similar to the Windows plugin, will report available space on all mounted partition on system. It also takes information from the NFS mounts. By default information for all filesystems is returned, but one or more filesystems may be specified as plugin parameters.

These plugins are very similar to Windows plugins. We don't need to use the full path to plugins, because module_plugin directive look for "plugin" directory under the agent's home dir. For execute them, we use this syntax:

module_plugin grep_log /var/log/syslog Syslog .

module_plugin pandora_df tmpfs /dev/sda1

And some special plugins working on Unix:

  • nagios_plugin_wrapper: This is not really a "plugin"; this is just a wrapper used to execute a Nagios plugin and process the output to generate a pandora module. It takes the standard output and put the result in the module description and gets the errorlevel to process a module_proc (boolean) module with its results. Just call the nagios plugin as parameter to nagios_plugin_wrapper with all it's needed parameters and it will generate a Pandora Fms module.
  • inventory: This is used in the inventory system. It will create a Inventory XML with information about the system, could be modified to gather different information, but default works only in Linux and gets packages, services in default runlevel and other options. Checkout the inventory documentation for more information.
  • pandora_update: This is used to use the autoupdate feature on software agents, checkout the agent configuration section for more information.

1.4.12.3 Examples using plugins

The plugins for software agents can return one data or a group of them. An example of a plugin that only returns one data could be the plugin ps.vbs for Windows. With the following line we execute the plugin:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\ps.vbs" IEXPLORE.EXE

The result will be a module that return 0 if the process is down and 1 if the process is running:

<module>
    <name><![CDATA[IEXPLORE.EXE]]></name>
    <description><![CDATA[Process IEXPLORE.EXE status]]></description>
    <![CDATA[1]]>
</module>

An example of a plugin that returns several data could be the plugin df.vbs for Windows. The line to execute the plugin would be:

module_plugin cscript.exe //B "%ProgramFiles%\Pandora_Agent\util\df.vbs"

The plugin returns a module per disk found, the result would be:

<module>
    <name><![CDATA[C:]]></name>
    <description><![CDATA[Drive C: free space in MB]]></description>
    <![CDATA[8050]]>
</module>

<module>
    <name><![CDATA[D:]]></name>
    <description><![CDATA[Drive D: free space in MB]]></description>
    <![CDATA[900]]>
</module>

1.4.13 Plugins of software agent editor

Since Pandora FMS 5.0, is possible manage the software agent plugins from the console without edit directly the configuration file.

If an agent has the remote configuration enabled, it will has the plugins editor tab in the manage view.



Plugin editor tab.png

The editor shows the plugins list, and allow to delete. add and disable it. In the policy plugins can be useful disable it because when the policy will be applied again, the plugins still disabled.



Plugin editor.png

The managed plugins from this editor can be edited from the agent configuration file too.



Plugin editor conf.png

1.4.14 How to create your own agents plugins

The plugins can be developed in any programming language. They must respect this two restrictions:

  • Whatever you want to do, it must be automatic (no interactive processing from the user), and must be done from the commandline (shell). You can use any kind of scripting language or compiled language, but in that case you must provide a standalone executable with all it's dependencies (libraries, dll, etc).
  • Plugin must report the information to the standard output (just using echo, printf, or the equivalent in your language), and use the XML syntax for Pandora FMS agent information, this is an example of a generic_data (numerical information) XML:
<module>
<name><![CDATA[Sample_Module]]></name>
<type><![CDATA[generic_data]]></type>
<![CDATA[47]]>
<description><![CDATA[47]]></description>
</module>

The <![CDATA[xxx]]> are used to "enclose" data and protect the XML from non-valid characters like <,>,& or %.

Before trying to create your own plugin, take a look at our Pandora FMS plugin library at http://pandorafms.org, and if you want to create your own, please upload to Pandora FMS public library to allow others use your plugin!.

1.4.15 Using nagios plugins from the agent

Nagios has a lot of amazing plugins you can use with Pandora FMS. One way is to use remote plugins with the Plugin Server, using the nagios compatibility, but they only get it's status, doesnt use the descriptive output which some plugins for nagios have.

Using the wrapper for use nagios plugins in the software agent will solve this problem. The wrapper comes with default with 3.2 Unix agent. A equivalent plugin for Pandora FMS Windows agents can be downloaded from our website at http://pandorafms.org resource library at [1]).

What does the plugin wrapper for nagios plugins ?

Execute the nagios plugin, using it's native parameters, and converting the output in a data useful for Pandora FMS, it has two kind of information:

  • Status information: NORMAL (1), CRITICAL (0), WARNING (2), UNKNOWN () and other (4). By default, it will use a proc module, so NORMAL and CRITICAL values are working "by default"; if you want to have information on WARNING and OTHER values you should setup the module thresholds manually.
  • Descriptive information: Usually string information, will be put on the description field on module, this is usually something like "OK: successfully logged in" or similar.

1.4.15.1 Example

You have a pop3 plugin (in /tmp/check_pop3_login) with exec permissions, which checks if pop3 account is working, just by connecting a remote host, send a user and password and see if everything is ok, so if you execute it from command line:

/tmp/check_pop3_login  mail.artica.es [email protected] mypass

It will return something like:

OK: successfully logged in.

And if it's not ok, will return :

Critical: unable to log on

Using the wrapper is simple, just need to put the wrapper and the module name you want before the call:

/etc/pandora/plugins/nagios_plugin_wrapper sancho_test /tmp/check_pop3_login  mail.artica.es [email protected] mypass

It will generate a full XML for agent plugin:

<module>
<name>sancho_test</name>
<type>generic_proc</type>
0
<description><![CDATA[Critical: unable to log on]]></description>
</module>

Or:

<module>
<name>sancho_test</name>
<type>generic_proc</type>
1
<description><![CDATA[OK: successfully logged in.]]></description>
</module>

The full entry in the pandora_agent.conf will be something like:

module_plugin nagios_plugin_wrapper POP3_artica.es /tmp/check_pop3_login mail.artica.es [email protected] mypass

This will be seen like this in the module like (on fail event):


Sample plugin wrapper.png

1.4.16 Monitoring with KeepAlive

There is an special module. It has an unique kind called "keep_alive" and is useful to give information when there is no agent contact. Is useful to know when an agent has stopped to send information and to notify us this.

When there is a module, remote or local, that gets information from the agent, the date of the last "contact" with the agent is updated, in a way that there is always data, though it is only a module of the total, the agent will have updated its date of last contact, that is usefule to know if the agent "does not answer". To be precise, an agent is considered "dead" when it has not updated its data in the double of time of its interval, that is, if it has a 5 minutes interval and more than 10 minutes have been past from there is no update, then the agent is considered "dead".

In this case, it would be when the keepalive module will appear, firing it and being in the Critical state in the monitor.

The configuration of these kind of modules is very easy, you need only to create a new module kind "dataserver":



Keepalive create.png


Once created, if the agent has data, in its interval, it will be always in "NORMAL" (green) status.



Keepalive1.png


If the agent stop sending data (in this example, it has interval of 1 minute), the it will automatically pop and it will be on CRITICAL status (red):



Keepalive2.png


It is important to say, that if we have a remote module, for example one Ping, apart from the data reported by the agent, the keepaliva module will never pop, so we are continually updating the agent through the Ping.

Apart from that, the keepalive module works the same as any other module. It could be associated to an alert and it could be used for any other element: reports, maps, etc.

1.4.17 Monitoring command snapshots

This way to monitor, possible from 4.0.3 version, allows administrator to use a special way to capture output from commands, different from parsing a single value or string. This module stores the information as text, but with the purpose to get the exact output of the command, not as single data. It will show you the same output format and contents as was return by the command.

An image is better than words, so:


Snapshot 1.png

This is the "netstat -an" command output, captured by Pandora FMS, after clicking in the special icon for command snapshot, only available when we got a multiline text -non splitted- output.


Snapshot 2.png


These are the different "snapshots" among time. Pandora will show information only when the agent got different information (remember, datatype continues being generic_data_string), if you want to store each data (not recommended!) you can use async_string. Showing information only on chagnes is perfect to report data on problems, and compare other events with the information in the command snapshot for that timerange.

To capture the commands output in a snapshot, you need to write an small pluegin which send all data, adding also the XML tags, and executing the plugin as an standard pluegin, with module_plugin syntax. Let's see the following example, which generates output for command netstat, as you read before.

#!/bin/bash
echo "<module>"
echo "<name>netstat</name>"
echo "<type>generic_data_string</type>"
echo "<data><![CDATA["
netstat -an | grep LIST
echo "]]></data>"
echo "</module>"

Save this script contents in a file in the agent (or remotely distribute with file collections) and execute using this syntax:

module_plugin <full path for file>

In this way, you will, the agent will generate command snapshot almost for any command (just replace "netstat...." by your command). Some useful suggestions for Unix systems are:

* top -b -n 1
* ps aux
* vmstat 1 5
* who 
* last -10

On Windows systems:

* tasklist
* netstat -an
* net start

Info.png

Remember that you need to do always inside an script, generating the XML. If you do that inside a module_exec call, each line reported by the command will be interpreted a line in different "data blocks", and you cannot see it as command snapshot.

 


Go back to Pandora FMS documentation index