Monitoring with EndPoints

EndPoints are installed and are running on the operating systems from which they collect information, performing a check for each module.

The EndPoint's own directives serve to collect certain data directly from the operating system (such as CPU usage, memory, events, etc.), executing operating system-specific commands following predefined scripts instructions.

The Pandora FMS Dataserver processes and stores in the database all the information generated and sent in XML files by the EndPoints.

All configuration and parameters are stored in the pandora_agent.conf file, which is also installed locally along with its EndPoint. Basic configuration is covered in "Pandora FMS EndPoint Configuration" and, for WUX monitoring, in the manual execution as plugin section. Advanced configuration is described below.

In the EndPoint configuration file, modules are defined with the following basic text structure:

module_begin
module_name <your module name>
module_type generic_data
module_exec <your command>
module_description <your description>
module_end

It is possible to remotely manage EndPoint files from the Pandora FMS Web Console. The configuration for each EndPoint is stored on the Pandora FMS server in two files: <md5>.conf and <md5>.md5, where <md5> is the hash of the EndPoint name. These files are stored respectively in:

/var/spool/pandora/data_in/conf

and

/var/spool/pandora/data_in/md5

Custom fields allow you to add additional information to the EndPoint. Custom fields can be created with PFMS API 1.0 and the set create_custom_field command or through the Web Console in the Management → Resources → Custom fields → Create field menu.

• The Enabled combo, Password type and Link type options are mutually exclusive; that is, only one of them (or none, default value) can be used.
• When the Display up front field is activated, the custom field information will be shown, if it has an established value, in the EndPoint general view. If no marked custom field has a value, the custom fields box will remain hidden.

  Additionally, it will be necessary to activate this token to send the Custom Fields information to the Command Center.

• Enabled combo: This parameter allows activating the configuration of parameters selectable from a dropdown. Once activated, a new field will appear in the corresponding custom field configuration window to enter the combo values separated by commas.
• Password type: The field value (password) will be shown through asterisks in the Web Console.
• Link type: Allows adding a custom field that will house a web link to be filled in via Web Console or in an XML received by an EndPoint. It is possible to include links in the custom fields of an XML in JSON format embedded with CDATA <![CDATA[…]]> instructions. If the JSON format of the link is:

["Web name","https://example.com"]

The XML would have this syntax:

<custom_fields>
  <name>![CDATA[web]]</name>
  <value>![CDATA[["Web name","https://example.com"|]]]</value>
</custom_fields>

Consult “XML Validation”, the Security Architecture for the Tentacle protocol (mechanism in charge of delivering data in XML format to the PFMS Data server), and the Security Architecture for the PFMS Data server (limiting EndPoint auto-creation and establishing a password for the EndPoint group to which each EndPoint belongs).

The custom fields can also be passed from the EndPoint configuration file, using the tokens custom_fieldx_name and custom_fieldx_value:

custom_field1_name Serial Number
custom_field1_value 56446456KS7000

The custom field named Serial Number is created by default when installing PFMS. As many custom fields as needed can be created, each of a different type (simple value, web link, password type, and options list type). The order of the numerical identifier for each custom field is irrelevant; you only need to ensure the name is exactly the same:

custom_field11_name Simple custom field name
custom_field11_value Simple custom field value

custom_field12_name Custom field Link type
custom_field12_value ["Pandora FMS web site","https://pandorafms.com"]

custom_field13_name Custom field Password type
custom_field13_value My;Password;

custom_field14_name Custom field Combo type
custom_field14_value Two

In Combo type custom fields, the value sent by the EndPoint must correspond exactly to one of its items; otherwise, the value will not be changed.

Most important parameters for the basic configuration of EndPoints:

• server_ip: IP address of the Pandora FMS server.
• server_path: Path to the Pandora FMS server incoming folder, by default

  /var/spool/pandora/data_in

• temporal: Folder, by default /tmp.
• interval: EndPoint execution interval, by default 300 seconds.
• logfile: EndPoint log file, by default:

  /var/log/pandora/pandora_agent.log

It is possible to configure a password (no spaces or symbols) for a group; this way, an EndPoint will not be added to a group unless the correct password has been specified in the EndPoint configuration file.

To add a new EndPoint to this group, edit its configuration file, add the following configuration option group_password, and restart the EndPoint.

According to the returned data:

• generic_data: Numerical.
• generic_data_inc: Incremental.
• generic_data_inc_abs: Absolute incremental.
• generic_proc: Boolean.
• generic_data_string: Alphanumerical.
• async_data: Asynchronous numerical.
• async_string: Asynchronous alphanumerical.
• async_proc: Asynchronous Boolean.
• Image module: They use a string-type module (generic_data_string or async_string) as a base. If the data contained in the module is a base64 encoded image (header data:image), it will be identified as an image and will enable a link to a window to retrieve the image in the views. Additionally, a history of the different images that make up the stored strings will be displayed in their respective history.

Local (or EndPoint) modules all have their EndPoint's interval as a “base”. However, they can take values that are multiples of that base if you modify the parameter module_interval with an integer multiplier greater than zero.

The creation of local modules in the Web Console is done through a form where, in addition to the common configuration for all modules (thresholds, type, group, etc.), a text box is available where the data to be set in the EndPoint configuration file is specified.

• By clicking the Load basic button, the content of Data configuration will be cleared with a basic template that must be modified according to the monitoring need.
• Once modified, clicking Check will verify that the template syntax is still correct (however, the rest of the commands will not be checked).

The EndPoint supports the execution of commands and scripts in post-condition mode. This means that actions can be performed depending on the value obtained in the module execution. The parameter module_condition is used for this purpose.

The parameter module_precondition allows evaluating a condition before the module execution and deciding with the result whether the module should be executed or not.

There are certain modules of special importance, such as critical processes or services in execution. To have more controlled monitoring of these cases, intensive monitoring exists.

It consists of warning in a shorter interval that a serious problem has appeared without the need to reduce the EndPoint's general interval.

EndPoint Configuration:

• interval: Mandatory, EndPoint sampling time in seconds, it is the general interval for all local modules.
• intensive_interval: Time in which it will notify if there is a problem, it will always be executed in this period, and if it matches the condition, it will be notified in this time period (otherwise, data will be sent in the interval).

Module Configuration:

• module_intensive_condition = <value>: If the module obtains the value indicated in this parameter as a result, it will notify in the previously defined intensive interval.
• Other operators that can be used are: <, >, !=, a range of values (m,n) and =~.

The sshd service is very important as it is used to connect via shell remotely and its operation needs to be monitored.

EndPoint configuration file:

interval 300
intensive_interval 10

Module:

module_begin
module_name SSH Daemon
module_type generic_data
module exec ps aux | grep sshd | grep -v grep | wc -l
module_intensive_condition = 0
module_end

If the service is missing, it will be notified within the next 10 seconds; if it is functioning, it will notify every 5 minutes (normal interval, 300 seconds).

The EndPoint supports the definition of scheduled modules that run at defined times. The syntax used is the same as that of the crontab file.

An EndPoint is capable of performing remote checks, replacing the main PFMS server and even distributing them in broker agents.

ICMP or ping checks are very useful to know if a machine is connected or not to a network.

Unix:

module_exec ping -c 1 dir_IP> /dev/null 2>&1; if [ $? -eq 0 ]; then echo 1; else echo 0; fi

MS Windows®:

module_ping dir_IP

Note: module_advanced_options allows advanced options for ping.exe.

TCP checks are useful for verifying that a machine's ports remain open and allowing knowledge of whether an application connects to the network or not.

Unix:

With the nmap command and its configuration parameters on the command line, it checks if port 80 is open at an IP address (response timeout of 5 seconds):

module_begin
module_name PortOpen
module_type generic_proc
module_exec nmap 192.168.100.54 -p 80 | grep open > /dev/null 2>&1; echo $?; if [ $? == 0 ]; then echo 1; else echo 0; fi
module_timeout 5
module_end

MS Windows®:

Parameters must be specified in:

• module_tcpcheck: Device IP address.
• module_port: Port number.
• module_timeout: Wait time for the response, must be located between module_begin and module_end.

Given case:

module_begin
module_name TcpCheck
module_type generic_proc
module_tcpcheck 192.168.100.54
module_port 80
module_timeout 5
module_end

SNMP checks are common in network device monitoring to check the status of interfaces, input/output bytes, etc.

Unix®:

module_exec snmpget dir_IP -v 1 -c public .1.3.6.1.2.1.2.2.1.1.148 | awk '{print $4}'

MS Windows®:

module_snmpget
module_snmpversion 1
module_snmp_community public
module_snmp_agent 192.168.100.54
module_snmp_oid .1.3.6.1.2.1.2.2.1.1.148
module_end

This mode allows redirecting data files generated by other EndPoints to the Pandora FMS server. The EndPoint acting in Proxy Mode can also perform monitoring tasks.

Parameter configuration:

• server_ip: Pandora FMS server IP address.
• proxy_mode: Activated (1) or deactivated (0).
• proxy_max_connection: Number of simultaneous proxy connections, by default 10.
• proxy_timeout: Wait time for response for the proxy, by default 1 second.
• proxy_address: IP address on which the proxy listens.
• proxy_port: Port number on which the proxy listens.

The EndPoint Broker Mode allows a single EndPoint to perform checks and manage configuration as if it were several different EndPoints.

When Broker Mode is activated in an EndPoint, a new configuration file is created. From that moment on, the original EndPoint and the new Broker will be managed separately with their independent configuration files, as if they were two totally separate EndPoints on the same machine.

To create a Broker, one or more lines with the parameter broker_agent <broker_name> are added (one line for each Broker).

In the Pandora FMS Web Console, Brokers are seen and managed as independent EndPoints.

For more information, visit the Local Inventory with EndPoints section.

For more information, visit the Log Collection and Monitoring topic.

An EndPoint is capable of receiving remote requests and executing orders.

To allow the PFMS server to send orders to the EndPoints under its charge, the following must be configured:

• udp_server: Zero 0 by default, set to one 1 to activate this functionality.
• udp_server_port: Listening port number on EndPoint.
• udp_server_auth_address: IP address of the Pandora FMS Server.

The EndPoint must be restarted for the changes to apply.

The script located in must be used:

/usr/share/pandora_server/util/udp_client.pl

It can be executed from the command line or used in an alert, using the command that comes pre-configured in the console Remote agent control.

In addition to the action of restarting the EndPoint service, custom actions of the following type can be specified:

process_<order_name>_start command

Unlike server plugins, executed by Pandora FMS server, EndPoint plugins report one or several modules at the same time.

In MS Windows®, the plugins registered by default are programmed in VBScript; to execute them, the cscript.exe interpreter is used.

From version 776 onwards, module_exec_powershell is available, which allows introducing more complex commands in PowerShell® with special characters and complex instructions (one instruction delivers results to the next) that are impossible through the module_exec module.

# Example of Powershell execution module
module_begin
module_name Powershell
module_type generic_data_string
module_exec_powershell <command_1> | <command_2> | … | <command_N>
module_end

Commands are entered as they are, without the need for quotation marks to be processed by the PFMS EndPoint (PowerShell commands, on the other hand, may need quotation marks).

If the command is not valid, an error is added to the EndPoint log (pandora_agent.log file).

Unix plugins are located by default in the EndPoint directory:

/usr/share/pandora_agent/plugins

It is possible to add a token in the EndPoint plugins configuration that, when enabled, allows the option of 'encapsulating' plugin definitions within the module_begin and module_end tags.

This enabled token allows inserting configuration blocks such as module_interval or module_crontab, among others.

plugins can be created in any programming language. You only need to take into account the general rules and the specific rules for their development.

Nagios has a large number of plugins that can be used with Pandora FMS. One way to do it is to use remote plugins with the Heavy Server, using Nagios compatibility.

A unique module in Pandora FMS is the type called keep_alive, used to alert if an EndPoint has stopped sending information.

You must go to the agent administration menu in Management → Resources → Manage agents and click on Modules of an agent.

Once the module list of the selected agent is visible, you must press the Create module button and select Endpoint module in the Select module type list and press Create. The module must be created as follows:

Commands that present extensive outputs, such as top or netstat -n, can be completely captured by a module and reproduced as they are. The module must be configured as a text type in the remote configuration similarly to the following code:

module_begin
module_name process_table
module_type generic_data_string
module_exec ps aux
module_description Command snapshot of running processes
module_group System
module_end

In the main agent editing view, module bottom section, the Data column will present an icon with the description Command Snapshot view where clicking will open a new window:

This method allows defining string-type modules (generic_data_string or async_string) that contain images in text format with base64 encoding, being able to show said image instead of a specific result.

A script file is created such as:

#!/bin/bash
echo "<module>"
echo "<name>Actual leader</name>"
echo "<type>async_string</type>"
echo "<data><![CDATA[data:image/jpeg;base64,/9j/4AAQSkZ....]]></data>"
echo "</module>"

Save that content to a file on the EndPoint (or distribute it by collections) and execute it as follows:

module_plugin <complete_path_to_the_file>

The module_proc parameter checks if a certain process name is operating on this machine. It should be configured similarly:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_description Process Command line
module_end

The module_async yes parameter must be added:

module_begin
module_name CMDProcess
module_type generic_proc
module_proc cmd.exe
module_async yes
module_description Process Command line
module_end

The Watchdog functionality for MS Windows® allows restarting an interrupted process.

In the following code, it is configured so that the Notepad program runs persistently through the installed EndPoint:

module_begin
module_name Notepad
module_type generic_data
module_proc notepad.exe
module_description Notepad
module_async yes
module_watchdog yes
module_user_session yes
module_start_command "%SystemRoot%\notepad.exe"
module_startdelay 3000
module_retrydelay 2000
module_retries 5
module_end

The module_service parameter checks if a certain service is running on the machine. The definition of a module using this parameter would be:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_end

The module_async yes parameter should be added to notify immediately when a process stops working:

module_begin
module_name Service_Dhcp
module_type generic_proc
module_service Dhcp
module_description Service DHCP Client
module_async yes
module_end

It works similarly to the Process Watchdog:

module_begin
module_name ServiceSched
module_type generic_proc
module_service Schedule
module_description Service Task scheduler
module_async yes
module_watchdog yes
module_end

Some modules come active and others must be activated via Remote Configuration (or by locally editing the EndPoint's .conf file).

←Back to Pandora FMS documentation index