Difference between revisions of "Pandora: Documentation en: Architecture"

From Pandora FMS Wiki
Jump to: navigation, search
(The XML Data File)
(Topologies, Schemes and Monitoring Models)
 
(48 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
[[Pandora:Documentation_en|Go back to Pandora FMS documentation index]]
 
[[Pandora:Documentation_en|Go back to Pandora FMS documentation index]]
  
= The Pandora FMS Architecture =
+
{{WIP}}
  
This chapter provides a general description of Pandora’s components, the way they relate to each other and how to use the Pandora FMS architecture to meet different challenges regarding the topology of its infrastructure.
+
= Pandora FMS Architecture =
  
Pandora FMS is modular and decentralized. The most important component is the MySQL database, where everything is stored. Each component of Pandora FMS can be replicated and works in a pure [http://en.wikipedia.org/wiki/High_availability '''HA'''] environment, be it passive, active or in a clustered environment (Active/Active with load balancing). There are also descriptions for methods to setup a high availability [http://en.wikipedia.org/wiki/SQL '''SQL'''] backend.
+
This chapter provides a general description of Pandora FMS and its components, the way they relate to each other and how to use the Pandora FMS architecture to meet different challenges regarding the topology of its infrastructure.
 +
 
 +
Pandora FMS can be modular and decentralized or simple and monolithic. The most vital component is '''the MySQL database''', where all the information is stored. Each component of Pandora FMS can be replicated and works in a full HA environment, be it passive, active or in a clustered environment (Active/Active with load balancing).  
  
<br>
 
 
<center>
 
<center>
 
[[image:Arquitectu_pando.png|center|700px]]
 
[[image:Arquitectu_pando.png|center|700px]]
Line 14: Line 15:
 
<br>
 
<br>
  
Pandora FMS consists of several elements - among them the Servers, which are in charge of collecting and processing data. The server also inputs the collected and processed data into the database.   The console is the part in charge of displaying the data present in the database and of interacting with the end user. The software agents are the applications which run on the monitored systems (usually servers), collect the information and send it to the Pandora FMS server.
+
Pandora FMS consists of several elements, among them, the ones that are in charge of collecting and processing the data, which are the servers. The servers, with the information generated by themselves or by the agents, enter the data into the database. The console is the part in charge of displaying the data present in the database and interacting with the end user. Software Agents are applications that run on monitored systems, and collect the information to send it to Pandora FMS servers.
  
 
== Pandora FMS Servers ==
 
== Pandora FMS Servers ==
 
<br>
 
<br>
Under Pandora FMS, there are twelve different servers in total, specialized in and responsible for the various tasks necessary to make Pandora what it is today.  The servers are integrated into a single application under the general name of 'Pandora Server' which is a multi-threaded application (multi-processing) that executes in sub-processes (threads) each one being different to the instances or to the specialized servers of Pandora FMS.  
+
Under Pandora FMS, there are twelve different servers in total, specialized in and responsible for the various tasks necessary to make Pandora what it is today.  The servers are integrated into a single application under the general name of 'Pandora Server' which is a multi-threaded application (multi-processing) that executes in sub-processes (threads) each one being different to the instances or to the specialized servers of Pandora FMS. Hereon there is a description of the different Pandora FMS specialized servers.
  
The Pandora FMS Server is the element in charge of performing the pertinent checks. It verifies and changes them according to their results. It is also in charge of firing the alerts established by the Pandora server administrator to notify those monitoring the various target systems of events.
+
Pandora FMS servers are the elements in charge of performing the appropriate checks. They verify them and change their status according to their results. They are also responsible for triggering the alerts that are set to monitor the data status.
  
The Pandora FMS Data Server can work with high availability and / or load balancing. In very large architecture it is possible to employ several servers simultaneously to handle large volumes of information, and may be distributed by geographic or functional zones.  
+
The Pandora FMS Data Server can work with high availability and / or load balancing. In a very large architecture, it is possible to employ several servers simultaneously to handle large volumes of information, and may be distributed by geographic or functional zones.  
 
   
 
   
The Pandora FMS Server is always working and verifying if any monitored elements are experiencing any difficulties and can take appropriate action(s) if those are defined as alerts. When faced with a problem, it executes the response defined in the alert, such as sending an SMS, an email or activating the execution of a script.
+
Pandora FMS servers are always working and verifying if any monitored elements are experiencing any difficulties and can take appropriate action(s) if those are defined as alerts. When faced with a problem, it executes the response defined in the alert, such as sending an SMS, an email or activating the execution of a script.
  
There can be simultaneous servers, one of them being a main server and the rest of the servers being slaves. Although there is a master and a slave server relationship, they work simultaneously.  The difference between them is that when one of the servers goes off-line, the other Pandora server continues to function, providing transparent and redundant Pandora services.
+
There may be simultaneous servers, one of them being a main server and the rest of the servers being slaves. Even though there is a master and a slave server relationship, they work simultaneously.  The difference between the two is that when a server of the same type is down (e. g. a network server) the master server is in charge of processing all the data associated with the server that is down.
  
 
The server receiving the data file from the agent, or processing the information (if this is of the remote type) is the one which fires the associated alerts after the data is processed.
 
The server receiving the data file from the agent, or processing the information (if this is of the remote type) is the one which fires the associated alerts after the data is processed.
  
Pandora FMS automatically manages each server's status, load levels and other parameters. The user can monitor the state of each server by the server's status section of the web console.
+
Pandora FMS automatically manages each server's status, load levels and other parameters. The user can monitor the state of each server through the server's status section of the web console.
  
 
=== The Data Server ===
 
=== The Data Server ===
 
<br>
 
<br>
The Pandora FMS Data Server processes the information sent by the Software agents. The software agents send [http://en.wikipedia.org/wiki/XML '''XML'''] data to the server in different ways ([http://en.wikipedia.org/wiki/File_Transfer_Protocol '''FTP'''], [http://en.wikipedia.org/wiki/Secure_Shell '''SSH'''] or Tentacle) and the server periodically determines whether there are any data files awaiting processing or not.  This process utilizes a disk directory as a bonding medium for the elements to be processed.
+
It processes the information sent by software agents. The software agents collect information locally from the systems on which they are installed and build an information packet in XML format. These XML packets are sent to the server. They are received in a specific directory on the server, the server processes all the files coming to this input directory and stores the information in the database.
  
It's possible to install different data servers on different systems or on the same host (which will be different virtual servers). Several servers may work together in very large environments, making the best use of available hardware (e.g. multiple [http://en.wikipedia.org/wiki/Central_processing_unit '''CPU'''] environments).
+
Different data servers can be installed on different systems or on the same host by means of virtual servers. Several servers can work together for very large environments that need to make better use of hardware (e. g. in multi-CPU environments).
  
The Data Server (like the rest of the servers) accesses the Pandora FMS database, which communicates with the web server, and contains the processed data packets.  The server executes as a [http://en.wikipedia.org/wiki/Daemon_%28computing%29 '''daemon'''] or service and processes the gathered packets into its file system.  In spite of its simplicity and its modest use of resources, the data server is one of the critical elements of the system as it processes all of the agents' information, and generates system alerts and events according to that data. The data server solely works by the XML data from the software agents without conducting any remote verification.
+
Despite its simplicity and scarce use of resources, the data server is one of the critical elements of the system, since it processes all agent information and generates alerts and system events according to those data. The data server only works with the XML data from the software agents and does not perform any kind of remote verification.
  
 
=== The Network Server ===
 
=== The Network Server ===
 
<br>
 
<br>
The Network Server executes remote monitoring tasks through the network: [http://en.wikipedia.org/wiki/Internet_Control_Message_Protocol '''ICMP'''] tests (ping, latencies) [http://en.wikipedia.org/wiki/Transmission_Control_Protocol '''TCP'''] and [http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol '''SNMP'''] requests. If an agent is assigned to a server, it's always assigned to a web server, not to a data server. The reason for that is that it's very important that the engines executing the Web servers have 'network visibility' to be able to carry out the monitoring tasks assigned to them and making it possible for a server to connect to a defined network. For example, if we create a module for a ping verification on '192.168.1.1' and the agent or module is assigned to a server on '192.168.2.0/24' without network access to '192.168.1.0/24' we'll always get a 'down' reply, because it's unable to establish contact.
+
It executes remote monitoring tasks through the network: ICMP checks (Ping, latency time), TCP requests and SNMP requests. When an agent is assigned to a server, the network server that will run the checks for that agent is being specified, so it is very important that the machines running the network servers have "network visibility" in order to be able to execute the network monitoring tasks they have been assigned. That is, if you ping systems on a particular network, the network server can get to that network:
 +
 
 +
For example, if a module is created to perform a ping check at 192.168.1.1 and this agent/module is assigned to a server on a 192.168.2.0/24 network without 192.168.1.0/24 network access, it will always return DOWN as it cannot contact it.
  
<center><br>
+
<center>
 
[[File:Pandora_1.3_Network%26DataServer_Arch.png|center|650px]]
 
[[File:Pandora_1.3_Network%26DataServer_Arch.png|center|650px]]
</center><br>
+
</center>
  
=== The SNMP Server (also known as the SNMP Trap Console) ===
+
=== The SNMP Console ===
 
<br>
 
<br>
The SNMP Server utilizes the standard daemon of the traps collection system called 'snmptrapd'.  This daemon receives SNMP traps and the Pandora FMS Server processes and stores them in the database. While processing and analyzing them, it can also fire the designated alerts on the [http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol '''SNMP'''] Pandora FMS console.
+
The SNMP Server, also called SNMP trap console, uses the standard daemon of the trap collection system called 'snmptrapd'.  This daemon receives SNMP traps and the Pandora FMS Server processes and stores them in the database. It can also trigger the defined alerts linked to SNMP traps.
  
 
=== The WMI Server ===
 
=== The WMI Server ===
 
<br>
 
<br>
[http://en.wikipedia.org/wiki/Windows_Management_Instrumentation '''WMI'''] is a Microsoft standard to obtain information from a Windows-based operating system and Microsoft Windows environmental applications. Pandora FMS has a dedicated server to conduct native centralized WMI calls.  Thanks to that server it's possible to collect data from Windows systems remotely and without the need for an agent.
+
WMI is a Microsoft standard to obtain information from a Windows-based operating system and Microsoft Windows environmental applications. Pandora FMS has a server devoted to '''remotely''' monitor Windows systems through WMI protocol.
  
 
=== The Recon Server ===
 
=== The Recon Server ===
<br>
+
 
The Recognition Server (or 'Recon Server') is used to explore the network regularly and to detect new systems in operation. The Recon Server is also able to assign a monitoring template to recently detected systems and to apply the modules automatically by default, as defined by that template, so they can immediately be used to monitor the new system. By using the applications of the [http://en.wikipedia.org/wiki/Nmap '''NMAP'''], xprobe and [http://en.wikipedia.org/wiki/Traceroute '''traceroute'''] systems, it's also capable of identifying systems by their operating system, based on the opened ports and to establish the network's topology, guided by the systems it already knows.
+
Recon server was the former name for Discovery server, which is used to explore the network regularly and detect new systems in operation. Discovery Server is also able to assign a monitoring template to recently detected systems and to apply the modules automatically by default, as defined by that template, so they can immediately be used to monitor the new system. Discovery is also capable of identifying Operating Systems using system application such as nmap, xprobe and traceroute and establish network topology based in the systems it already knows.
 +
 
 +
The Discovery server is also used to launch scheduled tasks, and launch specific monitoring against virtual environments, cloud, databases or all those applications or environments that require exploring what exists before starting to monitor it.
  
 
=== The Plugin Server ===
 
=== The Plugin Server ===
 
<br>
 
<br>
The Plugin Server executes complex remote user tests conducted in any language, is integrated in the Pandora FMS interface and centrally managed. This allows the advanced user to define their own complex, self-developed tests and to integrate them into Pandora FMS.
+
The Plugin Server runs complex checks remotely using custom scripts. They may be developed in any language and integrated in Pandora FMS interface, being managed centrally. This allows an advanced user to define their own complex tests, developed by themselves, and integrate them in the application so that they can be used in a simple and centralized way from Pandora FMS.
  
 
=== The Prediction Server ===
 
=== The Prediction Server ===
 
<br>
 
<br>
The Prediction Server is a small component of artificial intelligence that implements a statistically predicted data set which is based on past data with a scope of up to 30 days in four temporal references. This allows us to predict the value of a data item in 10 to 15 minute intervals, making an educated assumption whether a particular data set presents an anomaly.  At present, it's based on its past historical performance. You basically won't have to construct a dynamic baseline by a weekly profileThis server also manages the service monitoring calculations (BPM) from Pandora FMS versions 5 and above.
+
The Prediction Server is a small component of artificial intelligence that implements a statistically predicted data set which is based on past data with a scope of up to 30 days in four temporal references. This allows us to predict the value of a data item in 10 to 15 minute intervals and know whether a particular data set presents an anomaly regarding its history. You will basically have to build a weekly dynamic baseline.   
  
=== The web server (Goliath) ===
+
This server also manages calculating service monitoring (BPM) from Pandora FMS version 5.0. onwards.
  
{{Tip|The Web Server (Goliath), the Export Server, the Inventory Server, the Event Correlation Server and the Enterprise Network Server are all features which are only available on the Enterprise Version of Pandora FMS.}}
+
=== The web server (Goliat) ===
  
The Web Server is used to carry out transactional web monitoring.  It conducts synthetic network testing, which means complete network testing, including: User identification process, parameters for data transfer, contents verification, menu navigation, etc. It's basically intended for verification tests and to obtain latencies (in seconds) of the complete network navigation experience (including the resources linked to the page, like images, full texts, etc).
+
{{Tip|The Web check Server (Goliat), the Export Server, the Inventory Server, the Event Correlation Server and the Enterprise Network Server are only available on the Enterprise Version of Pandora FMS.}}
 +
 
 +
The Web check Server is used to do load tests. It carries out full web tests, from the user identifying process, parameter forwarding by form or content check to menu navigation, etc. It is used for availability checks (it works, it does not) and to obtain latency times (in seconds) of navigation full experience, including resources linked to the website (images, full texts, etc.).
  
 
=== The Export Server ===
 
=== The Export Server ===
 +
''(Enterprise version only)''
  
The Export Server of Pandora FMS permits the transfer of data from the monitored dispositive of a Pandora FMS installation to another, making it possible to facilitate replication of any data. This is particularly useful for large deployments with several Pandora FMS installations, for example, if we want to have some critical information centralized in only one of them.
+
Pandora FMS Export Server allows to transfer data from the monitored device of a Pandora FMS installation to another, making it possible to replicate any data. This is particularly useful for large deployments with several Pandora FMS installations, for example, if we want to have some critical information centralized in only one of them.
  
 
=== The Inventory Server ===
 
=== The Inventory Server ===
 +
''(Enterprise version only)''
  
The Inventory Server obtains and visualizes inventory information on the systems: Installed software and patches, hardware, memory, hard disks, services running on the system, etc. It's able to obtain this information both locally and remotely from the software agents.
+
The Inventory Server obtains and displays inventory information on the systems: installed software, hardware, memory, hard disks, services running on the system, etc. It can obtain this information both locally and remotely from the software agents.
  
 
=== The Event Correlation Server ===
 
=== The Event Correlation Server ===
 +
''(Enterprise version only)''
  
The Event Correlation Server can be used to correlate events and generate alerts.  This is a special server that does not monitor, and like the others, can be specified in the configuration for its start up. Compared to the rest, this server does not make use of the threads configuration or high availability.
+
The Event Correlation Server can be used to correlate events and generate alerts.  This is a special server that does not monitor, and like the others, it can be specified in the configuration for its startup. Compared to the rest, this server does not make use of thread configuration or high availability.
  
 
=== The Enterprise Network Server for SNMP and ICMP ===
 
=== The Enterprise Network Server for SNMP and ICMP ===
 +
''(Enterprise version only)''
 +
 +
They are two additional servers using advanced strategies to process ICMP (ping) and SNMP (polling) checks in a way that offers superior performance to that of the open-source version, in exchange for some delicate criteria (especially SNMP) as they work with OIDs previously validated by the open server.
 +
 +
=== Satellite server ===
  
There are two additional servers using advanced strategies to process ICMP (ping) and SNMP (polling) requests in a way that offers superior performance to the open-source version, in exchange for some delicate requests (especially SNMP) as they work with previously validated OID’s by the opened server.
+
''(Enterprise version only)''
  
They use low level binary tools to access the TCP/IP system of the server by conducting block surveillance in a much more efficient way.
+
This component is installed separately in Pandora FMS main server. It allows to explore and detect new systems, remotely monitor with high speed ICMP and SNMP, and it executes remote plugins and allows forwarding data files from software agents to the main server, acting as agent proxy. It sends monitoring data as XMLs over a tentacle connection, so no database connection is required.
  
=== Satellite server ===
+
You can read more about this in the specific section about [[Pandora:Documentation_en:Satellite|Distributed monitoring with Satellite server]]
 +
 
 +
=== WUX server ===
  
The Satellite Server is a special server, a hybrid between an agent and a server, it does remote monitoring, it also detects new systems, like a recon server, and executes remote plugins. It does not require a database connection to the central database.  Instead, it sends XML data to the data server. You can read more about this later, in the specific section about [[Pandora:Documentation_en:Satellite|Distributed monitoring with Satellite server]]
+
It is a server, which combined with the Selenium Grid, allows complex WEB transactions to be carried out in a distributed manner. It differs from simple WEB checks (Goliat) in that these transactions are executed in a real browser, and their output is captured and processed to display it step by step, including error screenshots, as well as detailed statistics of all WEB requests.
  
== The Pandora FMS Network Console ==
+
== The Pandora FMS Console ==
 
<br>
 
<br>
This is the Pandora FMS User Interface. This administration and operations console allows the user to control the status of the agents, access statistical information, generate graphics and tables of data, as well as managing incidences with its integrated system serving different users with different privileges. It is also capable of producing reports and defining new modules, agents and alerts as well as creating other user profiles - and all of that is centralized from the interface.
+
This is the Pandora FMS User Interface. This administration and operation console allows users with different privileges to control the status of the agents, see statistical information, generate graphics and data tables, as well as managing incidences with its integrated system. It is also capable of producing reports and defining new modules, agents and alerts as well as creating other user profiles in a centralized way.
  
The Web console is programmed in [http://en.wikipedia.org/wiki/PHP '''PHP'''] and doesn't require the end user to install any additional software, not [http://en.wikipedia.org/wiki/Java_%28programming_language%29 '''Java'''] or [http://en.wikipedia.org/wiki/Activex '''ActiveX''']. Graphics are available, however, in [http://en.wikipedia.org/wiki/Adobe_Flash '''Flash''']. To see them in that format you'll need the Flash application on your browser, which can be accessed from any modern platform which supports [http://en.wikipedia.org/wiki/Html '''HTML'''] and [http://en.wikipedia.org/wiki/Css '''CSS''']. We recommend Firefox 2.x or IE 7.x. The user experience with older browsers like IE6 is very poor, because most of the features implemented in the Pandora FMS 3.0 console are unusable (and therefore hidden) if you're trying to use them.
+
The web console is programmed in PHP and does not require any additional software to be installed by the end user. It can be accessed from any modern platform that supports HTML and CSS. Firefox 2.x or Chrome is recommended. The user experience with browsers such as Internet Explorer 6 is very poor, and essential console features may be lost.
  
The network console is able to operate and supervise multiple servers. This means you may add as many web consoles as you want, either for load distribution or to facilitate access due to logistical problems (large networks, several groups with different users, geographical and administrative differences, etc.) It's a prerogative to have access to the data storage center where Pandora FMS compiles everything. That means: Access to the database and in case of the Enterprise Version, synchronized access to the agent's repository of configurations via [http://en.wikipedia.org/wiki/Network_File_System '''NFS'''].
+
In the meantime, the network console is able run on multiple servers. This means you may add as many web consoles as you want, either for load distribution or to make access due to logistical problems easier (large networks, several groups with different users, geographical and administrative differences, etc.). Its only requirement is to be able to access the data container where Pandora FMS stores everything: the database, and in the case of the Enterprise version, to access the agent configuration repository in a synchronized way (via NFS).
 +
 
 +
== Pandora FMS Database ==
 +
<br>
 +
Pandora FMS uses a MySQL database where it stores all the information received in real time, normalizing all the data coming from the different source sources. It conforms the most important and critical component of any Pandora FMS installation, containing not only the information and historical data, but all the configurations made over time. Currently, only MySQL/MariaDB/Percona are supported.
  
== The Pandora FMS Database ==
+
<center>
 +
[[image:Arch-schema.png|center|900px]]
 +
</center>
 +
<center>''The database is Pandora FMS core''</center>
 
<br>
 
<br>
Pandora FMS utilizes a [http://en.wikipedia.org/wiki/Mysql '''MySQL'''] database. Pandora FMS hosts an asynchronous database with all the received information, performing a temporary cohesion and normalizing all the data received from several sources. Each agent's data module generates a data entry for each packet, which means a real production system can have a scope of ten million units of data or atoms of information.
 
  
This data is automatically managed from Pandora FMS which carries out a periodical and automated maintenance service of the database, allowing Pandora FMS to do without any manually assisted database administration system, be it by an operator or an administrator. This is done by periodically purging data after a certain time period (90 days in a preset option) as well as compacting the data which exceeds a predetermined and configurable number of days (the default value is '30 days').
+
These data are managed automatically from Pandora FMS, carrying out periodical and automatic database maintenance , not being necessary any kind of database administration task nor manual process assisted by an operator or administrator. This is done by periodically purging the data on a due date.
  
== The Software Agents of Pandora FMS ==
+
== Pandora FMS software agents ==
 
<br>
 
<br>
When we refer to an agent in Pandora FMS, we're describing three essential components in the collection of data:
+
When we refer to an agent in Pandora FMS, it is important to distinguish between two concepts.
  
* The master-agent, Pandora FMS.
+
*The Agent, or agent in console, as container.
* The software agent (software application, Pandora FMS agent running in a machine).
+
*The Software Agent, as software that runs on a computer.
* The physical agent (hardware).
 
  
=== The Agent ===
+
=== The Agent (container) ===
  
The Pandora FMS agent itself is basically an organizational element, created with Pandora FMS Web Console and associated with a group of modules (seen as individual monitoring elements). This agent can also (optionally) have one or more IP addresses associated to it.
+
The Pandora FMS agent itself is basically an organizational element, created with Pandora FMS Web Console and associated to a group of modules (seen as individual monitoring elements). This agent can also (optionally) have one or more IP addresses associated to it.
  
The agent can have associated remote modules, which would have been obtained by web servers, WMIs, plug ins, etc.
+
The agent can have associated remote modules, which would have been obtained by network servers, WMI, plugins, etc.
  
 
* Verification of whether the engine is connected or on line (PING).
 
* Verification of whether the engine is connected or on line (PING).
Line 128: Line 150:
 
* Verification of whether a network entity, hosted on a specific port of the hardware, is responding correctly.  
 
* Verification of whether a network entity, hosted on a specific port of the hardware, is responding correctly.  
 
* Verification of whether a network entity, hosted on a specific point of the hardware, has the desired content.
 
* Verification of whether a network entity, hosted on a specific point of the hardware, has the desired content.
* Hardware verification(s) by [http://en.wikipedia.org/wiki/Snmp SNMP] (ascertaining the MIB).
+
* Hardware verification(s) by SNMP (ascertaining the MIB).
* Latency time verification between the node and the Pandora FMS servers.
+
* Latency time verification between the node and Pandora FMS servers.
 +
 
 +
The agent can also have local type modules associated to it. Local modules are those defined in the software agent's configuration, which are also required to be defined within the network console's agent. When a data packet arrives for the first time from a software agent, the new agent will be created automatically, with its group of modules executed locally in the web console.  
  
The agent can also have local modules associated with it. Local modules are those defined in the software agent's configuration which are also required to be defined within the network console's agent. If the agent is in 'auto-learning mode' (the default setting), these local modules are created automatically in the web console when a packet of data arrives from the agent for the first time. Therefore, an 'Agent' may contain modules of both remote and local types. The remote-type modules are executed by the servers obtaining information remotely (prediction included) and the local modules are obtained by the Data Server.
+
Therefore, an 'Agent' may contain modules of both remote and local types. Remote-type modules are executed by the servers obtaining information remotely (network server, recon server...) and local modules are obtained and processed by the Data Server.
  
 
=== The Software Agent ===
 
=== The Software Agent ===
  
A software agent installed on a remote node is completely different from the one on the Pandora server or within Pandora's network console. The software agent gathers information local to the node from the engine where it's executed, gathering information on the node by commands.
+
The software agents are installed on computers to be monitored locally, retrieving the information from the computer itself. They are mainly used in servers for monitoring machine resources (CPU, RAM, disks...) and installed applications (MySQL, Apache, JBoss...). Generally speaking, server and equipment monitoring will be carried out with software agents, while network equipment monitoring will be done remotely without any software installation.
  
Pandora FMS Agents can be practically developed in any language as long as it meets the conditions of the data exchange API from the Pandora FMS Data Server (defined by the [http://en.wikipedia.org/wiki/Xmlhttp://en.wikipedia.org/wiki/Xml '''XML'''] data exchange). The window agents operate in a free environment for [http://en.wikipedia.org/wiki/C%2B%2B '''C++'''] and employ the same interface and modularity as the [http://en.wikipedia.org/wiki/Unix '''UNIX'''] agents, however coming with several characteristics of their own.
+
<br>
 
 
<br><br>
 
<center>''Illustration: Collection of Data in Pandora FMS''</center><br>
 
 
<center>
 
<center>
 
[[image:RecoleDatospandora.png|center|750px]]
 
[[image:RecoleDatospandora.png|center|750px]]
 +
''Illustration: Collection of Data in Pandora FMS''
 
</center>
 
</center>
  
These scripts are built from sub modules and each one of them collects a portion of information.
+
Each software agent performs several checks, called modules that correspond to a specific data, such as CPU usage. All the information of the checks carried out is presented in a single data file in XML format that is sent to Pandora FMS server.
 
+
 
Each agent collects several portions of information. These are compiled into one packet and stored in a single file named 'data package'.
+
The process of copying the data packet from the agent to the server is done regularly (Synchronous) every so often. This '''interval''' is defined in the software agent, which is the one that starts communications with the server.
 +
 
 +
The default interval is 300 seconds. Values lower than 100 (seconds) are not generally recommended, as they can negatively affect the host-system's performance, in addition to overloading the database and the Pandora FMS server itself.
  
The copy process of the data packet from agent to server is synchronously executed on a regular basis. That is, within regular intervals -defined by the agent- which can be modified in order not to clutter the data base with superfluous information, overload the web server, or to become detrimental to the system's performance.
+
It is important to remember that '''Pandora FMS is not a real time system''', but a general monitoring system for systems and applications in environments where real time is not a critical factor. It may be adapted to operate in environments with response times of 3 to 5 seconds.  
 
 
The interval can be defined up to 300 seconds, which is the decimal equivalent of 5 minutes. Lesser values to 100 (seconds) are not recommended, as they can negatively affect the host-system's performance. Such a low polling time can overload the database and the central processing system.
 
  
It's important to remember that Pandora FMS is not a 'real time system', but a general monitoring system for systems and applications in environments where [http://en.wikipedia.org/wiki/Real-time_computing '''real time'''] is not a critical factor. It may be adapted to operate in environments with response times of 3 to 5 seconds. <br>
+
<br>
<center>''Illustration: Logical diagram of an agent and a physical agent.''</center><br>
+
<center>
 
[[File:PandoraAgent_logical_schema.png|center|650px]]
 
[[File:PandoraAgent_logical_schema.png|center|650px]]
<br><br>
+
''Illustration: Logical diagram of an agent and a physical agent.''
 +
</center>
  
Packet transfers are conducted by the Tentacle Protocol, but they can also be transferred using [http://en.wikipedia.org/wiki/Secure_Shell '''SSH'''] or [http://en.wikipedia.org/wiki/Ftp '''FTP'''].  
+
XML packet transfers are conducted by the Tentacle Protocol, but they can also be transferred using [http://en.wikipedia.org/wiki/Secure_Shell '''SSH'''] or [http://en.wikipedia.org/wiki/Ftp '''FTP'''].  
  
With either [http://en.wikipedia.org/wiki/Secure_Shell '''SSH'''] or Tentacle, the process can be made secure, given that passwords don't travel through the network nor do they contain unencrypted confidential data, assuring the confidentiality, integrity and authentication of the connections between agent and server. The code-generating process and also the Tentacle protocol are detailed in the documentation on the installation and configuration of the agents and servers to be able to carry out the [http://en.wikipedia.org/wiki/Secure_copy '''SCP'''] ([http://en.wikipedia.org/wiki/Secure_Shell '''SSH''']) transfer automatically.
+
With either SSH or Tentacle, the process can be made secure, since passwords do not go through the network nor do they contain unencrypted confidential data, assuring the confidentiality, integrity and authentication of the connections between agent and server. The key-generating process for automatic transfer through SCP (SSH) and through Tentacle protocol is further explained in the documentation about agent and server installation and setup.
  
The transfer may also be conducted by [http://en.wikipedia.org/wiki/Ftp '''FTP'''] or any other file transfer protocols. However, we chose the Tentacle Protocol due to its security, user friendliness and the numerous options this system provides.  
+
The transfer may also be done through FTP or any other file transfer protocols. However, Tentacle protocol has been chosen due to its security, user friendliness and its multiple options.  
  
Please check the documentation annex regarding configuration of the transfer protocols.
+
Check the documentation annex regarding configuration of other transfer protocols.
  
Pandora FMS agents are designed to be executed from the agent they collect the data from, although the agents can also collect information stored in accessible engines from the host they are installed on. These are known as 'satellite agents'.
+
Pandora FMS agents are designed to be executed from the system they collect the data from, although the agents can also collect information stored in accessible engines from the host they are installed on by executing network commands.
  
It is also feasible to configure a node in such a way to be able to bear several Pandora FMS agents simultaneously. This predicament is quite rare. It occurs if we have e.g. a software and a satellite agent. The standard software agent monitors the engine where it's executed while the installed satellite agents (there can be several) are monitoring remote systems by [http://en.wikipedia.org/wiki/Telnet '''Telnet'''], [http://en.wikipedia.org/wiki/Snmp '''SNMP'''] or other proprietary protocols.
+
=== The XML Data File ===
  
=== The XML Data File ===
+
This data file contains an XML structure and its name is formed by combining the hostname of the host where the agent is located, a different serial number for each data packet and the ".data" extension indicating that it is a data packet.
  
The data file has the following syntax:
 
 
<pre>
 
<pre>
 
<host number>.<serial number>.data  
 
<host number>.<serial number>.data  
 
</pre>
 
</pre>
  
This data file has an [http://en.wikipedia.org/wiki/Xml '''XML'''] structure. It takes its name from a combination of the host names where the agent is located and a serial number, which differs in each packet, and the extension ".data" which indicates that this is a data packet.
+
<br>
 
+
<center>''Illustration: Logical structure of a software agent's modules.''
<br><br>
 
<center>''Illustration: Logical structure of a software agent's modules.''</center><br>
 
<center>
 
 
[[image:Xml_transfer.png|center|550px]]
 
[[image:Xml_transfer.png|center|550px]]
 
</center>
 
</center>
 +
 +
The ".data" file is the file that holds the data.  The verification file with the ".checksum" extension contains an MD5 hash of the data file. This allows to perform a final verification to ensure the data have not been altered in any way before being processed.
  
 
<pre>
 
<pre>
Line 189: Line 210:
 
</pre>
 
</pre>
  
The ".data" file is the file which holds the data.  The verification file with the ".checksum" extension contains an MD5 hash of the data file. These allow us to perform a final verification to ensure the data hasn't been altered in any way before being processed.
+
The XML data file contains all the information collected by the Agent during its execution. This data packet has a compact, flexible and light design that allows any user to use Pandora FMS agents or his own developments to generate information and for it be processed in Pandora FMS. The data file is an XML similar to the following:
 
 
The [http://en.wikipedia.org/wiki/Xml '''XML'''] data file the agent generates is at the heart of Pandora FMS. It contains a data packet along with the information gathered by the agent. This data packet has a compact design: Light and flexible that allows any user to use Pandora FMS agents or to generate information to be processed in Pandora FMS by other methods. The data file is an [http://en.wikipedia.org/wiki/Xml '''XML'''] file similar to the following:
 
  
 
<pre>
 
<pre>
Line 218: Line 237:
 
</pre>
 
</pre>
  
=== The Physical Agent ===
+
== Topologies, diagrams and monitoring models ==
**
 
Pandora FMS has a physical agent mounted on an ASUS and an Arduino automaton. This tandem, along with the connected sensors presently facilitates the monitoring of the following environmental features:
 
 
 
* Humidity
 
* Temperature
 
* Ambient lighting
 
* Presence
 
 
 
Because of the electrical nature of the sensors they are easily calibrated. Their values are also able to be processed by Pandora FMS without any difficulty.
 
The fact that the sensor is a wireless router opens up an entire world of possibilities to this type of sensors already present in the CPDs of some companies.
 
 
 
== Typologies, Schemes and Monitoring Models ==
 
  
There are different models to address the monitoring process, both local and remote. We enumerate the following common examples for different topologies in order to familiarize the reader with the possible problems and the solutions Pandora has to offer. Each of the solutions is described in successive chapters.
+
There are different models to address the monitoring process, both local and remote. The following common examples of different topologies may help the reader to become familiar with the possible problems and the solutions Pandora FMS offers. Each of these solutions is described in the chapters below.
  
 
=== Accessible Networks ===
 
=== Accessible Networks ===
  
This is the norm in small, simple networks but also in the very centralized and well organized ones. This one is the easiest model to implement.
+
This is quite usual in small, simple networks but also in very centralized and well organized ones. This one is the easiest model to implement.
  
* Network access for '''centralized remote monitoring'''. It implies that we can access every node from Pandora’s server to probe remotely.  
+
* Network access for '''centralized remote monitoring'''. It implies that all machines can be accessed from Pandora FMS to probe them remotely.  
  
* Network access for '''agent based monitoring'''. In this network, we're able to reach Pandora’s server from the agents installed on the monitoring engine.
+
* Network access for '''agent based monitoring'''. In this network, Pandora FMS server can be reached from the software agents installed on monitored machines.
  
 
=== Limited-Access Networks ===
 
=== Limited-Access Networks ===
  
* ''' Remote Network:''' This is an unreachable network for remote testing by Pandora FMS. We are using a software agent as a remote gatherer to test other systems on that end. We call these 'operating modes': A 'satellite agent mode' (when all testing is carried out within the same agent) and a 'broker agent mode' (if it impersonates several agents but all tests are actually carried out in the same physical engine).
+
* ''' Network not reachable through remote Pandora FMS checks:''' Here you have several options. First, using a software agent to execute remote checks to other systems (using the broker agent mode) or by using the Satellite server, which is capable of executing remote checks and has a series of advanced features.
  
 
<center>
 
<center>
<br><br>''Deployment model for remote and inaccessible networks in Broker Mode''<br>
+
[[File:Modo-broker.png|center|790px]]
[[File:Broker_example_no_access.png|center|790px]]
+
<br>''Deployment model for remote and inaccessible networks in Broker Mode''
 
</center>
 
</center>
<br><br>
+
<br>
 +
 
 +
* '''Software agents without access to a Pandora FMS Server.''' In this case, the proxy characteristics of the software agents are used, allowing those agents without direct access to use another agent with direct access for the connection, forwarding the XML files of all agents apart of their own. The ''Satellite Server'' can also work as a proxy agent.
  
* '''Software agents without access to a Pandora FMS Server.''' In this case, we're going to use the proxy characteristics of the software agents, allowing those agents without direct access to the server to use another agent with direct access to connect and facilitate the transactions.
 
 
<center>
 
<center>
<br><br>''Deployment model for remote networks by using the Proxy Agent Mode''<br>
+
[[File:Proxy-mode.png|center|790px]]
[[File:Proxy_agent_schema.png|center|790px]]
+
<br>''Deployment model for remote networks by using the Proxy Agent Mode''
 
</center>
 
</center>
<br><br>
+
<br>
 +
 
 +
* The need to monitor '''different networks for remote monitoring with the server:''' In this case, make use of the'' Satellite server'', or connect several different Pandora FMS servers to the same database, one server will run a set of checks, and another server another different set. The way to perform the deployment will be different, but in both cases, each component will be fully responsible for your network monitoring and management will be centralized from the Console.
 +
 
  
* The need to conduct '''remote server monitoring for different networks:''' In this situation we're going to mount several different servers of Pandora FMS. Connected to the same database, one server is going to execute a battery of predefined tests and a different one by another. Both servers operate within the same environment which are being managed from the console simultaneously.
+
<center>
 +
[[File:Esquema-satellite.png|center|790px]]
 +
<br>''Remote network deployment model using the Satellite Server''
 +
</center>
 +
<br>
  
 
=== Special Organizational Characteristics ===
 
=== Special Organizational Characteristics ===
  
* The need '''to monitor several headquarters''' by monitoring equipment and different configurations. In this case, we're using an export to duplicate a part of the monitoring in an independent environment segregated from Pandora FMS.
+
* The need to have '''several sites monitored''', with different monitoring equipment and configurations. In this case, the Export Server to duplicate part of the monitoring in a segregated Pandora FMS environment, which is independent.
 +
 
 
<center>
 
<center>
<br><br>'' Hierarchical export model along with an Export Server''<br>
+
[[File:Export-server.png|center|650px]]
[[File:ES1.png|center|650px]]
+
<br>''Hierarchical export model along with an Export Server''<br>
 
</center>
 
</center>
<br><br>
+
<br>
  
* '''Duality of Reporting:''' We can configure additional agents to support two different Pandora FMS Servers, although only one will be able to manage it.
+
* '''Reporting duality:''' Additionally, you may configure agents to report to two different Pandora FMS servers, although they can only be managed by one of them.
  
* '''Fragmented Management:''' It's pretty useful if you're required to '''delegate the administration of part of the equipment''' to different personnel with different access levels. This is more of a management issue rather than an architectural problem. It can be resolved by the assigned permissions within the management policies.
+
* '''Fragmented management:''' It is pretty useful if you are required to '''delegate the administration of part of the equipment''' to different personnel with different access levels. This is more of a management issue rather than an architectural problem. It can be solved by the assigned permissions on policies.
  
 
=== Large Environments ===
 
=== Large Environments ===
  
* A '''Large-Volume Network''', consisting of thousands of network testing processes which we distribute within different 'remote monitoring probes'. Given their large numbers (over 50,000) we can't centralize them into a single server. To facilitate monitoring we're going to use different servers in Broker Mode which distributes the monitoring by its own method.  
+
* '''Large-Volume Network''', consisting of thousands of network testing processes which must be distributed in different 'remote monitoring probes'. Given their large numbers (over 50,000), they cannot be centralized into a single server. To that end, use servers in broker mode that distribute the remote check load.  
  
 +
<br>
 
<center>
 
<center>
<br><br>''Distribution of remote testing model with agents in broker mode''<br>
 
 
[[File:Broker_scalation_example.png|center|770px]]
 
[[File:Broker_scalation_example.png|center|770px]]
 +
''Remote testing distribution model with agents in broker mode.''
 
</center>
 
</center>
<br><br>
 
  
In case of a primary hardware failure, we're required to setup a [http://en.wikipedia.org/wiki/High_availability '''HA Server'''] for security reasons.
+
*The  need to mount '''a server on HA''' for security reasons, in case primary hardware fails. You will see how to mount two servers, one "passive", waiting for the active to stop responding and start up. There are different ways to do this.
We're going to learn how to mount two servers: One ‘passive’, waiting in standby-mode for the active one to stop responding so it can start working. There are several ways to set them up in this way.
+
 
 +
* The need to '''monitor a large volume of systems and manage them in a centralized way''' (more than 2500 agents). In order to do so, different Pandora FMS Servers are configured to be coordinated by the system called 'Metaconsole'. They can be linearly scaled in this way.
  
* The need to '''monitor a large volume of systems and manage them in a centralized way''' (more than 2500 agents). In order to do so, we're configuring different Pandora FMS Servers, coordinated by the system we call 'metaconsole'. They can be linearly scaled in this way.
+
<br>
 
<center>
 
<center>
<br><br>''The metaconsole model''<br>
 
 
[[File:Pandora_metaconsole_overview2.png|center|750px]]
 
[[File:Pandora_metaconsole_overview2.png|center|750px]]
 +
''The metaconsole model''
 
</center>
 
</center>
<br><br>
 
 
  
 
[[Pandora:Documentation_en|Go back to Pandora FMS Documentation Index]]
 
[[Pandora:Documentation_en|Go back to Pandora FMS Documentation Index]]
  
 
[[Category:Pandora FMS]]
 
[[Category:Pandora FMS]]

Latest revision as of 09:50, 16 October 2019

Go back to Pandora FMS documentation index

Template wip.png

We are working on the translation of the Pandora FMS documentation. Sorry for any inconvenience.

 


1 Pandora FMS Architecture

This chapter provides a general description of Pandora FMS and its components, the way they relate to each other and how to use the Pandora FMS architecture to meet different challenges regarding the topology of its infrastructure.

Pandora FMS can be modular and decentralized or simple and monolithic. The most vital component is the MySQL database, where all the information is stored. Each component of Pandora FMS can be replicated and works in a full HA environment, be it passive, active or in a clustered environment (Active/Active with load balancing).

Arquitectu pando.png
Diagram of the global architecture of Pandora FMS


Pandora FMS consists of several elements, among them, the ones that are in charge of collecting and processing the data, which are the servers. The servers, with the information generated by themselves or by the agents, enter the data into the database. The console is the part in charge of displaying the data present in the database and interacting with the end user. Software Agents are applications that run on monitored systems, and collect the information to send it to Pandora FMS servers.

1.1 Pandora FMS Servers


Under Pandora FMS, there are twelve different servers in total, specialized in and responsible for the various tasks necessary to make Pandora what it is today. The servers are integrated into a single application under the general name of 'Pandora Server' which is a multi-threaded application (multi-processing) that executes in sub-processes (threads) each one being different to the instances or to the specialized servers of Pandora FMS. Hereon there is a description of the different Pandora FMS specialized servers.

Pandora FMS servers are the elements in charge of performing the appropriate checks. They verify them and change their status according to their results. They are also responsible for triggering the alerts that are set to monitor the data status.

The Pandora FMS Data Server can work with high availability and / or load balancing. In a very large architecture, it is possible to employ several servers simultaneously to handle large volumes of information, and may be distributed by geographic or functional zones.

Pandora FMS servers are always working and verifying if any monitored elements are experiencing any difficulties and can take appropriate action(s) if those are defined as alerts. When faced with a problem, it executes the response defined in the alert, such as sending an SMS, an email or activating the execution of a script.

There may be simultaneous servers, one of them being a main server and the rest of the servers being slaves. Even though there is a master and a slave server relationship, they work simultaneously. The difference between the two is that when a server of the same type is down (e. g. a network server) the master server is in charge of processing all the data associated with the server that is down.

The server receiving the data file from the agent, or processing the information (if this is of the remote type) is the one which fires the associated alerts after the data is processed.

Pandora FMS automatically manages each server's status, load levels and other parameters. The user can monitor the state of each server through the server's status section of the web console.

1.1.1 The Data Server


It processes the information sent by software agents. The software agents collect information locally from the systems on which they are installed and build an information packet in XML format. These XML packets are sent to the server. They are received in a specific directory on the server, the server processes all the files coming to this input directory and stores the information in the database.

Different data servers can be installed on different systems or on the same host by means of virtual servers. Several servers can work together for very large environments that need to make better use of hardware (e. g. in multi-CPU environments).

Despite its simplicity and scarce use of resources, the data server is one of the critical elements of the system, since it processes all agent information and generates alerts and system events according to those data. The data server only works with the XML data from the software agents and does not perform any kind of remote verification.

1.1.2 The Network Server


It executes remote monitoring tasks through the network: ICMP checks (Ping, latency time), TCP requests and SNMP requests. When an agent is assigned to a server, the network server that will run the checks for that agent is being specified, so it is very important that the machines running the network servers have "network visibility" in order to be able to execute the network monitoring tasks they have been assigned. That is, if you ping systems on a particular network, the network server can get to that network:

For example, if a module is created to perform a ping check at 192.168.1.1 and this agent/module is assigned to a server on a 192.168.2.0/24 network without 192.168.1.0/24 network access, it will always return DOWN as it cannot contact it.

Pandora 1.3 Network&DataServer Arch.png

1.1.3 The SNMP Console


The SNMP Server, also called SNMP trap console, uses the standard daemon of the trap collection system called 'snmptrapd'. This daemon receives SNMP traps and the Pandora FMS Server processes and stores them in the database. It can also trigger the defined alerts linked to SNMP traps.

1.1.4 The WMI Server


WMI is a Microsoft standard to obtain information from a Windows-based operating system and Microsoft Windows environmental applications. Pandora FMS has a server devoted to remotely monitor Windows systems through WMI protocol.

1.1.5 The Recon Server

Recon server was the former name for Discovery server, which is used to explore the network regularly and detect new systems in operation. Discovery Server is also able to assign a monitoring template to recently detected systems and to apply the modules automatically by default, as defined by that template, so they can immediately be used to monitor the new system. Discovery is also capable of identifying Operating Systems using system application such as nmap, xprobe and traceroute and establish network topology based in the systems it already knows.

The Discovery server is also used to launch scheduled tasks, and launch specific monitoring against virtual environments, cloud, databases or all those applications or environments that require exploring what exists before starting to monitor it.

1.1.6 The Plugin Server


The Plugin Server runs complex checks remotely using custom scripts. They may be developed in any language and integrated in Pandora FMS interface, being managed centrally. This allows an advanced user to define their own complex tests, developed by themselves, and integrate them in the application so that they can be used in a simple and centralized way from Pandora FMS.

1.1.7 The Prediction Server


The Prediction Server is a small component of artificial intelligence that implements a statistically predicted data set which is based on past data with a scope of up to 30 days in four temporal references. This allows us to predict the value of a data item in 10 to 15 minute intervals and know whether a particular data set presents an anomaly regarding its history. You will basically have to build a weekly dynamic baseline.

This server also manages calculating service monitoring (BPM) from Pandora FMS version 5.0. onwards.

1.1.8 The web server (Goliat)

Info.png

The Web check Server (Goliat), the Export Server, the Inventory Server, the Event Correlation Server and the Enterprise Network Server are only available on the Enterprise Version of Pandora FMS.

 


The Web check Server is used to do load tests. It carries out full web tests, from the user identifying process, parameter forwarding by form or content check to menu navigation, etc. It is used for availability checks (it works, it does not) and to obtain latency times (in seconds) of navigation full experience, including resources linked to the website (images, full texts, etc.).

1.1.9 The Export Server

(Enterprise version only)

Pandora FMS Export Server allows to transfer data from the monitored device of a Pandora FMS installation to another, making it possible to replicate any data. This is particularly useful for large deployments with several Pandora FMS installations, for example, if we want to have some critical information centralized in only one of them.

1.1.10 The Inventory Server

(Enterprise version only)

The Inventory Server obtains and displays inventory information on the systems: installed software, hardware, memory, hard disks, services running on the system, etc. It can obtain this information both locally and remotely from the software agents.

1.1.11 The Event Correlation Server

(Enterprise version only)

The Event Correlation Server can be used to correlate events and generate alerts. This is a special server that does not monitor, and like the others, it can be specified in the configuration for its startup. Compared to the rest, this server does not make use of thread configuration or high availability.

1.1.12 The Enterprise Network Server for SNMP and ICMP

(Enterprise version only)

They are two additional servers using advanced strategies to process ICMP (ping) and SNMP (polling) checks in a way that offers superior performance to that of the open-source version, in exchange for some delicate criteria (especially SNMP) as they work with OIDs previously validated by the open server.

1.1.13 Satellite server

(Enterprise version only)

This component is installed separately in Pandora FMS main server. It allows to explore and detect new systems, remotely monitor with high speed ICMP and SNMP, and it executes remote plugins and allows forwarding data files from software agents to the main server, acting as agent proxy. It sends monitoring data as XMLs over a tentacle connection, so no database connection is required.

You can read more about this in the specific section about Distributed monitoring with Satellite server

1.1.14 WUX server

It is a server, which combined with the Selenium Grid, allows complex WEB transactions to be carried out in a distributed manner. It differs from simple WEB checks (Goliat) in that these transactions are executed in a real browser, and their output is captured and processed to display it step by step, including error screenshots, as well as detailed statistics of all WEB requests.

1.2 The Pandora FMS Console


This is the Pandora FMS User Interface. This administration and operation console allows users with different privileges to control the status of the agents, see statistical information, generate graphics and data tables, as well as managing incidences with its integrated system. It is also capable of producing reports and defining new modules, agents and alerts as well as creating other user profiles in a centralized way.

The web console is programmed in PHP and does not require any additional software to be installed by the end user. It can be accessed from any modern platform that supports HTML and CSS. Firefox 2.x or Chrome is recommended. The user experience with browsers such as Internet Explorer 6 is very poor, and essential console features may be lost.

In the meantime, the network console is able run on multiple servers. This means you may add as many web consoles as you want, either for load distribution or to make access due to logistical problems easier (large networks, several groups with different users, geographical and administrative differences, etc.). Its only requirement is to be able to access the data container where Pandora FMS stores everything: the database, and in the case of the Enterprise version, to access the agent configuration repository in a synchronized way (via NFS).

1.3 Pandora FMS Database


Pandora FMS uses a MySQL database where it stores all the information received in real time, normalizing all the data coming from the different source sources. It conforms the most important and critical component of any Pandora FMS installation, containing not only the information and historical data, but all the configurations made over time. Currently, only MySQL/MariaDB/Percona are supported.

Arch-schema.png
The database is Pandora FMS core


These data are managed automatically from Pandora FMS, carrying out periodical and automatic database maintenance , not being necessary any kind of database administration task nor manual process assisted by an operator or administrator. This is done by periodically purging the data on a due date.

1.4 Pandora FMS software agents


When we refer to an agent in Pandora FMS, it is important to distinguish between two concepts.

  • The Agent, or agent in console, as container.
  • The Software Agent, as software that runs on a computer.

1.4.1 The Agent (container)

The Pandora FMS agent itself is basically an organizational element, created with Pandora FMS Web Console and associated to a group of modules (seen as individual monitoring elements). This agent can also (optionally) have one or more IP addresses associated to it.

The agent can have associated remote modules, which would have been obtained by network servers, WMI, plugins, etc.

  • Verification of whether the engine is connected or on line (PING).
  • Verification of whether a given port is opened or closed.
  • Verification of whether a network entity, hosted on a specific port of the hardware, is responding correctly.
  • Verification of whether a network entity, hosted on a specific point of the hardware, has the desired content.
  • Hardware verification(s) by SNMP (ascertaining the MIB).
  • Latency time verification between the node and Pandora FMS servers.

The agent can also have local type modules associated to it. Local modules are those defined in the software agent's configuration, which are also required to be defined within the network console's agent. When a data packet arrives for the first time from a software agent, the new agent will be created automatically, with its group of modules executed locally in the web console.

Therefore, an 'Agent' may contain modules of both remote and local types. Remote-type modules are executed by the servers obtaining information remotely (network server, recon server...) and local modules are obtained and processed by the Data Server.

1.4.2 The Software Agent

The software agents are installed on computers to be monitored locally, retrieving the information from the computer itself. They are mainly used in servers for monitoring machine resources (CPU, RAM, disks...) and installed applications (MySQL, Apache, JBoss...). Generally speaking, server and equipment monitoring will be carried out with software agents, while network equipment monitoring will be done remotely without any software installation.


RecoleDatospandora.png

Illustration: Collection of Data in Pandora FMS

Each software agent performs several checks, called modules that correspond to a specific data, such as CPU usage. All the information of the checks carried out is presented in a single data file in XML format that is sent to Pandora FMS server.

The process of copying the data packet from the agent to the server is done regularly (Synchronous) every so often. This interval is defined in the software agent, which is the one that starts communications with the server.

The default interval is 300 seconds. Values lower than 100 (seconds) are not generally recommended, as they can negatively affect the host-system's performance, in addition to overloading the database and the Pandora FMS server itself.

It is important to remember that Pandora FMS is not a real time system, but a general monitoring system for systems and applications in environments where real time is not a critical factor. It may be adapted to operate in environments with response times of 3 to 5 seconds.


PandoraAgent logical schema.png

Illustration: Logical diagram of an agent and a physical agent.

XML packet transfers are conducted by the Tentacle Protocol, but they can also be transferred using SSH or FTP.

With either SSH or Tentacle, the process can be made secure, since passwords do not go through the network nor do they contain unencrypted confidential data, assuring the confidentiality, integrity and authentication of the connections between agent and server. The key-generating process for automatic transfer through SCP (SSH) and through Tentacle protocol is further explained in the documentation about agent and server installation and setup.

The transfer may also be done through FTP or any other file transfer protocols. However, Tentacle protocol has been chosen due to its security, user friendliness and its multiple options.

Check the documentation annex regarding configuration of other transfer protocols.

Pandora FMS agents are designed to be executed from the system they collect the data from, although the agents can also collect information stored in accessible engines from the host they are installed on by executing network commands.

1.4.3 The XML Data File

This data file contains an XML structure and its name is formed by combining the hostname of the host where the agent is located, a different serial number for each data packet and the ".data" extension indicating that it is a data packet.

<host number>.<serial number>.data 


Illustration: Logical structure of a software agent's modules.
Xml transfer.png

The ".data" file is the file that holds the data. The verification file with the ".checksum" extension contains an MD5 hash of the data file. This allows to perform a final verification to ensure the data have not been altered in any way before being processed.

<host number>.<serial number>.checksum

The XML data file contains all the information collected by the Agent during its execution. This data packet has a compact, flexible and light design that allows any user to use Pandora FMS agents or his own developments to generate information and for it be processed in Pandora FMS. The data file is an XML similar to the following:

 <agent data os_name=”SunOS” os_version=”5.8” timestamp=”300” agent_name=”pdges01” version=”1.0”>
 <module>
 <name>FTP Daemon</name>
 <type>generic_proc</type>
 <data>0</data>
 </module>
 <module>
 <name>DiskFree</name>
 <type>generic_data</type>
 <data>5200000</data>
 </module>
 <module>
 <name>UsersConnected</name>
 <type>generic_data_inc</type>
 <data>119</data>
 </module>
 <module>
 <name>LastLogin</name>
 <type>generic_data_string</type>
 <data>slerena</data>
 </module>
 </agent_data>

1.5 Topologies, diagrams and monitoring models

There are different models to address the monitoring process, both local and remote. The following common examples of different topologies may help the reader to become familiar with the possible problems and the solutions Pandora FMS offers. Each of these solutions is described in the chapters below.

1.5.1 Accessible Networks

This is quite usual in small, simple networks but also in very centralized and well organized ones. This one is the easiest model to implement.

  • Network access for centralized remote monitoring. It implies that all machines can be accessed from Pandora FMS to probe them remotely.
  • Network access for agent based monitoring. In this network, Pandora FMS server can be reached from the software agents installed on monitored machines.

1.5.2 Limited-Access Networks

  • Network not reachable through remote Pandora FMS checks: Here you have several options. First, using a software agent to execute remote checks to other systems (using the broker agent mode) or by using the Satellite server, which is capable of executing remote checks and has a series of advanced features.
Modo-broker.png


Deployment model for remote and inaccessible networks in Broker Mode


  • Software agents without access to a Pandora FMS Server. In this case, the proxy characteristics of the software agents are used, allowing those agents without direct access to use another agent with direct access for the connection, forwarding the XML files of all agents apart of their own. The Satellite Server can also work as a proxy agent.
Proxy-mode.png


Deployment model for remote networks by using the Proxy Agent Mode


  • The need to monitor different networks for remote monitoring with the server: In this case, make use of the Satellite server, or connect several different Pandora FMS servers to the same database, one server will run a set of checks, and another server another different set. The way to perform the deployment will be different, but in both cases, each component will be fully responsible for your network monitoring and management will be centralized from the Console.


Esquema-satellite.png


Remote network deployment model using the Satellite Server


1.5.3 Special Organizational Characteristics

  • The need to have several sites monitored, with different monitoring equipment and configurations. In this case, the Export Server to duplicate part of the monitoring in a segregated Pandora FMS environment, which is independent.
Export-server.png


Hierarchical export model along with an Export Server


  • Reporting duality: Additionally, you may configure agents to report to two different Pandora FMS servers, although they can only be managed by one of them.
  • Fragmented management: It is pretty useful if you are required to delegate the administration of part of the equipment to different personnel with different access levels. This is more of a management issue rather than an architectural problem. It can be solved by the assigned permissions on policies.

1.5.4 Large Environments

  • Large-Volume Network, consisting of thousands of network testing processes which must be distributed in different 'remote monitoring probes'. Given their large numbers (over 50,000), they cannot be centralized into a single server. To that end, use servers in broker mode that distribute the remote check load.


Broker scalation example.png

Remote testing distribution model with agents in broker mode.

  • The need to mount a server on HA for security reasons, in case primary hardware fails. You will see how to mount two servers, one "passive", waiting for the active to stop responding and start up. There are different ways to do this.
  • The need to monitor a large volume of systems and manage them in a centralized way (more than 2500 agents). In order to do so, different Pandora FMS Servers are configured to be coordinated by the system called 'Metaconsole'. They can be linearly scaled in this way.


Pandora metaconsole overview2.png

The metaconsole model

Go back to Pandora FMS Documentation Index