Table of Contents
Pandora FMS Server maintenance
Pandora FMS infrastructure does not need external maintenance, but it is very important to purge old data, and maintain the database as much compacted as possible. There is an essential tool for the proper functioning of Pandora FMS which is in charge of carrying out these tasks. This tool is here:
Or here in case of Pandora FMS Enterprise version:
This tool, hereinafter pandora_db.pl is included in Pandora FMS server package, so it must be executed from a system where there is a Pandora FMS server installed. If there are two systems, one for the console and one for the server, pandora_db must be executed where Pandora FMS server is hosted. Pandora_db is a key tool for Pandora FMS to work properly, and that is why its execution is programmed in system cron tasks with an hourly interval. Its execution is configured within the file:
This tool performs all database maintenance tasks automatically:
- It deletes old data.
- It compresses existing data, interpolating them at several intervals, so that graphics are the same but the space needed to store them is much smaller (this is one reason why Pandora FMS is able to process such an amount of information).
- It checks the consistency of the database for non-existing modules, or modules that are not used because they can not be started (these modules appear in the tactical view as uninitialized modules).
- It deletes the daily agent contact information. Pandora FMS does not need more than 24hr agent contact history, and if it builds up, it slows down database access.
- In the enterprise version, it moves all old data to the standby history database.
As mentioned before, pandora_db execution is configured in system cron tasks and although this execution is automatically included in Pandora FMS server installation, it is convenient to check it. Therefore, /etc/cron.hourly/pandora_db must exist and contain this line:
"/usr/share/pandora_server/util/pandora_db.pl" "/etc/pandora/pandora_server.conf" >/dev/null 2>&1
Or in Pandora FMS Enterprise version:
"/usr/bin/pandora_db" "/etc/pandora/pandora_server.conf" >/dev/null 2>&1
It is equally important to check permissions and the file's owner. The appropriate file permissions would be 755, which can be granted by executing:
chmod 755 /etc/cron.hourly/pandora_db
Regarding the owner, it must be “root” both for the user and group, which is set executing:
chown root:root /etc/cron.hourly/pandora_db
Maintenance tool manual execution
If necessary, it is possible to launch pandora_db execution manually as it was exposed in the previous section. Execute from a shell console the command:
Or in Pandora FMS enterprise version:
It should show an output similar to this one:
Pandora FMS DB Tool 7.0NG.719 PS180221 Copyright © 2004-2015 Artica ST
This program is Free Software, licensed under the terms of GPL License v2 You can download latest versions and documentation at http://www.pandorafms.org Pandora DB now initialized and running (PURGE=7 days, COMPACT=0 days, STEP=1) . [*] Pandora FMS Enterprise module loaded. Starting at 2018-03-12 12:40:54 12:40:55 [PURGE] Deleting old extended session data. 12:40:55 [PURGE] Deleting old inventory data. 12:40:55 [PURGE] No data in tagente_datos_inventory. 12:40:55 [PURGE] No data to purge in tagente_datos. 12:40:55 [PURGE] Deleting old export data from tserver_export_data 12:40:55 [PURGE] Deleting old session data from tsessions_php 12:40:55 [PURGE] No data in tagente_datos_log4x. 12:40:55 [PURGE] No data in tagente_datos_string. 12:40:55 [PURGE] Deleting old event data at tevento table (More than 7 days). 12:40:55 [PURGE] Deleting old audit data (More than 7 days). 12:40:55 [PURGE] Deleting old SNMP traps (More than 7 days). 12:40:55 [ENTERPRISE] Deleting old policy queue entries (More than 7 days)... 12:40:55 [ENTERPRISE] Deleting invalid service elements... 12:40:55 [PURGE] Deleting old GIS data (More than 7 days). 12:40:55 [PURGE] Deleting pending delete modules (data table). 12:40:55 [PURGE] Deleting pending delete modules (status, module table). 12:40:55 [PURGE] Deleting old access data (More than 24hr) 12:40:55 [PURGE] No agent access data to purge. 12:40:55 [PURGE] Delete contents in report that have some deleted modules. 12:40:55 [PURGE] Delete contents in report that have some deleted agents. 12:40:55 [PURGE] Delete empty contents in report (like SLA or Exception). 12:40:55 [PURGE] Delete autodisabled agents where last contact is bigger than 30 days. 12:40:55 [PURGE] Deleting old netflow data. 12:40:55 [PURGE] Deleting old log data. 12:40:55 [PURGE] Deleting log data older than 90 days. 12:40:55 [PURGE] Deleting old special days. 12:40:55 [CHECKDB] Ignoring not-init data. 12:40:55 [CHECKDB] Checking database consistency (Missing status). 12:40:55 [CHECKDB] Checking database consistency (Missing module). 12:40:55 [CHECKDB] Updating empty aliases. 12:40:55 [INTEGRITY] Cleaning up group stats. 12:40:55 [INTEGRITY] Deleting orphan alerts. 12:40:55 [INTEGRITY] Deleting orphan modules. [HISTORYDB] Moving data older than 5 days to the history DB... [HISTORYDB] Moving events older than 5 days to the history DB... 12:40:55 [ENTERPRISE] Moving SNMP modules back to the Enterprise SNMP Server. 12:40:55 [ENTERPRISE] Dynamically updating critical min and max values. Ending at 2018-03-12 12:40:55
This may take hours in overloaded systems, so it is recommended to execute the process in the background.
To execute manually the maintenance tool and leave it in the background, execute:
nohup /usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server.conf
Or in Pandora FMS Enterprise version:
nohup /usr/bin/pandora_db /etc/pandora/pandora_server.conf
The process will take some time until it loads completely in the background. Then, you may close the shell console window, while the process will still be executed.
In some installations the tool directory could change. The most common one is:
In Pandora FMS previous versions, it could be found at:
It is very important to make sure you are using the updated tool version, and not that of a previous version. If you execute the program without arguments, it will show the tool version at the head of the message and it must match the version installed on the server.
The command mysqldump will do a complete database dump of the table structure as well as its contents. This command has several options to do a backup, although the focus here will be its most basic usage. That means doing it from the same system where the database is hosted. Indicate the database name from which the backup will be done and the credentials to access it:
mysqldump -u <user> -p <data_base>
For instance, to backup “pandora” database and dump the result to a file execute:
mysqldump -u root -p pandora > /backup/pandoradb_backup.sql
The result will be a database copy in the directory /backup/pandoradb_backup.sql.
Having a backup performed this way, you may restore the database completely. The process is quite simple, just log in MySQL, create the database to restore and load the backup in that database. Following the previous example, just execute:
[[email protected] ~]# mysql -u root -p mysql> create database pandora; mysql> use pandora; mysql> source /backup/padnoradb_backup.sql
Finally set configured user permissions again both in Pandora FMS console and server so they have total access to the database:
grant all privileges on pandora.* to [email protected] identified by 'mypassword';
It is important to bear in mind that this is JUST for performing a database backup/recovery and not any other files such as server setup.
Pandora FMS complete backup and restore
The previous point dealt with Pandora FMS database backup, and this one discusses the steps to back up and restore Pandora FMS completely.
There is a script in Pandora FMS server distribution that is useful to backup and restore Pandora FMS completely. This script is intended for backup and restoration in systems where the server and console are located in the same machine. If there are several components in your environment, then you should use the tool with the most adequate parameters for its use or modify them so they are tailored to your needs.
In order for it to carry out its tasks, this script needs to be executed as root.
It is located at:
If it is executed without parameters, it will provide some help:
Pandora FMS Command line backup tool. http://www.pandorafms.org (c) 2009 Sancho Lerena <[email protected]>, Artica Soluciones Tecnologicas Syntax: -c Path to Pandora FMS console, p.e: /srv/www/htdocs/pandora_console -d Destination path for backup file. p.e: /tmp -s Source filename for backup restore. p.e: /tmp/pandorafms -f Restore also files -q Quiet. No output message (used for scripts/cron) -b No database backup/restore Please BE SURE TO USE RESTORE (-s) option. This will OVERWRITE ALL your PandoraFMS install, including files, configuration and data. Please backup first!
This script is designed to back up and restore the following components:
- Server configuration file(s).
- Files waiting for execution, and also agent remote configuration files.
- Complete DB.
- Complete WEB Console.
Backup source and destination options
This script obtains access credentials to the database directly from the WEB console configuration. That is why the full path with the -c must be sent to the WEB console. That same parameter is also used to back up the web console itself.
The backup's destination is specified with the -d parameter. The compressed backup file will be lodged in that path, with a name similar to pandorafms_backup_xxxxxxx.tar.gz.
That way the following command can perform a full environment backup
/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -d /tmp/
With the -b parameter the database backup can be prevented:
/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -d /tmp/ -b
To restore the database with the script, replace the parameter -d by -s, indicating in this particular case the path to the backup:
/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-16-53.tar.gz
This is the default restoration, with no directories included.
Database and directory restore
The -f option allows restoring backup files (overwriting the current ones). Since overwritting current configuration files could have serious consequences, use -f if you want to do a backup recovery and you want it to restore all Pandora FMS files (console and server).
/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-16-53.tar.gz -f
Remember restoring user permissions to the database with the grant command.
File Restoration (not database)
Same as with the previous option, it is possible to restore only the files, without including the database. To do it, add the -boption to the previous execution.
Data Restoration, without Files
It is the default option. For doing this, you will not have to use neither the -b option nor the -f option.
/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-16-53.tar.gz -f -b
To do a full Pandora FMs backup, execute:
/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -d /tmp/
It will return something similar to this:
Backup completed and placed in /tmp//pandorafms_backup_2018-03-12-15-33-13.tar.gz
This means that the backup is at /tmppandorafms_backup_2018-03-12-15-33-13.tar.gz.
To restore the backup automatically, you need a console with the authentication credentials on the database correctly defined.
/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-33-13.tar.gz -f
It will give back something similar to:
Detected Pandora FMS backup at /tmp/pandorafms_backup_2018-03-12-15-33-13.tar.gz, please wait... Dropping current database Restoring backup database Restoring files and configuration Done. Backup in /tmp/pandorafms_backup_2018-03-12-15-33-13.tar.gz restored
Restore backup in case of console loss
If you have lost the Pandora FMS console, but you have a backup generated by this tool, then first restore the console directory. So, decompress manually its backup:
cd /tmp tar zxvf pandorafms_backup_2018-03-12-15-33-13.tar.gz
This will decompress your WEB console full directory in /tmp. In case of the generated backup of the previous example, it creates a directory named:
Copy the content of all that directory to to your web publication directory, which may vary depending on your distribution:
cp -R /tmp/var/www/html/pandora_console /var/www/html
Then restore the backup as usual.
Manual startup/shutdown for Pandora FMS servers
To start and / or stop the server manually Pandora FMS is running the following in a shell console:
Version NG 756 or later.
From version NG 756 onwards the previous instructions will also start the pandora_ha service.
To start and/or stop manually only the Pandora FMS server, you have to execute the following in a shell console:
You can monitor the status of pandora_ha through systemd with:
systemctl status pandora_ha.service
Watchdog implementation for Pandora FMS
In the code repository, there is a small script that is used as a “watchdog” (Watchdog). This script performs a Pandora FMS status monitoring. Thus you may perform a recovery operation (trying to start Pandora FMS), and if that fails, a warning must be issued. This tool is at /usr/share/pandora_server/util/pandora_watchdog.sh .
Script for alert generation
The pandora_watchdog.sh script looks for a file in /usr/bin/pandora_alert with the instrucctions to generate the alert. This file must be created defining the code to be executed when the watchdog cannot lift the Pandora FMs server. In this example, in addition to warning through a SMS, it stops the Tentacle server:
#!/bin/bash sendsms +34458474843 "The Pandora FMS has suffered some issue and it cannot be started." /etc/init.d/tentacle_serverd stop
Grant execution permissions to this script:
chmod 750 /usr/bin/pandora_alert
To launch the watchdog and leave it workin in the background, execute the following:
nohup /usr/share/pandora_server/util/pandora_watchdog.sh &
When starting a watchdog, bear in mind that if for maintenance reasons you wish to manually stop Pandora FMS server, you must also previously have stopped the watchdog process to prevent it from trying to start the sercice automatically continuously.
A history database is a database where old module data is transferred to make the main Pandora FMS database more responsive for everyday operations. That data will still be available seamlessly to the Pandora FMS console when viewing reports, module charts etc.
Setting up a historical database
To configure a historical database, it is necessary to have a new server where to host it (different from the one of the main database). Once you have that server where MySQL is installed, follow these steps:
- Create the new historical database:
[[email protected] ~]# mysql -u root -p mysql> create database pandora_history;
- Create the Pandora FMS database schema. You may use the script /var/www/html/pandora_console/pandoradb.sql included in Pandora FMS console, copying it to the historical database server:
cat pandoradb.sql | mysql -u root -p -D pandora_history
- Grant permissions to a user which will be used from the Pandora FMS server and console to send and check historical data:
GRANT ALL PRIVILEGES ON pandora_history.* TO 'user'@'%' IDENTIFIED BY 'password';
- In Pandora FMS console, go to Setup> Setup> History database and configure the host, port, database, user and password to access the historical database.
The last fields of this form (Days, step and delay) will define the way in which the data will be sent to the historical database, that is, the oldest data (those with more Days) will be moved to the historical database in blocks of made of rows (Step) waiting for seconds (Delay) between one block and the next to avoid overloads.
In the same screen, it is also possible to decide whether to send the events with more “Event days” to the historical database, although it must be taken into account that including the events will considerably increase the growth rate of the historical database, and that these will only be consulted when generating reports, not in the event view.
The historical database is a feature of the Enterprise version that uses the /usr/bin/pandora_db binary to transfer data.
Setting up history database purge and compactation
History database is supposed to contain “all history data”, but if you want to delete old data, or compact them, you will need to execute pandora_db script using specific data from the database and a “fake” configuration file, to be able to use the maintenance script as if you were using a normal database.
First, enter some data from the history database in tconfig table. Use this SQL queries to create a minimal configuration for the behaviour of the pandora_db running against the history database.Connect your database, using mysql CLI tool.
This is an example, replace values as you need (but let history_db_enabled to zero):
INSERT INTO `tconfig` VALUES (1,'days_purge','180'); INSERT INTO `tconfig` VALUES (2,'history_db_enabled','0'); INSERT INTO `tconfig` VALUES (3,'days_compact','120'); INSERT INTO `tconfig` VALUES (4,'step_compact','1'); INSERT INTO `tconfig` VALUES (5,'event_purge','180'); INSERT INTO `tconfig` VALUES (6,'string_purge','180'); INSERT INTO `tconfig` VALUES (7,'MR','0');
This means data in history database is stored for 6 months (180 days) from the execution date, and compact data more tha 4 months old (120 days). If you have 1 month worth of data in your “main” database, you will have a total of 6 months of data, because the last month has no data in the history database, but it has them in the main database. You may type in any value here, there is no limit of data storage in history database. Just remember, history database MUST BE IN A DIFFERENT PHYSICAL SERVER, not in the same host/system where the main database is and/or Pandora FMS server or consol is running.
Secondly, create an aditional pandora_server.conf file, use this “small version” to create your own and name it /etc/pandora/pandora_server_history_db.conf :
dbengine mysql dbname pandora4_history dbuser pandora4_history dbpass 1234 dbhost 192.168.50.23 log_file /var/log/pandora/pandora_db_history.log
Now you can execute pandora_db tool against the history database configuration and program it for its periodical execution:
The INSERT values can be also entered in the console as explained in the following link: historical_database_maintenance_options
This process SHOULD NOT affect your main operation because it is running against a different database in a different server.