Pandora: Documentation en: Server Management

From Pandora FMS Wiki
Jump to: navigation, search

Go back to Pandora FMS documentation index

Template wip.png

We are working on the translation of the Pandora FMS documentation. Sorry for any inconvenience.

 


1 Pandora FMS Server maintenance

1.1 Database management

Pandora FMS infrastructure does not need external maintenance, but it is very important to purge old data, and maintain the database compacted. There is an essential tool for the proper functioning of Pandora FMS which is in charge of carrying out these tasks. This tool is here:

/usr/share/pandora_server/util/pandora_db.pl

Or here in case of Pandora FMs Enterprise version:

/usr/bin/pandora_db

This tool, hereinafter pandora_db.pl is included in Pandora FMS server package, so it must be executed from a system where there is a Pandora FMS server installed. If you for example have two systems, one for the console and one for the server, pandora_db must be executed where Pandora FMS server is hosted. Pandora_db is a key tool for Pandora FMS to work properly, and that is why its execution is programmed in system cron tasks with an **hourly periodicity**. Its execution is configured within the file:

/etc/cron.hourly/pandora_db

This tool performs all database maintenance tasks automatically:

  • It deletes old data.
  • It compresses existing data, interpolating them at several intervals, so that graphics are the same but the space needed to store them is much smaller (this is one reason why Pandora FMS is able to process such an amount of information).
  • It checks the consistency of the database for non-existing modules, or modules that are not used because they can not be started (these modules appear in the tactical view as uninitialized modules).
  • It deletes the daily agent contact information. Pandora FMS does not need more than 24hr agent contact history, and if it builds up, it slows down database access.
  • In the enterprise version, it moves all old data to the standby history database.

As mentioned before, pandora_db execution is configured in system cron tasks and although this execution is automatically included in Pandora FMS server installation, it is convenient to check it. Therefore, /etc/cron.hourly/pandora_db must exist and contain this line:

"/usr/share/pandora_server/util/pandora_db.pl" "/etc/pandora/pandora_server.conf" >/dev/null 2>&1

Or in Pandora FMS Enterprise version:

"/usr/bin/pandora_db" "/etc/pandora/pandora_server.conf" >/dev/null 2>&1

It is equally important to check permissions and the file's owner. The appropriate file permissions would be 755, which can be granted by executing:

chmod 755 /etc/cron.hourly/pandora_db

Regarding the owner, it must be "root" both for the user and group, which is set executing:

chown root:root /etc/cron.hourly/pandora_db

1.2 Maintenance tool manual execution

If necessary, it is possible to launch pandora_db execution manually as it was exposed in the previous section. Execute from a shell console the command:

/usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server.conf

Or in Pandora FMS enterprise version:

/usr/bin/pandora_db /etc/pandora/pandora_server.conf

It should show an output similar to this one:

Pandora FMS DB Tool 7.0NG.719 PS180221 Copyright (c) 2004-2015 Artica ST

This program is Free Software, licensed under the terms of GPL License v2
You can download latest versions and documentation at http://www.pandorafms.org

Pandora DB now initialized and running (PURGE=7 days, COMPACT=0 days, STEP=1) .

 [*] Pandora FMS Enterprise module loaded.

Starting at 2018-03-12 12:40:54
12:40:55 [PURGE] Deleting old extended session data.
12:40:55 [PURGE] Deleting old inventory data.
12:40:55 [PURGE] No data in tagente_datos_inventory.
12:40:55 [PURGE] No data to purge in tagente_datos.
12:40:55 [PURGE] Deleting old export data from tserver_export_data

12:40:55 [PURGE] Deleting old session data from tsessions_php

12:40:55 [PURGE] No data in tagente_datos_log4x.
12:40:55 [PURGE] No data in tagente_datos_string.
12:40:55 [PURGE] Deleting old event data at tevento table (More than 7 days).
12:40:55 [PURGE] Deleting old audit data (More than 7 days).
12:40:55 [PURGE] Deleting old SNMP traps (More than 7 days).
12:40:55 [ENTERPRISE] Deleting old policy queue entries (More than 7 days)...
12:40:55 [ENTERPRISE] Deleting invalid service elements...
12:40:55 [PURGE] Deleting old GIS data (More than 7 days).
12:40:55 [PURGE] Deleting pending delete modules (data table).
12:40:55 [PURGE] Deleting pending delete modules (status, module table).
12:40:55 [PURGE] Deleting old access data (More than 24hr)
12:40:55 [PURGE] No agent access data to purge.
12:40:55 [PURGE] Delete contents in report that have some deleted modules.
12:40:55 [PURGE] Delete contents in report that have some deleted agents.
12:40:55 [PURGE] Delete empty contents in report (like SLA or Exception).
12:40:55 [PURGE] Delete autodisabled agents where last contact is bigger than 30 days.
12:40:55 [PURGE] Deleting old netflow data.
12:40:55 [PURGE] Deleting old log data.
12:40:55 [PURGE] Deleting log data older than 90 days.
12:40:55 [PURGE] Deleting old special days.
12:40:55 [CHECKDB] Ignoring not-init data.
12:40:55 [CHECKDB] Checking database consistency (Missing status).
12:40:55 [CHECKDB] Checking database consistency (Missing module).
12:40:55 [CHECKDB] Updating empty aliases.
12:40:55 [INTEGRITY] Cleaning up group stats.
12:40:55 [INTEGRITY] Deleting orphan alerts.
12:40:55 [INTEGRITY] Deleting orphan modules.
[HISTORYDB] Moving data older than 5 days to the history DB...
[HISTORYDB] Moving events older than 5 days to the history DB...
12:40:55 [ENTERPRISE] Moving SNMP modules back to the Enterprise SNMP Server.
12:40:55 [ENTERPRISE] Dynamically updating critical min and max values.
Ending at 2018-03-12 12:40:55

Template warning.png

This may take hours in overloaded systems, so it is recommended to execute the process in the background.

 


To execute manually the maintenance tool and leave it in the background, execute:

nohup /usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server.conf

Or in Pandora FMS Enterprise version:

nohup /usr/bin/pandora_db /etc/pandora/pandora_server.conf

The process will take some time until it loads completely in the background. Then, you may close the shell console window, while the process will still be executed.


Info.png

In some installations the tool directory could change. The most common one is:

/usr/share/pandora_server/util/

In Pandora FMS previous versions, it could be found at:

/usr/share/pandora/util/

 


Template warning.png

It is very important to make sure you are using the updated tool version, and not that of a previous version. If you execute the program without arguments, it will show the tool version at the head of the message and it must match the version installed on the server.

 


1.3 Database Backup

The command mysqldump will do a complete database dump of the table structure as well as its contents. This command has several options to do a backup, although the focus here will be its most basic usage, that means doing it from the same system where the database is hosted. Indicate the database name from which the backup will be done and the credentials to access it:

mysqldump -u <user> -p <data_base>

For instance, to backup "pandora" database and dump the result to a file execute:

mysqldump -u root -p pandora > /backup/pandoradb_backup.sql

The result will be a database copy in the directory/backup/pandoradb_backup.sql.

Having a backup performed this way. you may restore the database completely. The process is quite simple, just log in MySQL, create the database to restore and load the backup in that database. Following the previous example, just execute:

[[email protected] ~]# mysql -u root -p
mysql> create database pandora;
mysql> use pandora;
mysql> source /backup/padnoradb_backup.sql

Finally set configured user permissions again both in Pandora FMS console and server so they have total access to the database:

grant all privileges on pandora.* to [email protected] identified by 'mypassword';

Template warning.png

It is important to bear in mind that this is JUST for performing a database backup/recovery and not any other files such as server setup.

 


1.4 Pandora FMS complete backup and restore

The previous point dealt with Pandora FMS database backup, and this one discusses the steps to back up and restore Pandora FMS completely. There is an script in Pandora FMS server distribution that is useful to backup and restore Pandora FMS completely. This script is intended for backup and restoration in systems where the server and console are located in the same machine. If there are several components in your environment, then you should use the tool with the most adequate parameters for its use or modify them so they are tailored to your needs.

In order for it to carry out its tasks, this script needs to be executed as root.

It is located at:

/usr/share/pandora_server/util/pandora_backup.sh 

If it is executed without parameters, it will provide some help:

Pandora FMS Command line backup tool. http://www.pandorafms.org
(c) 2009 Sancho Lerena <[email protected]>, Artica Soluciones Tecnologicas

Syntax:
		-c Path to Pandora FMS console, p.e: /srv/www/htdocs/pandora_console
		-d Destination path for backup file. p.e: /tmp
		-s Source filename for backup restore. p.e: /tmp/pandorafms
		-f Restore also files
		-q Quiet. No output message (used for scripts/cron)
 		-b No database backup/restore


Please BE SURE TO USE RESTORE (-s) option. This will OVERWRITE ALL your
PandoraFMS install, including files, configuration and data. Please backup first!

This script is designed to back up and restore the following components:

  • Server configuration file(s).
  • Files waiting for execution, and also agent remote configuration files.
  • Complete DB.
  • Complete WEB Console.

Backup origin and destination options

This script obtains access credentials to the database directly from the WEB console configuration. That is why the full path with the -c must be sent to the WEB console. That same parameter is also used to back up the web console itself.

The backup's destination is specified with the -d parameter. The compressed backup file will be lodged in that path, with a name similar to pandorafms_backup_xxxxxxx.tar.gz.

That way the following command can perform a full environment backup

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -d /tmp/

With the -b parameter the database backup can be prevented:

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -d /tmp/ -b

Database restore

To restore the database with the script, replace the parameter -d by -s, indicating in this particular case the path to the backup:

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-16-53.tar.gz

This is the default restoration, with no directories included.

Database and directory restore

The -f option allows restoring backup files (overwriting the current ones). Since overwritting current configuration files could have serious consequences, use -f if we want to do a backup recovery and we want to it restore all Pandora FMS files (console and server).

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-16-53.tar.gz -f

Template warning.png

Remember restoring user permissions to the database with the grant command.

 


File Restoration (not database)

Same as with the previous option, it is possible to restore only the files, without including the database. To do it, add the -boption to the previous execution.

Data Restoration, without Files

It is the default option. For doing this, you will not have to use neither the -b option nor the -f option.

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-16-53.tar.gz -f -b

1.4.1 Usage examples

Create backup

To do a full Pandora FMs backup execute:

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -d /tmp/

It will return something similar to this:

Backup completed and placed in /tmp//pandorafms_backup_2018-03-12-15-33-13.tar.gz

This means that the backup is at /tmp//pandorafms_backup_2018-03-12-15-33-13.tar.gz.

Restore backup

To restore the backup automatically, you need a console with the authentication credentials on the database correctly defined.

Execute:

/usr/share/pandora_server/util/pandora_backup.sh -c /var/www/html/pandora_console/ -s /tmp/pandorafms_backup_2018-03-12-15-33-13.tar.gz -f 

It will give back something similar to:

Detected Pandora FMS backup at /tmp/pandorafms_backup_2018-03-12-15-33-13.tar.gz, please wait...
Dropping current database
Restoring backup database
Restoring files and configuration
Done. Backup in /tmp/pandorafms_backup_2018-03-12-15-33-13.tar.gz restored

Restore backup in case of console loss

If you have lost the Pandora FMS console but you have a backup generated by this tool,then first restore the console directory. So, decompress manually its backup:

cd /tmp
tar zxvf pandorafms_backup_2018-03-12-15-33-13.tar.gz

This will decompress your WEB console full directory in /tmp. In case of the generated backup of the previous example, it creates a directory named:

/tmp/var/www/html/pandora_console

Copy the content of all that directory to to your web publication directory, which may vary depending on your distribution:

cp -R /tmp/var/www/html/pandora_console /var/www/html

Then restore the backup as usual.

1.5 Manual startup/shutdown for Pandora FMS servers

To start and / or stop the server manually Pandora FMS is running the following in a console shell':

Stop daemon:

/etc/init.d/pandora_server stop

Start daemon:

/etc/init.d/pandora_server start

Restart daemon:

/etc/init.d/pandora_server restart

1.6 Watchdog implementation for Pandora FMS

In the repository there is a small script that is used as a "watchdog" (Watchdog). This script performs a monitoring Pandora (who monitors who monitored?). Thus we can perform a recovery operation (trying to lift Pandora), and if that fails, we can tell the event.

1.6.1 pandora_watchdog.sh

#!/bin/bash
# Copyright (c) 2005-2012 Artica ST
# Author: Sancho Lerena <[email protected]> 2009
# Licence: GPL2
#
# daemon_watchdog
#
# Generic watchdog to detect if a daemon is running. If cannot restart, execute
# a custom-user defined command to notify daemon is down and continues in
# standby (without notifying / checking) until daemon is alive again.

# Default configuration is for Pandora FMS Server daemon

# =====================================================================
# Configuration begins here. Please use "" if data contain blank spaces

export DAEMON_WATCHDOG=pandora_watchdog.sh
# DAEMON_WATCHDOG: Name of this script. Used to check if its running already

export DAEMON_CHECK="/usr/bin/pandora_server /etc/pandora/pandora_server.conf"
# DAEMON_CHECK: Daemon monitored, please use full path and parameters like
#               are shown doing a ps aux of ps -Alf

export DAEMON_RESTART="/etc/init.d/pandora_server restart"
# DAEMON_RESTART: Command to try to restart the daemon

export DAEMON_DEADWAIT=90
# DAEMON_DEADWAIT: Time this script checks after detect that
#                  daemon is down before to consider is really down. 

export DAEMON_ALERT="/usr/bin/pandora_alert"
# DAEMON_ALERT: Command/Script executed if after detecting daemon is down,
#               and waiting DAEMON_DEADWAIT, and daemon continues down. 

export DAEMON_LOOP=7
# DAEMON_LOOP: Interval within daemon_wathdog checks if daemon is alive.
#              DO NOT use values under 3-5 seconds or could be CPU consuming.
#              NEVER NEVER NEVER use 0 value or gets 100% CPU!.

# Configuration stop here
# =====================================================================

# Check if another instance of this script

RUNNING_CHECK=`ps aux | grep "$DAEMON_WATCHDOG" | grep -v grep |wc -l`
if [ $RUNNING_CHECK -gt 2 ]
then
        echo "Aborting, seems that there are more '$DAEMON_WATCHDOG' running in this system"
        logger $DAEMON_WATCHDOG aborted execution because another watchdog seems to be running
        exit -1
fi


# This value always must be 0 at start. Do not alter
export DAEMON_STANDBY=0

# This function replace pidof, not working in the same way in different linux distros
function pidof_daemon () (
        # This sets COLUMNS to XXX chars, because if command is run
        # in a "strech" term, ps aux don't report more than COLUMNS
        # characters and this will not work.
        COLUMNS=300
        DAEMON_PID=`ps aux | grep "$DAEMON_CHECK" | grep -v grep | tail -1 | awk '{ print $2 }'`
        echo $DAEMON_PID
)

# Main script

if [ ! -f `echo $DAEMON_CHECK | awk '{ print $1 }'` ]
then
        echo "Daemon you want to check is not present in the system. Aborting watchdog"
        exit
fi

while [ 1 ]
do

        DAEMON_PID=`pidof_daemon`
        if [ -z "$DAEMON_PID" ]
        then

echo "Checkpoint #1 $DAEMON_PID "
 
                if [ $DAEMON_STANDBY == 0 ]
                then

                        # Daemon down, first detection
                        # Restart it ! 

                        logger $DAEMON_WATCHDOG restarting $DAEMON_CHECK
                        $DAEMON_RESTART 2> /dev/null > /dev/null

                        # Just WAIT another DAEMON_DEADWAIT before consider it DEAD

echo "Going to DAEMON_DEADEWAIT"
 
                        sleep $DAEMON_DEADWAIT
                        DAEMON_PID=`pidof_daemon` 

                        if [ -z "$DAEMON_PID" ]
                        then

                                # Is dead and can't be restarted properly. Execute alert

echo "I cannot startup again the process"

                                logger $DAEMON_WATCHDOG $DAEMON_CHECK is dead, alerting !
                                $DAEMON_ALERT  2> /dev/null > /dev/null 

                                # Watchdog process puts in STANDBY mode until process get alive again
                                logger $DAEMON_WATCHDOG "Entering in Stabdby mode"

                                DAEMON_STANDBY=1
                       fi
               fi
       else

echo "Checkpoint #1B $DAEMON_PID "

                DAEMON_STANDBY=0
        fi

        sleep $DAEMON_LOOP
done

1.6.2 /usr/bin/pandora_alert

This is the script that acts when the watchdog process cannot start the process that monitors (pandora). In our case, besides alert by SMS, it disables Tentacle.

There will be given rights with chmod 750 / usr / bin / pandora_alert


#!/bin/bash
sendsms +34458474843 "Pandora FMS Server stopped and can't be started"
/etc/init.d/tentacle_serverd stop


1.6.3 Watchdog Startup

If you have copied pandora_watchdog.sh to /usr/bin, the manual way to start the wathdog will be:

nohup /usr/bin/pandora_watchdog.sh &

1.6.4 Remarks

Having a watchdog running on the system can cause unpleasant consequences if we do not consider that there is a watchdog. If for example, we want to make a Pandora to disconnect maintenance, the watchdog will automatically rise again, so we will go "crazy" if we do not stop watchdog first.

1.7 History database

A history database is a database where old module data is moved to make the main Pandora FMS database more responsive for everyday operations. That data will still be available seamlessly to the Pandora FMS console when viewing reports, module charts etc.

1.7.1 Setting up a historical database

To configure a historical database, it is necessary to have a new server in which to host it (different from the one of the main database). Once you have that server where MySQL is installed, follow these steps:

  • Create the new historical database:
[[email protected] ~]# mysql -u root -p
mysql> create database pandora_history;
  • Create the Pandora FMS database schema. You can use the script /var/www/html/pandora_console/pandoradb.sql included in Pandora FMS console, copying it to the historical database server:
cat pandoradb.sql | mysql -u root -p -D pandora_history
  • Grant permissions to a user which will be used from the Pandora FMS server and console to send and check historical data:
GRANT ALL PRIVILEGES ON pandora_history.* TO 'user'@'%' IDENTIFIED BY 'password';
  • In Pandora FMS console, go to Setup> Setup> History database and configure the host, port, database, user and password to access the historical database.


History db.png


The last fields of this form (Days, step and delay) will define the way in which the data will be sent to the historical database, that is, the oldest data (those with more Days) will be moved to the historical database in blocks of made of rows (Step) waiting for seconds (Delay) between one block and the next to avoid overloads.

In the same screen, it is also possible to decide whether to send the events with more Event days to the historical database, although it must be taken into account that including the events will considerably increase the growth rate of the historical database, and that these will only be consulted when generating reports, not in the event view.

Info.png

The historical database is a feature of the Enterprise version that uses the /usr/bin/pandora_db binary to transfer data

 


1.7.2 Setting up history database purge and compactation

History database is supposed to contains "all history data", but if you want to delete old data, or compact them, you will need to execute pandora_db script with a "fake" configuration data, to let it think is the "normal" behaviour. To do this, you need to enter some data in your History Database.

First, you need to "recreate" a functional table values in tconfig, with useful values to be used by pandora_db tool. Use this SQL queries ON YOUR history database to create a minimal configuration for the behaviour of the pandora_db running against history database. You fist, need to connect your database, using mysql CLI tool.

This is an example, replace values as you needed (but let history_db_enabled to zero):

INSERT INTO `tconfig` VALUES (1,'days_purge','180');
INSERT INTO `tconfig` VALUES (2,'history_db_enabled','0');
INSERT INTO `tconfig` VALUES (3,'days_compact','120');
INSERT INTO `tconfig` VALUES (4,'step_compact','1');
INSERT INTO `tconfig` VALUES (5,'event_purge','180');
INSERT INTO `tconfig` VALUES (6,'string_purge','180');

This means data in history is stored for 6 months (starting from NOW), and compacted from 4th month. If you have 1 month of data in your "main" database, you will have a total of 6 months of data, because the last month is coming from the main database, and the other five from the history database. You can put any value here, there is no limit of data storage in history database. Just remember, history database MUST BE IN A DIFFERENT PHYSICAL SERVER, not in the same host/system where is the main database and/or running Pandora FMS server or console.

Second, you need to create an aditional pandora_server.conf file, use this "small version" to create your own (replace the values for your history database values), name it /etc/pandora/pandora_server_history_db.conf :

dbengine mysql
dbname pandora4_history
dbuser pandora4_history
dbpass 1234
dbhost 192.168.50.23
log_file /var/log/pandora/pandora_db_history.log


Now you can execute pandora_db tool against the "fake" configuration:

/usr/share/pandora_server/util/pandora_db.pl /etc/pandora/pandora_server_history_db.conf


The INSERT values can be also entered in the console as explained in the following link: Historical database maintenance options

Info.png

This process SHOULD NOT affect your main operation because is running against a different database in a different server. The only possible delay should be if someone is trying to render a big ammount of data from history, that will take more time than usual.

 



Go back to Pandora FMS documentation index