High Availability Database Cluster with Pandora FMS

Introduction

Pandora FMS relies on a MySQL database for configuration and data storage. A database failure can temporarily bring your monitoring solution to a halt. The Pandora FMS high availability database cluster allows you to easily deploy a fault-tolerant, robust architecture.

Cluster resources are managed by Pacemaker, an advanced, scalable High Availability database cluster resource manager. Corosync provides a closed process group communication model for creating replicated state machines. Percona, a backwards-compatible replacement for MySQL, was chosen as the default RDBMS for its scalability, availability, security and backup features.

Active/passive replication takes place from a single master node (writable) to any number of slaves (read only). A virtual IP address always points to the current master. If the master node fails, one of the slaves is promoted to master and the virtual IP address is updated accordingly.

Base de datos de alta disponibilidad

The Pandora FMS Database HA Tool, pandora_ha, monitors the cluster and makes sure the Pandora FMS Server is always running, restarting it when needed. pandora_ha itself is monitored by systemd.

Prerequisites and installation

Throughout this article, we will configure and test a two node cluster with Pandora FMS. You will need an already installed Pandora FMS Enterprise version and two additional hosts running CentOS version 7.

To configure the cluster follow the official Pandora FMS High Availability Database Cluster installation guide. Although it is a straightforward, step-by-step guide, some experience with the Linux command line is recommended.

Once finished, ssh to any of the two nodes and run pcs status to check the status of the cluster. Both nodes should be online:

HA cluster status

The cluster is up and running. You can check its status from your Pandora FMS Console too. Simply navigate to Servers -> Manage database HA (tip: clicking on the loupe icon displays the output of pcs status without having to ssh to that node):

HA cluster status in Pandora FMS

The Pandora FMS High Availability Database Tool automatically creates agents to monitor your cluster. They can be accessed from the previous view by clicking on the node’s name displayed in the Hostname column:

HA cluster modules

You should configure alerts for, at least, the Node Status, Slave IO Running and Slave SQL Running modules. These modules go critical when there’s a problem, so you can use Pandora’s default Critical condition alert template:

HA cluster alerts

Operating the cluster

Let’s see what happens when things go wrong. In this section we will test the behaviour of our high availability database cluster in different situations.

Manual failover

We may need to stop one of the nodes for maintenance, for example. Stopping a slave is a trivial case. Let’s stop the master instead:

HA Cluster manual failover

The former master is on standby and the slave has been promoted to master:

HA Cluster standby

Let’s get it back online:

HA cluster manual failover unstandby

The former master is online, and it is a slave now. The new master will not relinquish its role unless it fails.

 HA cluster cleanup

Our Pandora FMS server is running, as expected:

HA cluster server status

Crash of the master

If the process of the database server crashes, it will be automatically restarted:

HA cluster crash

However, if there is more than one crash in a one hour span, the node will be blocked and a slave will be promoted to master:

HA cluster broom

Once the problem is fixed, click on the broom icon to recover the blocked node.

escoba cluster HA

Halting or rebooting one node

Halting or rebooting any node will leave the cluster with a master and the operation will not be affected:

HA cluster halt

HA cluster halt 2

If seconds behind master is greater than 0 when you bring the node back online, it means the slave has not yet replicated all the transactions that have taken place in the master. This value should start decreasing on its own until it reaches 0:

HA cluster

Broken replication

When the master is not in synchrony with the slave (i.e., the master is ahead of the slave) and it fails, it may not be able to start replication from the new master when it gets back online:

HA cluster replication broken

Some of the changes that were not replicated by the former slave may conflict with new changes that need to be replicated now:

 HA cluster replication broken 2

The easiest way to fix a broken node is to follow the instructions in the fixing a broken node section of the Pandora FMS High Availability Database Cluster installation guide, which implies reconstructing the node’s database from the master and starting replication again.

Adding additional nodes

Although not covered in this article, additional nodes can be added to the cluster for increased resilience. See the Pandora FMS High Availability Database Cluster installation guide for more information.

Conclusion

When using a Pandora FMS High-Availability Database Cluster, failures on slave nodes are not even noticed by the Pandora FMS Server. However, they should be acted upon quickly before other nodes fail.

A failure in the master node will cause a small interruption until one of the slaves is promoted to master, after which normal operation will automatically resume.

Your monitoring infrastructure will never be down again because of a database failure!

You can watch the tutorial on HA Database Cluster with Pandora FMS here:

Shares