Features Release

The next alert system in Pandora

December 19, 2008

The next alert system in Pandora

Another one of the systems we want to improve in the next Pandora FMS version is the alert system. At this moment, reusing alerts between different modules is impossible. If you want to do so you must define a new alert, with the same parameters, in another module. That makes using alerts annoying and that’s not what we want.

One of our partners, based on his own experience, propose us to rewritte the alert system and, after reviewing and discussing it, we accepted the plan and start working on it. We’re in the begining of the development, which may be a good moment to talk about it…

The proposal defines new terms for the alert. One is “template“, which we could defined as a set of rules and conditions to fire an alert. It’s not assigned to any module of an agent, but defines values, thresholds and similar in a wide way. There would be new types to check if the values of a module are satisfied or not, these values are:

  • Regular expression
  • Min & max
  • Min (ignoring max value)
  • Max (ignoring min value)
  • Equal to
  • Not equal to

The creation is done using a three steps wizard. This is because we would like to reduce the size of the forms in Pandora. A new user (or even an advanced one) may be scared when confronting a form with thousands of parameters fo fill. We want to guide the user step by step, so he can undestand easily how the system works and what he needs to put on each field. Here’s an animation of the three steps in the creation wizard.

New flashy wizards in Pandora!
New flashy wizards in Pandora!

Let’s continue with the other term in this new system. We have “commands“, which will be very similar to the alerts we have now. A command is the process that the server would do when an alert would be fired. You know you can email, play a sound through the computer speaker, send an SMS to a mobile phone, add a line to the syslog or whatever you want. These actions are done launching system commands in the server. These commands may receives some parameters, depending of the action we want to do, so there are some macros which would be replaced for the right values before the server runs the final command.

Nothing really new in this screenshot
Nothing really new in this screenshot

Other new term is “action“. Let’s assume you always want to send an email to [email protected] In Pandora FMS 2, when you are creating an alert in a module, you must set this email in every alert you create. That’s very annoying, and if your email changes to [email protected] you must review every defined alert in every module in every agent. That would finish with the patient of any sysadmin in the world… We love sysadmins, and we don’t want that to happen, do we? So, an “action” is a first customization for the “commands“. In that example, you would have an action name “Email Bob” which sets the right email to Bob once and will be reused many. As you can see in the editor, we added a live preview of how the final command would look like.

With live preview of the command
With live preview of the command

Ok, so we have “commands“, “actions” and “templates“. Let’s put it all together and define an alert in a module like “If the module value does not match the template, let’s do some action“. A very important thing here is that we won’t be limited to 1 action, we could do as many actions as we want!

(Sorry, I have no screenshot for this yet)

A real world example of an alert in Pandora would be:

If the CPU on MyServer is higher than 95(%) during 3 hours and today is Saturday or Sunday then: Email to Bob, send a SMS to John and add an internal event to the Pandora FMS system

It is amazing, isn’t it?

The interface to define an alert in a module would be (and here’s the major improvement) very easy to use. All you have to do is locate the module you want, create an alert and assign template (“CPU high“) and actions (“Email to Bob“, “Send SMS to John“, “Internal event“). We’re working in make it using AJAX so, besides we would be cool by using a cool technology, the interface will be easy and quick to use.

Note: This system will be available in the free version, so you just have to sit and wait :)

Note: Migrating for the current system to the new one is going to be a hard task, so we will ask for help to build an easy migrator tool.


Written by:



6 comments
  1. How about being able to add multiple alert templates to a module? You would basically have a template with multiple modules assigned to it and multiple actions. The alerts could then be defined and sit outside the agent-module connection.

  2. Evi, Of course that would possible in this system. You can assign many templates to a module. I didn't mention nothing about it because I think it was obvious :) Un saludo

  3. Avatar

    Glen Eustace

    I am not sure whether this scenario will handle escalation, but one feature I liked about nagios was that I could have an action performed the first time an alert is fired ( say occurence 1 - 5 ) and then a different action if it fires again ( i.e. occurence 6 ). The idea being that 1 -5 might send an email to the NOC and occurence 6 sends an additional SMS to the NOC manager. It was a pretty flexible system. Perhaps if the poposed changes will not facilitate this, it might be considered.

  4. Hi Glen, Sorry for the late reply, but I was out for the holidays. We discuss about your proposition and since it might be useful and not quite difficult, whe decided to implemented it. So from now on, you can configure something like: If "CPU is high" on module "CPU" then - Send an email if it's the first time. - Send an SMS if it's between the third and the fifth time the alert fires. - Create an internal log entry always Un saludo

  5. Avatar

    Marcus

    Hi Guys, stumbled over Pandora some days ago and giving it a try as it seems the most promising monitoring solution as an alternative for (slowly-developing) nagios. Implementing a flexible escalation system is one further step forward. Keep on going! :-)

  6. Avatar

    centr 2009

    I to like Your editing.

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.