Diferent modules intervals on same machine/agent

[Sancho]

Like it

Up

0

Down

Drop it

::

Not for this version, but its a feature we will implement in the next version.
I personally wanted to do for this version (1.2), but we doesnt have enought resources.

If your concerns are about “database usage”, so for example, insert an entry in DB with the same data( for example boolean data), now with Pandora 1.2 and its new management of data, based on store only changes over existing data, so data without changes is not stored. This would permit more agents and more modules.

Whatever, all your ideas are very important to make Pandora more usable and useful.

[JoaoA]

Like it

Up

0

Down

Drop it

::

Hi.

(Disclaimer: I am not the same Joao that started this thread, but I work with him. 😉 )

Actually our concern was not so much database usage, but resource usage on the agent’s machine.

While monitoring simple things like CPU and Memory usage does not impact machine performance much, some other modules that, for example, perform queries on a database or search for error messages on large log files can have a much bigger impact on the system, especially on servers with an already high load.

So our concern would be to run the modules only as needed, for example: CPU usage every 5 minutes, memory usage every 5 minutes, the log search every hour and the database query every day.

We have encountered only two major obstacles so far, both on the web interface:

1) The agent page only shows the information received on the last data transfer, which means that modules only appear in the listing when they send data (for the oracle query that would be once a day, and then only for 5 minutes, since the next data transfer would not include it). Due to this you have no way to check the graph for that module unless it sent data in the last transfer.

2) The graphics assume a scale equal to the minimum interval for the agent, so if our agent’s configured interval is 300 (5 minutes) and we have a module that sends data every 30 minutes, the graph would show a spike every 6 intervals but remain flat otherwise.

Obstacles we had already solved in 1.1, but haven’t tried in 1.2:

1) The agent’s module configuration now has an aditional parameter “module_interval”. A value of 1 here means the module is executed every time, a value of 2 every other time, etc.

2) The pandora_server no longer complains when it receives empty data for a module, and whenever the data is empty it also does not insert anything into the database.

[Sancho]

Like it

Up

0

Down

Drop it

::

1) The agent’s module configuration now has an aditional parameter “module_interval”. A value of 1 here means the module is executed every time, a value of 2 every other time, etc.

This could be interesting, could you email me your changes there ?.

About he another two questions:

1) The agent page only shows the information received on the last data transfer, which means that modules only appear in the listing when they send data (for the oracle query that would be once a day, and then only for 5 minutes, since the next data transfer would not include it). Due to this you have no way to check the graph for that module unless it sent data in the last transfer.

2) The graphics assume a scale equal to the minimum interval for the agent, so if our agent’s configured interval is 300 (5 minutes) and we have a module that sends data every 30 minutes, the graph would show a spike every 6 intervals but remain flat otherwise.

All of this two questions are now resolved in Pandora 1.2, if I understand well :-), so why are you waiting so long to test it ? 🙂

[JoaoA]

Like it

Up

0

Down

Drop it

::

In Pandora_agent.conf, each module now has an extra parameter “module_interval”, which MUST come before the “module_exec” parameter:

module_begin
module_name CPU_user
module_description User CPU usage
module_type generic_data
module_interval 1
module_exec vmstat 2 2| tail -1 | awk '{ print $20 }'
module_end

In pandora_agent.sh, we made the following changes:

Read the value of the new module configuration parameter “module_interval” and determine if we should run the module this time:

               if [ ! -z "`echo $a | grep '^module_interval'`" ]
                then
                        # Determine if execution is to be done
                        interval=`echo $a | cut -c 17-`
                        execute=`expr ( $CONTADOR - 1 ) % $interval`
                fi

Module_exec bloc tests variable $execute that was determined in “module_interval” bloc. If 0 module is executed and value is placed in XML output, otherwise the last value returned is used.

               if [ ! -z "`echo $a | grep '^module_exec'`" ]
                then
                        # Check if it's necessary to execute module
                        if [ $execute -eq 0 ]
                        then
                                # Execution to be done
                                execution=`echo $a | cut -c 13-`
                                res=`eval $execution`
                                if [ -z "$flux_string" ]
                                then
                                        res=`eval expr $res`
                                fi
                                echo "$res" >> $DATA

                                # Save value if module won't be executed next time
                                if [ $interval -ne 1 ]
                                then
                                        echo $res > ${PANDORA_SAVED_FILES}${name}
                                fi
                        else
                                # Set last execution value
                                echo "`cat ${PANDORA_SAVED_FILES}${name}`" >> $DATA
                        fi
                fi

[Sancho]

Like it

Up

0

Down

Drop it

::

This is a good implementation. I’m thinking about include in the next release of agents. I need to test it in different plattforms, but at first sight seems to be a good and simple solution to make agent modules running with different intervals. Thanks guys 🙂

[Joao]

Like it

Up

0

Down

Drop it

::

Good news then! 🙂 Glad to help!
Still some room for improvement I think…

Done so far:
– Database insertion avoidance for equal data packages (beta 1.2)
– Module interval configuration for overload avoidance (suggested in this post)

Suggestion:
– Data package transfer minimization by avoiding sending same data to the server. Besides not running module which should be an improvement by itself, not sending redundant data to the server would reduce network traffic (useful specially for big data packages / many installed modules). Server would have to be set not to receive data for a specific module as a normal situation.

Just a thought for evaluation.
Cheers

[Sancho]

Like it

Up

0

Down

Drop it

::

I was thinking about this problem some weeks ago, but I cannot imagine no example of “huge” data transfer between agent and server.

The problem could be to “economize” server stress, if you really send the same data, ok, database storage is not incremented but, what about database query, server XML process and CPU usage?. If an agent simply doesnt send unneeded data, huge ammount of server and database stress will be economized in a big enviroment.

I think that using your patches on agents and working a few hours its possible to make that, I need to take some spare hours of my job and work hard with that. We have 1.2 Documentation in some “freeze” state I don’t like.

Thanks for the idea. I really like to have a community of users with this kind of continuous and valuable feedback.

[Sancho]

Like it

Up

0

Down

Drop it

::

Actually I’ve done a new version of Pandora Agent 1.2 using some of your ideas:

– You can specify (optional) module_interval x where x is 1,2,3…x, 1 means exec this module on each cycle, 2 means exec this module each two cyles….

– Modules who has interval must specify module_interval BEFORE module_exec token.

– Modules who doesn’t execute in a “cycle” doesnt send data to server

Here is the code, I’ll add to official version after a few testing weeks. Please take a look at code if you want and tell you think about it 🙂

[GameOver]

Like it

Up

0

Down

Drop it

::

Any ideas why I can’t download the attachment?

BTW, thanks for the great application. I’ve just installed it and definitely love it.

Cheers,

Nam Nguyen

[Sancho]

Like it

Up

0

Down

Drop it

::

You can download a more updated Beta version at sourceforge, in Pandora project page at pandora.sf.net. Just look for “Download” and grab it. PLease, contact us if you have problems, questions, sugestions or you want to invite us a couple of beers :-)))

Thanks for your comments!