-
Diferent modules intervals on same machine/agent
Hi all
I’m setting up Pandora, monitoring a system where it suits to have a single agent installed per machine.
On each machine agent, we can configure the interval at which the agent runs all the modules specified in the configuration file.
In our case, however, we wanted to do something more. Some of the installed modules do not need to send information to the server at all times the agent runs. This would ease the resources on the monitored machine (CPU usage, memory, module run time, …), preventing us from flooding the network and the server with unnecessary data packages.
An example is always useful (using default agent modules):
– Agent installed on machine ‘test’ with interval ‘300’: all modules run every 5 minutes.
– Installed modules:’ cpu_user’, ‘memfree’, ‘disk_var_free’
In this case, we would want to have CPU and memory related agents running every 5 minutes, whereas the /var disk free module only every 24h. Main idea would even be not to run the module script to save resources.Is it possible to configure Pandora in such way? Any suggestions?
Cheers
-
10 Replies
-
Not for this version, but its a feature we will implement in the next version.
I personally wanted to do for this version (1.2), but we doesnt have enought resources.If your concerns are about “database usage”, so for example, insert an entry in DB with the same data( for example boolean data), now with Pandora 1.2 and its new management of data, based on store only changes over existing data, so data without changes is not stored. This would permit more agents and more modules.
Whatever, all your ideas are very important to make Pandora more usable and useful.
-
Hi.
(Disclaimer: I am not the same Joao that started this thread, but I work with him. 😉 )
Actually our concern was not so much database usage, but resource usage on the agent’s machine.
While monitoring simple things like CPU and Memory usage does not impact machine performance much, some other modules that, for example, perform queries on a database or search for error messages on large log files can have a much bigger impact on the system, especially on servers with an already high load.
So our concern would be to run the modules only as needed, for example: CPU usage every 5 minutes, memory usage every 5 minutes, the log search every hour and the database query every day.
We have encountered only two major obstacles so far, both on the web interface:
1) The agent page only shows the information received on the last data transfer, which means that modules only appear in the listing when they send data (for the oracle query that would be once a day, and then only for 5 minutes, since the next data transfer would not include it). Due to this you have no way to check the graph for that module unless it sent data in the last transfer.
2) The graphics assume a scale equal to the minimum interval for the agent, so if our agent’s configured interval is 300 (5 minutes) and we have a module that sends data every 30 minutes, the graph would show a spike every 6 intervals but remain flat otherwise.
Obstacles we had already solved in 1.1, but haven’t tried in 1.2:
1) The agent’s module configuration now has an aditional parameter “module_interval”. A value of 1 here means the module is executed every time, a value of 2 every other time, etc.
2) The pandora_server no longer complains when it receives empty data for a module, and whenever the data is empty it also does not insert anything into the database.
-
1) The agent’s module configuration now has an aditional parameter “module_interval”. A value of 1 here means the module is executed every time, a value of 2 every other time, etc.
This could be interesting, could you email me your changes there ?.
About he another two questions:
1) The agent page only shows the information received on the last data transfer, which means that modules only appear in the listing when they send data (for the oracle query that would be once a day, and then only for 5 minutes, since the next data transfer would not include it). Due to this you have no way to check the graph for that module unless it sent data in the last transfer.
2) The graphics assume a scale equal to the minimum interval for the agent, so if our agent’s configured interval is 300 (5 minutes) and we have a module that sends data every 30 minutes, the graph would show a spike every 6 intervals but remain flat otherwise.
All of this two questions are now resolved in Pandora 1.2, if I understand well :-), so why are you waiting so long to test it ? 🙂
-
In Pandora_agent.conf, each module now has an extra parameter “module_interval”, which MUST come before the “module_exec” parameter:
module_begin module_name CPU_user module_description User CPU usage module_type generic_data module_interval 1 module_exec vmstat 2 2| tail -1 | awk '{ print $20 }' module_endIn pandora_agent.sh, we made the following changes:
Read the value of the new module configuration parameter “module_interval” and determine if we should run the module this time:
if [ ! -z "`echo $a | grep '^module_interval'`" ] then # Determine if execution is to be done interval=`echo $a | cut -c 17-` execute=`expr ( $CONTADOR - 1 ) % $interval` fiModule_exec bloc tests variable $execute that was determined in “module_interval” bloc. If 0 module is executed and value is placed in XML output, otherwise the last value returned is used.
if [ ! -z "`echo $a | grep '^module_exec'`" ] then # Check if it's necessary to execute module if [ $execute -eq 0 ] then # Execution to be done execution=`echo $a | cut -c 13-` res=`eval $execution` if [ -z "$flux_string" ] then res=`eval expr $res` fi echo "$res" >> $DATA # Save value if module won't be executed next time if [ $interval -ne 1 ] then echo $res > ${PANDORA_SAVED_FILES}${name} fi else # Set last execution value echo "`cat ${PANDORA_SAVED_FILES}${name}`" >> $DATA fi fi -
-
Good news then! 🙂 Glad to help!
Still some room for improvement I think…Done so far:
– Database insertion avoidance for equal data packages (beta 1.2)
– Module interval configuration for overload avoidance (suggested in this post)Suggestion:
– Data package transfer minimization by avoiding sending same data to the server. Besides not running module which should be an improvement by itself, not sending redundant data to the server would reduce network traffic (useful specially for big data packages / many installed modules). Server would have to be set not to receive data for a specific module as a normal situation.Just a thought for evaluation.
Cheers -
I was thinking about this problem some weeks ago, but I cannot imagine no example of “huge” data transfer between agent and server.
The problem could be to “economize” server stress, if you really send the same data, ok, database storage is not incremented but, what about database query, server XML process and CPU usage?. If an agent simply doesnt send unneeded data, huge ammount of server and database stress will be economized in a big enviroment.
I think that using your patches on agents and working a few hours its possible to make that, I need to take some spare hours of my job and work hard with that. We have 1.2 Documentation in some “freeze” state I don’t like.
Thanks for the idea. I really like to have a community of users with this kind of continuous and valuable feedback.
-
Actually I’ve done a new version of Pandora Agent 1.2 using some of your ideas:
– You can specify (optional) module_interval x where x is 1,2,3…x, 1 means exec this module on each cycle, 2 means exec this module each two cyles….
– Modules who has interval must specify module_interval BEFORE module_exec token.
– Modules who doesn’t execute in a “cycle” doesnt send data to server
Here is the code, I’ll add to official version after a few testing weeks. Please take a look at code if you want and tell you think about it 🙂
-
-
You can download a more updated Beta version at sourceforge, in Pandora project page at pandora.sf.net. Just look for “Download” and grab it. PLease, contact us if you have problems, questions, sugestions or you want to invite us a couple of beers :-)))
Thanks for your comments!
