-
Memory Usage / Leak
Hello,
I’ve got a Community Pandora version, and I manage since now 3 months a lot of devices. Everything works fine, but when I look at the Memory usage, it’s growing, and my free memory is less and less. Even by restarting the server, I’m never been able to obtain back a normal Value.
Do you know if there is a “memory leak” in the system, as every 2 month, I must add “memory to my server”….
Thanks for your help
Christophe M.
-
18 Replies
-
-
-
Hello,
It’s the Pandora Server.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1574 root 20 0 376220 19104 5088 S 2.7 0.2 631:05.35 php
1712 root 20 0 4289172 1.5g 5756 S 2.3 20.9 1256:16 pandora_server
984 cwagent 20 0 770744 30732 5012 S 0.3 0.4 50:34.53 amazon-cloudwat
1482 mysql 20 0 4963180 2.6g 8864 S 0.3 36.2 1741:41 mysqld
4777 root 20 0 162104 2292 1592 R 0.3 0.0 0:00.05 top
1 root 20 0 43692 3616 2136 S 0.0 0.0 1:56.22 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.11 kthreadd PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1574 root 20 0 376220 19104 5088 S 2.7 0.2 631:05.35 php
1712 root 20 0 4289172 1.5g 5756 S 2.3 20.9 1256:16 pandora_server
984 cwagent 20 0 770744 30732 5012 S 0.3 0.4 50:34.53 amazon-cloudwat
1482 mysql 20 0 4963180 2.6g 8864 S 0.3 36.2 1741:41 mysqld
4777 root 20 0 162104 2292 1592 R 0.3 0.0 0:00.05 top
1 root 20 0 43692 3616 2136 S 0.0 0.0 1:56.22 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.11 kthreaddHere is the result, main part are used by the Pandora and MySQL database…
Thanks for help
-
-
-
-
-
Hi Christophe.
Threads and configuration in pandora_server.conf seems correct for a lightly loaded environment. Seeing that the memory usage is high in the mysql server and the Pandora FMS server, it seems that both processes are having problems when processing all the information.
Could you tell us the following information?
– Days stored in the database data, events and traps.
– Current size of the tables tagente_datos, tagente_datos_string, ttrap and tevento.
– Mysql optimization ( send us /etc/my.cnf ). It should contain a setting similar to this one:
https://pandorafms.com/manual/en/documentation/05_big_environments/08_optimization#mycnf
Thanks
-
Hello
I keep 45 days of data. The current size of my tables :
6826687 t_agente_datos
521323 t_agente_datos_string
1060389 ttrap
1065938 tevento
and link my my.cnf file.
Thanks for help
Christophe
-
Hi Christophe.
Seeing the data that you have stored and the time that you have been running the environment, it is possible that the database is fragmented and for that reason there may be a degradation in the reading and writing with the database.
Anyway, if you do not have problems of unknowns and queuing in the servers, it is possible that you do not have problems with the current data stored in database. As you have the community version you cannot install a second historical database reducing the data in the main one to 15 days to make it work faster and sending the rest of the data to the historical one, being able to store in it up to years. https://pandorafms.com/manual/en/documentation/05_big_environments/07_server_management?s%5B%5D=historical&s%5B%5D=database#history_database
A question, do you have monitored the exact memory consumption of the pandora_server process? Do you notice an increase in it inside the ps aux command? It would be nice to see how the graph of this module looks like…
module_begin
module_name Pandora Server memory
module_type generic_data
module_exec ps aux | grep pandora_server | grep “D” | gawk ‘{print $4}’
module_end
In the case that it is the free memory that is decreasing in the server and the available memory is kept because it starts to pass to buffer/cache memory, it is a normal behavior of the server due to the memory management of the linux systems,
Some time ago a memory leak was discovered in openssl / curl libraries. I have seen that you have enabled the webserver, do you have active web monitoring to URL with https? In the Enterprise version it is compiled into the binary and stopped happening ( webserver recently only was Enterprise), but it is possible that the openssl, curl and perl-Net-SSLeay versions may cause extra memory consumption.
You can try to uncomment the following line ( web_engine lwp ) in the pandora_server.conf to use LWP instead of curl in case you have monitoring of this type added and it can cause to these libraries.
Regards
-
Hello,
Yes, I’m in community and can’t save in the history.
The problem is more that the system is running from ore than 45 days, and the available memory is going down, and down. We have to add memory and restart the server (hosted in AWS) to get some space freed.
I’ve put the images of the memory usage (free memory and percentage of usage).
I don’t use the HTTPs verification, only SNMP messages and Cloudwatch plugin.
I see that during last days, there is a stabilisation… perhaps I’ve reached the time of purge of the 45 days (?)
Regards
Christophe
-
-
here it is
-
Ok, try to leave the server for a few days without restarting the server process to see if the memory is still. Actually the important is the available memory and that should not go down in case you do not add new monitoring or run heavier tasks in the next days…
It is possible as you say that you have reached the maximum of stored data and there is no further degradation in the system. -
-
-
-
Just a quick look, but it looks like very much better, as the top value has been reached now, and it’s stabilized since few days now…
Thanks
