Pandora FMS Plugin JMX

Plugin for monitoring application servers based on JMX technology

Introduction

View.

Plug-in for monitoring application servers based on JMX technology.

Type: Server plug-in

Compatibility matrix

Developed using IBM WebSphere 8.5.5.13, and Weblogic 12.1.3.0.0.0 using agent
WAR Jolokia 1.5.0

Compatible with:

Websphere
Weblogic
Tomcat
WebSphere
JBoss
Kafka

Prerequisites

It is required to deploy the Jolokia application on the application server we want to monitor.
monitor.

On the machine that will run the plugin, the jmx4perl libraries must be installed.

A series of basic checks are performed "by default", although they can be
can be suppressed or customised.

An "open" interface is available to specify queries to Mbeans,
allowing you to manage the monitoring of Java resources.

A connection to the Tentacle service associated to your Pandora FMS server is required (for local executions of the plugin) under
local executions of the plugin) under the following conditions:

● If you have as_server_plugin enabled.
If you have agent_per_instance enabled.

Deployment of this plugin by binaries does not require any special requirements.

Requires the PandoraFMS::PluginTools.pm library.

This library is available with the installation of Pandora FMS OpenSource package.

You can download the latest version from:

https://github.com/pandorafms/pandorafms/blob/develop/pandora_server/lib/PandoraFMS/PluginTools.pm

 

Configuration

JOLOKIA INSTALLATION

Download the Jolokia agent from the official website: https://jolokia.org/download.html

Jolokia Agent Securisation (WAR)

You can assign security policies to restrict both access and use of the options available in the Jolokia agent.

These restrictions can be specified in a policy XML file. This file is divided into sections with which you can control various parameters:

IP-based restrictions
General access can be allowed based on the IP address of an HTTP client. This restriction is specified in the<remote> section, it will contain <host> elements. The source can be an IP address, a hostname, or a network in CIDR format.

The following example allows access from localhost and all clients on the network
10.0.0.0/16:

<remote>
<host>localhost</host>
<host>10.0.0.0/16</host>
</remote>

Command Restrictions

You can restrict the commands that will be allowed to be executed in a general way, by specifying which commands are allowed. To do this, use the <commands> section by specifying each command with a <command> element.

The following example enables the commands READ, LIST, SEARCH and VERSION . The other commands ( WRITE, EXEC ) would not be available:

<commands>
<command>read</command>
<command>list</command>
<command>version</command>
<command>search</command>
</commands>

There are restrictions on mbeans queries and request sources. Full help is available at the following link: https://jolokia.org/reference/html/security.html

https://pandorafms.com/guides/public/link/233#bkmrk-ejemplo-de-pol%C3%ADtica-

Example security policy

This example allows connection from localhost and network 10.0.0.0.0/16. The available commands are also limited to READ,LIST,VERSION and SEARCH.

<?xml version="1.0" encoding="UTF-8"?>
<restrict>
<remote>
<host>127.0.0.1</host>
<host>localhost</host>
<host>10.0.0.0/16</host>
</remote>
<commands>
<command>read</command>
<command>list</command>
<command>version</command>
<command>search</command>
</commands>
<http>
<method>post</method>
</http>
</restrict>

Inclusion of a security policy

To repackage a policy in the WAR file of the Jolokia agent, you must follow these steps
following steps:

# Donwload war agent
$ jolokia

 

Note: You can download it manually from: http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-war/1.5.0/jolokiawar-1.5.0.war

# Download the sample policy template
# in the local directory as "jolokia-access.xml".
$ jolokia download --policy

Note : If you received errors when executing the previous steps, you can download the sample policy from http://www.jolokia.org/templates/jolokia-access.xml.

Edit the policy and place the jolokia-war-VERSION.war file and the jolokia-access.xml file in the same directory.

# Repack the to add the policy file 'jolokia-access.xml' form the cwd
$ jolokia repack --policy jolokia.war

You should receive an output similar to the following:

$ jolokia repack --policy jolokia-war-1.5.0.war
* Adding policy WEB-INF/classes/jolokia-access.xml to
jolokia-war-1.5.0.war

image-1626162151046.png

You can perform a verification of the contents of the WAR file with the following command:

# Show current state (policy file include: yes/no)
$ jolokia jolokia.war

Jolokia on WebSphere

To install the proxy on IBM Websphere. Perform the following steps:
Select from the side menu:
Applications > New Application

image-1626162310635.png

Select the Jolokia WAR file.

image-1626162342696.png

Press next .

image-1626162379452.png

Keep the default options and press next.

image-1626162412810.png

Modify the name of the application to "jolokia", no further changes are necessary on this screen. Press next.

image-1626162445456.png

Select the check box to map the new application to the desired server. Click next.

image-1626162556452.png

Select the check box to map the default virtual host. Click Next.

image-1626162605409.png

Set the root context of the application to /jolokia.
Press next.
Check the summary. Press finish.

image-1626162641824.png

Verify in the presented log that the application has been installed successfully. If so, press "Save directly to master configuration".

The application is now deployed. We will have to start it to be able to interact with it from the JMX plugin.

Navigate to the list of applications and select "Jolokia".

image-1626162679119.png

You should receive a message that the application has started successfully:

image-1626162757402.png

You can validate the installation of Jolokia at the following link:

https://localhost:9443/jolokia

You should receive an output in JSON format with general information about the Jolokia agent.

image-1626162796600.png

Jolokia on WebSphere with security enabled

To use the JMX plugin for data extraction from security-enabled IBM WebSphere environments, it will be necessary to make modifications to the Jolokia application (Jolokia file).
enabled environments, modifications to the Jolokia application (WAR file) are required.
WAR FILE).

Enable role-specific security mapping

This step adds the information for the role mapping to the jolokia.war file, which we will subsequently
later deployed on our application server.

image-1626162849849.png

IBM WebSphere Application Server Configuration

You will need to create a user with monitoring permissions. Users and groups > Manage users > Create user

image-1626162885353.png

And assign it the specific role "Monitor": Users and Groups > Manage User Roles > Add

image-1626162914432.png

Once the user has been created and the correct role has been assigned, we carry out the mapping in the Jolokia application configuration:
Applications > Company applications > Jolokia > Security role mapping to user/group.

image-1626162946991.png

Once the configuration is done, save the changes and restart the Jolokia application.

NOTE : It has been confirmed that in older versions of IBM WebSphere, there is a need to perform the user mapping by adding the ALL group to the mapping with the supervisor role
mapping of users by adding the ALL group to the mapping with supervisor role:

https://pandorafms.com/guides/public/link/233#bkmrk-los-cambios-a-realiz

The changes to be made are:

Activate "all" and "all authenticated" in the JMX - user mapping in the configuration of
Jolokia configuration:

image-1626162996442.png

image-1626163024800.png

Jolokia on Weblogic

Previous steps

To ensure full functionality of the JMX plugin in Weblogic, you should check a number of previous configurations. If you decide to skip these steps, monitoring may be limited.

Note : If your monitoring needs are limited to application listing, status, number of invocations, resource consumption and general application server status, you can ignore this section.

Access the server configuration screen at:

Base Domain > Environment > Servers

image-1626163090059.png

Press Lock and Edit to enable editing of the configuration:

image-1626163152631.png

Click on the server where the Jolokia application will be deployed:

image-1626163186857.png

Habilite IIOP

image-1626163208884.png

Press save .
Select Base Domain from the menu

image-1626163249802.png

On the Security tab, enable Anonymous Admin Lookup:

image-1626163274391.png

Deployment of Jolokia

Go to the menu section
Domain structure > Deployments

image-1626163307119.png

On the screen that appears, select the Install option.

image-1626163336700.png

Select the upload your file(s) link to display the file upload form:

image-1626163369156.png

Select the Jolokia WAR file from your computer and press next :

image-1626163396252.png

The location will be populated with the WAR file upload path. Press next
to start the installation process.

image-1626163426983.png

Select the custom role security model and press next .

image-1626163454508.png

Update the name of the deployment and keep the proposed configuration.
Press Next .

image-1626163483322.png

Click Finish .
At this step, the application is deployed on the JMX application server. In
the Configuration tab, select the entry point from where the service will be served, by configuring the
service, configuring the "root context":

image-1626163521604.png

image-1626163528992.png

Press save to save your changes.

https://pandorafms.com/guides/public/link/233#bkmrk-presione-activar-cam

Press Activate changes to apply the changes.

image-1626163568251.png

Once the changes have been applied, the Jolokia application will be displayed ready to be started.
to be started:

image-1626163607191.png

Select the checkbox and click start > Start all requests

image-1626163639102.png

After the start-up process, the application will appear as Active .

image-1626163686299.png

Edite el rol de seguridad de la aplicación para agregar las condiciones de acceso que desee (permitir grupos, usuarios, etc.)

image-1626163707893.png

image-1626163722260.png

 

You can validate the installation of Jolokia at the following link:

https://localhost:7001/jolokia

You should receive an output in JSON format with general information about the Jolokia agent.

image-1626163765493.png

Jolokia in Apache Kafka

To monitor an Apache Kafka server, the JVM agent Jolokia is required instead of the WAR agent. Just download it from the official website:

https://jolokia.org/download.html

Get the PID of the Apache Kafka server:

image-1626163812587.png

And we relate the JVM agent to the server by means of the server with

$ java –jar agente.jar start pid

You can validate the installation of Jolokia at the following link:

http://localhost:8778/jolokia/

JMX4PERL INSTALLATION

On the computer where you will run the JMX plugin you will need to install the necessary Perl libraries:

JMX::Jmx4Perl
JMX::Jmx4Perl::Alias
JMX::Jmx4Perl::Request

To install the libraries you need an internet connection and CPAN.

# cpan install JMX::Jmx4Perl

Although the following applications are extremely useful for exploring the mbean of our application servers, it is possible that they may conflict because of libraries or packages yet to be installed.

jmx4perl
check_jmx4perl
j4psh

j4psh
https://pandorafms.com/guides/public/link/233#bkmrk-en-un-entorno-de-pro

In a production environment we recommend ignoring suggestions for additional applications and installing only the JMX::Jmx4Perl library.

Do not respond to all requests to install additional components:

Install 'jmx4perl' ? (y/n) [y ]n
Install 'check_jmx4perl' ? (y/n) [y ]n
Install 'cacti_jmx4perl' ? (y/n) [y ]n
Install 'j4psh' ? (y/n) [y ]n

Manual execution

To run the plugin, configure the configuration file according to the instructions above.
instructions.

Running the plugin:

Code version without deployed libraries:

perl -I PandoraFMS pandora_plugin_jmx.pl jmx.conf

Where, PandoraFMS is a directory containing the PluginTools.pm library.

Note: If you have the PandoraFMS (opensource) libraries deployed, it is not necessary to include the -I PandoraFMS argument.

Configuration in Pandora

CONFIGURATION

The configuration file of the JMX monitoring plugin is divided into blocks:

Connection data

########################################################################
## Connection data
########################################################################
## URL endpoint of the target server
## Tomcat (example)
#server_url http://localhost:8080/jolokia
## Websphere (example)
#server_url http://localhost:49163/jolokia
## Weblogic (example)
#server_url http://192.168.70.237:7001/jolokia
## Product; product id of an specific technology
## (tomcat, jboss, weblogic, websphere, etc). By default the plugin uses autodetection
#product weblogic
## Timeout in seconds for an HTTP request (By default 180)
# timeout 180

server_url

Indicates the url address where the Jolokia application is deployed.

product

Debug

########################################################################
## Others
########################################################################
## Plugin log output will be fully verbose (with debug information)
#debug 1

debug

Set the value of this field to 1 to see detailed information in the plugin execution log. Default in /tmp/pandora_jmx_plugin.log

Scope of monitoring

########################################################################
## Monitoring checks
########################################################################
## Server info checks
server_info
## OS info checks
os_info
## Monitor deployed web applications
webapp_info
## Include also monitoring for internal applications (console, consolehelp,...)
#include_internal_webapps 0
## This list will be used later by "servlets_list_info"
## check like a white list (only the following servlets will be checked)
## Tomcat monitoring
#use_only_servlets_begin
#WebModule=//localhost/examples,name=HelloWorldExample
#WebModule=//localhost/host-manager,name=HTMLHostManager
#WebModule=//localhost/examples,name=wsSnake
#use_only_servlets_end
## Websphere monitoring
#use_only_servlets_begin
#WebModule=DefaultCell01,name=jolokia_war
#WebModule=DefaultCell01,name=Helloworld_war
#use_only_servlets_end
## Weblogic monitoring
#use_only_servlets_begin
#WebModule=WL_PROXY,name=JspServlet
#use_only_servlets_end
# Servlets info
# (by default all Servlets deployed in the App server; will use use_only_servlets list
to restrict target Servlets checked)
#servlets_list_info
## Servlet monitoring, retrieve attributes from detected servlets
#servlet_attributes startTime loadTime,servletClass,requestCount,loadOnStartup

You can configure the plugin to report the information you need:

server_info

Generate modules with JMX server information (1) or not (0).

os_info

Add OS monitoring modules (1) or not (0).

webapp_info

Monitor the applications deployed on the JMX server (1) or not (0). If you combine this parameter with agent_per_instance , as many agents will be created as applications you have deployed. This functionality has been tested with Weblogic only.
may not behave as expected with other application servers.

include_internal_webapps

Include in the list of monitored web applications, the internal applications of the JMX server (1) or not (0).

servlets_list_info

Servlet-based monitoring allows us to receive information about servlet availability. You can configure specific servlets to monitor by specifying:

use_only_servlets_begin

This will restrict servlet monitoring to the servlets specified line-by-line after this tag.

use_only_servlets_end

Indicates the end of the servlet definition.

Note: If you use servlets_list_info and do not define the use_only_servlets block, all available servlets will be parsed.

Inclusion of custom mbeans

########################################################################
## Request of specifics attributes
########################################################################
# Example 1: Get an specific parameter ("used") from "HeapMemoryUsage" attribute from
the MBean "java.lang:type=Memory"
mbean_check_begin
mbean_check_name MemoryUsed
mbean_check_type generic_data
mbean_check_module_group Module group
mbean_check_description In Bytes
mbean java.lang:type=Memory
attribute HeapMemoryUsage
path used
mbean_check_end
# Tomcat server specific checks
#include jmx.tomcat.conf
# Websphere server specific checks
#include jmx.websphere.conf
# Weblogic server specific checks
#include jmx.weblogic.conf
# Kafka server specific checks
#include jmx.kafka.conf

You can define your own checks to retrieve specific information from mbeans. To do so, define as many check blocks as you need:

mbean_check_begin

Specifies the start of a specific mbean check definition block.

mbean_check_name

Is the module name as it will appear in Pandora FMS.

mbean_check_type

It indicates the type of data that will be assigned to the module in Pandora FMS.

mbean_check_module_group

You can customize the module group that will be applied to the monitors generated by the mbean data extraction.

mbean_check_description

It is the module description as it will appear in Pandora FMS.

mbean

Configure here the mbean to query.

path

Indicate the path where the mbean you want to check is hosted.

mbean_check_wmin

Specify the lower limit of the Warning threshold for the module.

mbean_check_wmax

Specifies the upper limit of the Warning threshold for the module.

mbean_check_wstr

Specifies the text as the Warning threshold for modules of type string.

mbean_check_winv

Reverses the direction of the Warning thresholds for the module.

mbean_check_cmin

Specifies the lower limit of the Critical threshold for the module.

mbean_check_cmax

Indicates the upper limit of the Critical threshold for the module.

mbean_check_cstr

Specifies the text as the Critical threshold for modules of type string.

mbean_check_cinv

Reverses the direction of Critical thresholds for the module.

mbean_check_end

Specifies the end of a specific mbean check definition block.

 

https://pandorafms.com/guides/public/link/236#bkmrk-macro-de-detecci%C3%B3n-d

mbean _jokeritem_ dynamic detection macro
In order to monitor complex mbeans, you can use the _jokeritem_ macro in such a way that as many modules as there are mbeans are generated.
macro in such a way that as many modules are generated as data the mbean provides:

Example configuration.

mbean_check_begin
mbean_check_name jms.messagescurrentcount._jokeritem_
mbean_check_module_group jvm.jms
mbean_check_type generic_data
mbean_check_description The current number of messages stored on this JMS server. This
number does not include the pending messages.
mbean com.bea:Name=_jokeritem_,ServerRuntime=_serverruntime_,Type=JMSServerRuntime
attribute MessagesCurrentCount
mbean_check_end

Macro for server runtime selection
It is now possible to select the target runtime server dynamically using the _serverruntime_ macro.

Example configuration:

mbean
com.bea:Name=ThreadPoolRuntime,ServerRuntime=_serverruntime_,Type=ThreadPoolRuntime

include
You can specify as many extra configuration files as you need.

Data delivery configuration

########################################################################
## Request of all list of JMX information
## (Warning!!! this output is very very verbose, so use this check carefully)
########################################################################
#list_all_mbeans
## Report style
as_agent_plugin 0
agent_per_instance 1
## Custom header
module_header jmx_check_
## agent configuration
#agent_name custom_agent_name
#agent_interval 300
#agent_group JMX
## Pandora Server configuration
transfer_mode tentacle
tentacle_ip your.pandora.server.ip
tentacle_port 41121
tentacle_opts
tentacle_client "tentacle_client"
temp /tmp
local_folder /var/spool/pandora/data_in

You can configure the data delivery in Pandora FMS in several ways:

list_all_mbeans

It will generate a module with a list in text with all the available mbeans.

as_agent_plugin

If it is active (1), it will dump all the module information by screen in XML format to be executed from an agent.
XML format to be executed from an agent. If not active (0), it will deliver the information by
transferring complete XML to the server configured as destination.

agent_per_instance

If it is active (1), it will send the information of instances as individual agents to
Pandora FMS.

If the monitoring is based exclusively on mbeans, they will be grouped per
servlet.

● If the monitoring is based on web applications (webapp_info), as many agents will be created as many
agents will be created as many as the number of web applications deployed.

● An agent representing the application server itself will always be created.
If disabled (0), all module information will be sent in one single agent (application server name or
agent (application server name or the one specified in agent_name ).

module_header

Specifies a custom header for the modules generated by the plugin.

mode

XML file transfer mode, it can be:

● local: copies files to local_folder.

● tentacle: transfer files to tentacle_ip via tentacle_port

tentacle_ip

IP address or FQDN where Pandora FMS is working.

tentacle_port

Port where the Tentacle service associated to your Pandora FMS server is listening.
listening.

tentacle_opts

Extra options for the Tentacle client.

tentacle_client

Path where to find the Tentacle client binary.

local_folder

Location to move files to in local mode.

temp

Temporary location to store files before sending them.

Modules generated by the plugin

The standard execution of this plugin will return the following modules by default (with
all optional blocks enabled):

Classic monitoring:

 

Tomcat :

Active Sessions
Max Sessions Allowed
Configuración connector SSL
Connector State
JSP Count

 

Class Load Time
Servlet count allocated
Servlet load tim
Servlet class
Servlet request count
Servlet load on startup
Webmodule configured
Webmodule start time
Webmodule request count
Webmodule state name


Websphere :

Active sessions
Invalid Sessions
Create count
Close count
Allocate count
Fault count
Percent used
Uso medio de CPU por componente web
Cache size
Directorio home de Websphere
Módulos j2EE de un cierto componente web
Prevent Jre Memory Leaks
Detect App CLLeaks
Bootstrap port
ORB Request timeout
ORB Request Retries Count
Nivel de traza
Tamaño del buffer de trazas
Tamaño máximo del pool de hilos
Tamaño mínimo del pool de hilos
Inactivity Timeout




Weblogic :

Locked Users Current Count
Lockout Duration
Lockout Enabled
Invalid Login Attempts Total Count
Pool Max Capacity
Execution Time Average
Invocation Total Count
Diagnostics Read Count
Diagnostics Delete Count
Diagnostics Update Count
Diagnostics Create Count
Mbean Completed Requests
Mbean Bytes Received Count
Mbean Bytes Sent Count
Mbean Connections Count
Mbean Public URL
Mbean Messages Sent Count
Mbean Messages Received Count
Mbean Execution Time Average
Mbean Execution Time Total
Mbean Invocation Total Count
Servlet State
Servlet Server Startup Time
Servlet Execution Time Average
Servlet Reload Total Count
Servlet Execution Time High
Servlet Invocation Total Count
Webmodule status
Webmodule Open Sessions High Count
Webmodule Session Invalidation Interval Secs
Webmodule Sessions Opened Total Count
Webmodule Module Id

Kafka :

Active Controller Brokers
Broker leaders
Incoming Bytes per second
Incoming Messages per second
ISR expansion Rate
ISR shrink rate
Network processors avg idle time
Number of disputed leader elections rate
Offline Controller Brokers
Outgoing Bytes per second
Request handlers avg idle time
Requests in fetch API purgatory
Requests in producer API purgatory
System’s partitions
Total time to serve a specified request (fetch API consumer)
Total time to serve a specified request (fetch API follower)
Total time to serve a specified request
(producer API)
Unclear leader election rate
Under-replicated partitions

 

Example of web application monitoring

This specific monitoring is only compatible with Weblogic in version 723
of the plugin.

With the following basic configuration:

## Weblogic
server_url http://10.1.2.3:7001/jolokia
## product
product weblogic
# checks
server_info
os_info
webapp_info
include_internal_webapps 0
# Specific beans
mbean_check_begin
mbean_check_name MemoryUsed
mbean_check_type generic_data
mbean_check_description In Bytes
mbean java.lang:type=Memory
attribute HeapMemoryUsage
path used
mbean_check_end
## Report style
as_agent_plugin 0
agent_per_instance 1
## Agent personalization
agent_name my JMX appserver
agent_interval 300
agent_group JMX
## Pandora Server configuration
transfer_mode tentacle
tentacle_ip your.pandora.server.ip
tentacle_port 41121
tentacle_opts
tentacle_client "tentacle_client"
temp /tmp
local_folder /var/spool/pandora/data_in

This configuration will return the following modules (remember that an agent will be created for each web application you have deployed on your application server).

List of modules

Modules assigned to the main agent (represents the application server itself)

check_jmx_classes_total
check_jmx_threads_current
check_jmx_threads_peak
check_jmx_runtime_name
check_jmx_jvm
check_jmx_jvm_startup
check_jmx_max_file_desc
check_jmx_file_desc_open
check_jmx_memory_used
check_jmx_MemoryUsed

Modules assigned to each of the agents that represent the web applications
deployed:

APP_NAME State
APP_NAME InvocationTotalCount
APP_NAME ReloadTotalCount
APP_NAME Executiontimetotal
APP_NAME ExecutionTimeAverage
APP_NAME Status