Kafka

The plugin displays the performance metrics of producers and consumers.

Introduction

Ver. 11-10-2022

The plugin gets metrics on producer and consumer performance

This is ported from the Java producer, for details see:

 

https://kafka.apache.org/documentation/#consumer_monitoring
https://kafka.apache.org/documentation/#producer_monitoring

Tipo: Server plug-in

Compatibility matrix

Systems where it has been tested

CentOS 7, Fedora, rocky linux

Systems where it should work

Any linux system

Pre requisites

Parameters

Parameter Description
-s, --bootstrap_servers host[:port]' string (or list of 'host[:port]' strings) that the producer must contact to start the initial cluster metadata. This does not have to be the complete list of nodes. It only needs to have at least one broker responding to a metadata API request. The default port is 9092. If no server is specified, the default port will be localhost:9092.
--tentacle_port Tentacle port Default 41121(optional)
--tentacle_address Ip of tentacle server to send data to (optional)
-a, --agent_alias To give a name to the agent that will contain the modules, by default: "Kafka" (optional).
-A, --use_alias_as_name Using the Alias name as agent name
--module_prefix To prefix the modules in case you want to differentiate them from other executions (optional).
-g,--group Pandora FMS target group (optional)
--data_dir Pandora FMS data directory. By default it is /var/spool/pandora/data_in/ (optional)
--as_agent_plugin It is optional, if you want the plugin to be an agent plugin and put the modules in the pandora agent, execute this with a 1 (optional)

Manual execution

 

./pandora_kafka --bootstrap_servers <string_conexion (ip with port)> [ --agent_alias <agent_alias> ] [ --use_alias_as_name <use_alias_as_name> ] [ --as_agent_plugin <as_agent_plugin> ] [ --module_prefix <module_prefix> ] [ --tentacle_port <tentacle_port> ] [ --tentacle_address <tentacle_address> ] [ -g <group> ] [ --data_dir <data dir> ] 

Manual execution with

--as_agent_plugin

image-1665498491460.png

Manual execution with out --as_agent_plugin

image-1665505277866.png

Help

./pandora_kafka -h

image-1665498388456.png

Configuration in PandoraFMS

As server plugin

Manual installation

Go to servers > plugins:

 

image-1629974405286.png

Click on add:

image-1629974430627.png

We put in name and the description that is preferred.

We put as command the execution with the path of the plugin:

 

Remember that the recommended path for the use of the server plugins is: /usr/share/pandora_server/util/plugin/

And in parameters of the plugin we will introduce these followed by the macro "_field<N>_", the obligatory one for it to work is --conf.Although it is not mandatory, it is highly recommended the use of the --agent_alias parameter, since it allows us to customize the name of the agent that will contain the modules created for each customized query. It is also advisable to use the --module_prefix parameter, since this will allow us to assign a prefix to the modules, which can be useful to recognize them faster and differentiate them if we create several executions for different databases.

 

Once this is done, we will click on "create".

Once this is done, the only thing left to do is to call it, so we will go to some agent's view and create an add-in module:

image-1646741530197.png

We will give it a name and in the section "plugin" we will put the one we have just configured.

Once this is done, click on create.

If the module is shown with 1, it means that it is running correctly.

 

Modules generated by the plugin

The plugin will create an agent with a customizable name with the --agent_alias parameter or with the default "Kafka" name that will contain all the modules of the execution.

image-1665505347992.png

Módules

 Producer

kafka-metrics-count

count

producer-metrics

connection-close-rate
connection-creation-rate
select-rate
io-wait-time-ns-avg
io-wait-ratio
io-time-ns-avg
io-ratio
connection-count
network-io-rate
outgoing-byte-rate
request-rate
request-size-avg
request-size-max
incoming-byte-rate
response-rate
request-latency-avg
request-latency-max
bufferpool-wait-ratio
batch-size-avg
batch-size-max
compression-rate-avg
record-queue-time-avg
record-queue-time-max
produce-throttle-time-avg
produce-throttle-time-max
record-send-rate
records-per-request-avg
byte-rate
record-retry-rate
record-error-rate
record-size-max
record-size-avg
requests-in-flight
metadata-age

producer-node-metrics.node-<node>

outgoing-byte-rate
request-rate
request-size-avg
request-size-max
incoming-byte-rate
response-rate
request-latency-avg
request-latency-max

Consumer

kafka-metrics-count

count

consumer-metrics

connection-close-rate
connection-creation-rate
select-rate
io-wait-time-ns-avg
io-wait-ratio
io-time-ns-avg
io-ratio
connection-count
network-io-rate
outgoing-byte-rate
request-rate
request-size-avg
request-size-max
incoming-byte-rate
response-rate
request-latency-avg
request-latency-max

consumer-node-metrics.node-<node>

outgoing-byte-rate
request-rate
request-size-avg
request-size-max
incoming-byte-rate
response-rate
request-latency-avg
request-latency-max

consumer-fetch-manager-metrics

fetch-size-avg
fetch-size-max
bytes-consumed-rate
records-per-request-avg
records-consumed-rate
fetch-latency-avg
fetch-latency-max
fetch-rate
records-lag-max
fetch-throttle-time-avg
fetch-throttle-time-max

consumer-coordinator-metrics

heartbeat-response-time-max
heartbeat-rate
join-time-avg
join-time-max
join-rate
sync-time-avg
sync-time-max
sync-rate
last-heartbeat-seconds-ago
commit-latency-avg
commit-latency-max
commit-rate
assigned-partitions

Otros

topics
Gets all topics that the user is authorized to view. This will always issue a remote call to the cluster to get the latest information.
bootstrap_connected
Returns True if bootstrap is connected.
subscriptions
Get the subscription to the current topic.
list_consumer_groups
List all consumer groups known to the pool.
least_loaded_node
Choose the node with the least pending requests, with fallbacks.
in_flight_request_count
Flight request count.

 

Example with some modules to illustrate how they would look in the agent

image-1665505419982.png