Kafka

• Introduction
• Compatibility matrix
• Pre requisites
• Parameters
• Manual execution
• Configuration in PandoraFMS
• Modules generated by the plugin

Introduction

Ver. 11-10-2022

The plugin gets metrics on producer and consumer performance

This is ported from the Java producer, for details see:

https://kafka.apache.org/documentation/#consumer_monitoring
https://kafka.apache.org/documentation/#producer_monitoring

Compatibility matrix

Pre requisites

• Have the Pandora FMS Data Server enabled
• Have the Pandora FMS Plugin Server enabled

Parameters

Manual execution

./pandora_kafka\
  --bootstrap_servers <ip:port[,ip:port,...]> \
  [ --topics <topic1,topic2,...> ] \
  [ --topic_regex <regex_pattern> ] \
  [ --agent_prefix <prefix> ] \
  [ --module_prefix <prefix> ] \
  [ --transfer_mode <tentacle|local> ] \
  [ --tentacle_ip <ip> ] \
  [ --tentacle_port <port> ] \
  [ --verbose ]

For example:

./pandora_kafka \
  --bootstrap_servers kafka1:9092,kafka2:9092 \
  --topic_regex "^prod-.*" \
  --agent_prefix KAFKA_ \
  --module_prefix METRIC_ \
  --transfer_mode tentacle \
  --tentacle_ip 10.0.0.20 \
  --tentacle_port 41121 \
  --verbose

Configuration in PandoraFMS

As server plugin

Manual installation

Go to servers > plugins:

Click on add:

We put in name and the description that is preferred.

We put as command the execution with the path of the plugin:

/path_pandora_kafka

Remember that the recommended path for the use of the server plugins is: /usr/share/pandora_server/util/plugin/

And in parameters of the plugin we will introduce these followed by the macro "_field<N>_", the obligatory one for it to work is --conf.Although it is not mandatory, it is highly recommended the use of the --agent_alias parameter, since it allows us to customize the name of the agent that will contain the modules created for each customized query. It is also advisable to use the --module_prefix parameter, since this will allow us to assign a prefix to the modules, which can be useful to recognize them faster and differentiate them if we create several executions for different databases.

Once this is done, we will click on "create".

Once this is done, the only thing left to do is to call it, so we will go to some agent's view and create an add-in module:

We will give it a name and in the section "plugin" we will put the one we have just configured.

Once this is done, click on create.

If the module is shown with 1, it means that it is running correctly.

Modules generated by the plugin

The plugin will create an agent with a customizable name with the --agent_alias parameter or with the default "Kafka" name that will contain all the modules of the execution.

Módules

 Producer

kafka-metrics-count

producer-metrics

producer-node-metrics.node-<node>

Consumer

kafka-metrics-count

consumer-metrics

consumer-node-metrics.node-<node>

consumer-fetch-manager-metrics

consumer-coordinator-metrics