Kafka
The plugin displays the performance metrics of producers and consumers.
- Introduction
- Compatibility matrix
- Pre requisites
- Parameters
- Manual execution
- Configuration in PandoraFMS
- Modules generated by the plugin
Introduction
Ver. 11-10-2022
The plugin gets metrics on producer and consumer performance
This is ported from the Java producer, for details see:
https://kafka.apache.org/documentation/#consumer_monitoring
https://kafka.apache.org/documentation/#producer_monitoring
Compatibility matrix
|
Systems where it has been tested
|
CentOS 7, Fedora, rocky linux |
|
Systems where it should work
|
Any linux system |
Pre requisites
- Have the Pandora FMS Data Server enabled
- Have the Pandora FMS Plugin Server enabled
Parameters
| Parameter | Description |
| -s, --bootstrap_servers | host[:port]' string (or list of 'host[:port]' strings) that the producer must contact to start the initial cluster metadata. This does not have to be the complete list of nodes. It only needs to have at least one broker responding to a metadata API request. The default port is 9092. If no server is specified, the default port will be localhost:9092. |
| --tentacle_port | Tentacle port Default 41121(optional) |
| --tentacle_address | Ip of tentacle server to send data to (optional) |
| -a, --agent_alias | To give a name to the agent that will contain the modules, by default: "Kafka" (optional). |
| -A, --use_alias_as_name | Using the Alias name as agent name |
| --module_prefix | To prefix the modules in case you want to differentiate them from other executions (optional). |
| -g,--group | Pandora FMS target group (optional) |
| --data_dir | Pandora FMS data directory. By default it is /var/spool/pandora/data_in/ (optional) |
| --as_agent_plugin | It is optional, if you want the plugin to be an agent plugin and put the modules in the pandora agent, execute this with a 1 (optional) |
Manual execution
./pandora_kafka --bootstrap_servers <string_conexion (ip with port)> [ --agent_alias <agent_alias> ] [ --use_alias_as_name <use_alias_as_name> ] [ --as_agent_plugin <as_agent_plugin> ] [ --module_prefix <module_prefix> ] [ --tentacle_port <tentacle_port> ] [ --tentacle_address <tentacle_address> ] [ -g <group> ] [ --data_dir <data dir> ] Manual execution with
--as_agent_plugin
Manual execution with out --as_agent_plugin
Help
./pandora_kafka -h
Configuration in PandoraFMS
As server plugin
Manual installation
Go to servers > plugins:
Click on add:
We put in name and the description that is preferred.
We put as command the execution with the path of the plugin:
/path_pandora_kafka
Remember that the recommended path for the use of the server plugins is: /usr/share/pandora_server/util/plugin/
And in parameters of the plugin we will introduce these followed by the macro "_field<N>_", the obligatory one for it to work is --conf.Although it is not mandatory, it is highly recommended the use of the --agent_alias parameter, since it allows us to customize the name of the agent that will contain the modules created for each customized query. It is also advisable to use the --module_prefix parameter, since this will allow us to assign a prefix to the modules, which can be useful to recognize them faster and differentiate them if we create several executions for different databases.
Once this is done, we will click on "create".
Once this is done, the only thing left to do is to call it, so we will go to some agent's view and create an add-in module:
We will give it a name and in the section "plugin" we will put the one we have just configured.
Once this is done, click on create.
If the module is shown with 1, it means that it is running correctly.
Modules generated by the plugin
The plugin will create an agent with a customizable name with the --agent_alias parameter or with the default "Kafka" name that will contain all the modules of the execution.
Módules
Producer
kafka-metrics-count
| count |
producer-metrics
| connection-close-rate |
| connection-creation-rate |
| select-rate |
| io-wait-time-ns-avg |
| io-wait-ratio |
| io-time-ns-avg |
| io-ratio |
| connection-count |
| network-io-rate |
| outgoing-byte-rate |
| request-rate |
| request-size-avg |
| request-size-max |
| incoming-byte-rate |
| response-rate |
| request-latency-avg |
| request-latency-max |
| bufferpool-wait-ratio |
| batch-size-avg |
| batch-size-max |
| compression-rate-avg |
| record-queue-time-avg |
| record-queue-time-max |
| produce-throttle-time-avg |
| produce-throttle-time-max |
| record-send-rate |
| records-per-request-avg |
| byte-rate |
| record-retry-rate |
| record-error-rate |
| record-size-max |
| record-size-avg |
| requests-in-flight |
| metadata-age |
producer-node-metrics.node-<node>
| outgoing-byte-rate |
| request-rate |
| request-size-avg |
| request-size-max |
| incoming-byte-rate |
| response-rate |
| request-latency-avg |
| request-latency-max |
Consumer
kafka-metrics-count
| count |
consumer-metrics
| connection-close-rate |
| connection-creation-rate |
| select-rate |
| io-wait-time-ns-avg |
| io-wait-ratio |
| io-time-ns-avg |
| io-ratio |
| connection-count |
| network-io-rate |
| outgoing-byte-rate |
| request-rate |
| request-size-avg |
| request-size-max |
| incoming-byte-rate |
| response-rate |
| request-latency-avg |
| request-latency-max |
consumer-node-metrics.node-<node>
| outgoing-byte-rate |
| request-rate |
| request-size-avg |
| request-size-max |
| incoming-byte-rate |
| response-rate |
| request-latency-avg |
| request-latency-max |
consumer-fetch-manager-metrics
| fetch-size-avg |
| fetch-size-max |
| bytes-consumed-rate |
| records-per-request-avg |
| records-consumed-rate |
| fetch-latency-avg |
| fetch-latency-max |
| fetch-rate |
| records-lag-max |
| fetch-throttle-time-avg |
| fetch-throttle-time-max |
consumer-coordinator-metrics
| heartbeat-response-time-max |
| heartbeat-rate |
| join-time-avg |
| join-time-max |
| join-rate |
| sync-time-avg |
| sync-time-max |
| sync-rate |
| last-heartbeat-seconds-ago |
| commit-latency-avg |
| commit-latency-max |
| commit-rate |
| assigned-partitions |
Otros
|
topics
|
Gets all topics that the user is authorized to view. This will always issue a remote call to the cluster to get the latest information.
|
|
bootstrap_connected
|
Returns True if bootstrap is connected.
|
|
subscriptions
|
Get the subscription to the current topic.
|
|
list_consumer_groups
|
List all consumer groups known to the pool.
|
|
least_loaded_node
|
Choose the node with the least pending requests, with fallbacks.
|
|
in_flight_request_count
|
Flight request count.
|
Example with some modules to illustrate how they would look in the agent
