# Kafka

# Introduction

**Plugin version:** 1.1

The Kafka plugin retrieves the internal metrics of a Kafka producer and consumer using the `kafka-python` library. It creates one Pandora FMS agent for each specified broker and sends its metrics in XML format.

The collected metrics are based on Kafka client metrics. For more information, see the official documentation:

- [Consumer monitoring](https://kafka.apache.org/documentation/#consumer_monitoring)
- [Producer monitoring](https://kafka.apache.org/documentation/#producer_monitoring)

**Type:** Server plugin.

# Compatibility matrix

<table border="1" id="bkmrk-sistemas-donde-se-ha" style="width: 642px;"><tbody><tr><td style="width: 234px;"><div>**<span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span class="ryNqvb">Systems where it has been tested</span></span></span>**</div></td><td style="width: 408px;">Rocky linux

</td></tr><tr><td style="width: 234px;"><div>**<span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span class="ryNqvb">Systems where it should work</span></span></span>**</div></td><td style="width: 408px;">Any linux system

</td></tr></tbody></table>

# Pre requisites

- The Pandora FMS **Data Server** and **Plugin Server** must be enabled.
- The Pandora FMS server must have network connectivity to every configured Kafka broker.
- When running the Python source directly, install the dependencies listed in the `.req` file: `pandoraPlugintools-basic` and `kafka-python==2.2.20`.
- The plugin uses the default `kafka-python` configuration. This version does not include parameters for configuring SASL authentication or SSL/TLS connections.

# Parameters

<table id="bkmrk-parameter-descriptio"><thead><tr><th>Parameter</th><th>Description</th><th>Default</th></tr></thead><tbody><tr><td>`-h`, `--help`</td><td>Displays the help text, plugin version, and execution example.</td><td>—</td></tr><tr><td>`--bootstrap_servers`</td><td>One or more comma-separated Kafka brokers in `host:port` format. One agent is created for each specified value.</td><td>Required</td></tr><tr><td>`--topics`</td><td>Comma-separated list of topics to which the consumer subscribes.</td><td>Empty string</td></tr><tr><td>`--topic_regex`</td><td>Regular expression used to subscribe to topics. When specified, it takes precedence over `--topics`.</td><td>Empty string</td></tr><tr><td>`--agent_prefix`</td><td>Prefix added to the name and alias of each generated agent.</td><td>Empty string</td></tr><tr><td>`--module_prefix`</td><td>Prefix added to the name of every generated module.</td><td>Empty string</td></tr><tr><td>`--transfer_mode`</td><td>Transfer mode used by `pandoraPlugintools` to send the XML files.</td><td>`tentacle`</td></tr><tr><td>`--tentacle_ip`</td><td>Tentacle server IP address.</td><td>`127.0.0.1`</td></tr><tr><td>`--tentacle_port`</td><td>Tentacle server port.</td><td>`41121`</td></tr><tr><td>`--verbose`</td><td>Displays error and debug messages during execution.</td><td>Disabled</td></tr></tbody></table>

# Manual execution

Use the following syntax:

```bash
./pandora_kafka \
  --bootstrap_servers <host:port[,host:port,...]> \
  [--topics <topic1,topic2,...>] \
  [--topic_regex <regex>] \
  [--agent_prefix <prefix>] \
  [--module_prefix <prefix>] \
  [--transfer_mode <mode>] \
  [--tentacle_ip <ip>] \
  [--tentacle_port <port>] \
  [--verbose]

```

Example using a regular expression to select topics:

```bash
./pandora_kafka \
  --bootstrap_servers kafka1:9092,kafka2:9092 \
  --topic_regex '^prod-.*' \
  --agent_prefix 'KAFKA_' \
  --module_prefix 'METRIC_' \
  --transfer_mode tentacle \
  --tentacle_ip 10.0.0.20 \
  --tentacle_port 41121 \
  --verbose

```

A specific list of topics can also be selected:

```bash
./pandora_kafka \
  --bootstrap_servers kafka1:9092 \
  --topics orders,payments

```

If both `--topics` and `--topic_regex` are specified, the plugin uses `--topic_regex`.

When the execution finishes, the plugin prints `1` if no error occurred while processing the brokers and `0` if any broker raised an exception. The `--verbose` option displays details about the detected errors.

# Configuration in PandoraFMS

### As a server plugin

Go to **Servers &gt; Plugins** in the Pandora FMS console and click **Add**.

Set the desired name and description. In the command field, enter the full path to the executable:

```text
/usr/share/pandora_server/util/plugin/pandora_kafka

```

The recommended directory for server plugins is:

```text
/usr/share/pandora_server/util/plugin/

```

In the plugin parameters, include `--bootstrap_servers` as a required field and add any other arguments you need using `_field<N>_` macros.

Example:

```text
--bootstrap_servers _field1_ --topics _field2_ --agent_prefix _field3_ --module_prefix _field4_ --transfer_mode tentacle --tentacle_ip 127.0.0.1 --tentacle_port 41121

```

After creating the server plugin, open an agent, create a plugin module, and select the configured plugin. This module receives the final `1` or `0` returned by the execution, while the Kafka metrics are sent through separate XML agents.

# Modules generated by the plugin

### Agents

The plugin creates one agent for each entry in `--bootstrap_servers`.

The agent name and alias use the following format:

```text
<agent_prefix><host:port>

```

For example, with `--agent_prefix KAFKA_` and the `kafka1:9092` broker, the generated agent is:

```text
KAFKA_kafka1:9092

```

Agents are created with a 300-second interval and the `Created by plugin` description.

### Modules

The plugin iterates over all metrics returned by `producer.metrics()` and `consumer.metrics()`. Therefore, the exact list may vary depending on the `kafka-python` version, Kafka version, available nodes, and client activity.

All modules use the `generic_data` type and have no unit or description. Module names use the following format:

```text
<module_prefix><metric_group>.<metric_name>

```

Floating-point values that are `NaN` or infinite are converted to `0.0`.

The main generated metric groups are listed below:

<table id="bkmrk-source-metric-group-"><thead><tr><th>Source</th><th>Metric group</th><th>Module examples without a prefix</th><th>Type</th></tr></thead><tbody><tr><td>Producer and consumer</td><td>`kafka-metrics`</td><td>`kafka-metrics.count`</td><td>`generic_data`</td></tr><tr><td>Producer</td><td>`producer-metrics`</td><td>`producer-metrics.connection-count`, `producer-metrics.request-rate`, `producer-metrics.request-latency-avg`, `producer-metrics.record-send-rate`, `producer-metrics.record-error-rate`</td><td>`generic_data`</td></tr><tr><td>Producer</td><td>`producer-node-metrics.node-<node>`</td><td>`producer-node-metrics.node-<node>.outgoing-byte-rate`, `producer-node-metrics.node-<node>.request-rate`, `producer-node-metrics.node-<node>.response-rate`</td><td>`generic_data`</td></tr><tr><td>Consumer</td><td>`consumer-metrics`</td><td>`consumer-metrics.connection-count`, `consumer-metrics.request-rate`, `consumer-metrics.request-latency-avg`, `consumer-metrics.incoming-byte-rate`</td><td>`generic_data`</td></tr><tr><td>Consumer</td><td>`consumer-node-metrics.node-<node>`</td><td>`consumer-node-metrics.node-<node>.incoming-byte-rate`, `consumer-node-metrics.node-<node>.request-rate`, `consumer-node-metrics.node-<node>.response-rate`</td><td>`generic_data`</td></tr><tr><td>Consumer</td><td>`consumer-fetch-manager-metrics`</td><td>`consumer-fetch-manager-metrics.fetch-size-avg`, `consumer-fetch-manager-metrics.bytes-consumed-rate`, `consumer-fetch-manager-metrics.records-consumed-rate`, `consumer-fetch-manager-metrics.records-lag-max`</td><td>`generic_data`</td></tr><tr><td>Consumer</td><td>`consumer-coordinator-metrics`</td><td>`consumer-coordinator-metrics.heartbeat-rate`, `consumer-coordinator-metrics.join-rate`, `consumer-coordinator-metrics.commit-rate`, `consumer-coordinator-metrics.assigned-partitions`</td><td>`generic_data`</td></tr></tbody></table>

For example, when `--module_prefix METRIC_` is used, the `producer-metrics.request-rate` module is created as:

```text
METRIC_producer-metrics.request-rate

```