Plugin Nvidia GPU


Introduction

Ver. 1.0

This agent plugin monitors NVIDIA GPU statistics by shelling out to nvidia-smi. It emits Pandora FMS <module> XML blocks to stdout, which the local EndPoint ingests via the module_plugin directive.

Type: agent plugin (agent-side, single-binary)

It reports 18 metrics per GPU (Utilization, Temperature, Memory, Power, Fan, Clock, Throttling, ECC errors, Performance State, Name) plus 3 host-wide globals (GPU count, driver version, CUDA version). Module count is 18N + 3 (17N + 3 with --include-fan=false), where N is the number of GPUs.

Compatibility matrix

The plugin runs on any host with an NVIDIA GPU and the NVIDIA driver loaded. It shells out to nvidia-smi (any modern version). Cross-compiled binaries are provided for Linux (amd64, arm64) and Windows (amd64).

Prerequisites

The plugin handles missing driver gracefully: it emits GPU_Count=0 (CRITICAL) and exits 0. It does NOT require Python, Perl, or any runtime beyond the binary itself.

Configuration

The plugin is zero-config: every flag has a default and the binary works with no arguments. All configuration is done via CLI flags passed in the module_plugin directive.

CLI flags

Flag Type Default Description
--debug bool false Verbose stderr logging
--smi-path string "" Path to nvidia-smi binary. Empty = auto-detect
--include-fan bool true Emit fan-speed modules. Disable on fanless server GPUs
--max-count int 32 Soft cap on GPU count; warns to stderr if exceeded
--smi-timeout int 10 Per-invocation timeout for nvidia-smi in seconds

All output goes to stdout (XML) and stderr (debug logs, warnings). The EndPoint reads stdout only.

Manual execution

To run the plugin manually:

./pandora-nvidia-gpu

It emits one <module> block per metric directly to stdout:

<module>
    <name><![CDATA[GPU_0_Status]]></name>
    <type><![CDATA[generic_proc]]></type>
    <data><![CDATA[1]]></data>
    <module_group><![CDATA[GPU]]></module_group>
</module>
<module>
    <name><![CDATA[GPU_0_Utilization]]></name>
    <type><![CDATA[generic_data]]></type>
    <data><![CDATA[85.2]]></data>
    <unit><![CDATA[%]]></unit>
    <module_group><![CDATA[GPU]]></module_group>
</module>

Scheduled execution

The plugin runs via module_plugin in the Pandora FMS EndPoint. Add one of the following lines to /etc/pandora/pandora_agent.conf:

# Zero-config (nvidia-smi auto-detected)
module_plugin /etc/pandora/plugins/pandora-nvidia-gpu

# Fanless server GPU
module_plugin /etc/pandora/plugins/pandora-nvidia-gpu --include-fan=false

# Windows with explicit nvidia-smi path
module_plugin "C:\Program Files\pandora_agent\plugins\pandora-nvidia-gpu.exe" --smi-path "C:\Windows\System32\nvidia-smi.exe"

# Custom GPU limit
module_plugin /etc/pandora/plugins/pandora-nvidia-gpu --max-count 64

After adding the line, restart the pandora_agent daemon. The EndPoint invokes the binary on every agent cycle and attaches the returned modules to the agent's XML report.

Modules generated by the plugin

The plugin generates modules in two groups: per-GPU and global.

Per-GPU modules (18 per GPU)

Each GPU i (0-indexed) emits these modules:

Module name Type Unit Description
GPU_<i>_Status generic_proc - 1 if GPU detected, 0 if not
GPU_<i>_Utilization generic_data % GPU compute utilization (0-100)
GPU_<i>_Throttle_Active generic_data - Active throttle reason bitmask (decimal)
GPU_<i>_Throttle_Reasons generic_data_string - Comma-separated throttle reason labels
GPU_<i>_Memory_Controller_Utilization generic_data % Memory controller utilization
GPU_<i>_Memory_Used generic_data MiB Used memory
GPU_<i>_Memory_Free generic_data MiB Free memory
GPU_<i>_Memory_Total generic_data MiB Total memory
GPU_<i>_Memory_Used_Percent generic_data % Used/total * 100, rounded to 1dp
GPU_<i>_Temperature generic_data °C GPU temperature (warning 70-89, critical 90-110)
GPU_<i>_Power_Draw generic_data W Current power draw
GPU_<i>_Power_Limit generic_data W Power limit
GPU_<i>_Fan_Speed generic_data % Fan speed (omitted with --include-fan=false)
GPU_<i>_Graphics_Clock generic_data MHz Graphics clock frequency
GPU_<i>_Performance_State generic_data_string - P-state label (e.g. P0)
GPU_<i>_Performance_State_Number generic_data PNumber P-state numeric value
GPU_<i>_Critical_Errors generic_data - Uncorrected ECC error count (0 = NORMAL, 1+ = CRITICAL)
GPU_<i>_Name generic_data_string - GPU model name

Global modules (3 total)

Module name Type Description
GPU_Count generic_proc Number of GPUs detected (0 = CRITICAL)
GPU_Driver_Version generic_data_string NVIDIA driver version
GPU_CUDA_Version generic_data_string Maximum supported CUDA version

Threshold policy

Module Normal Warning Critical
Temperature [0, 69] [70, 89] [90, 110]
Memory_Used_Percent [0, 69] [70, 84] [85, 100]
Critical_Errors 0 (none) [1, 1000000]

Thresholds are hardcoded. To override, use alert templates in the Pandora FMS console.

Module groups