[et_pb_section fb_built=”1″ admin_label=”Section” _builder_version=”4.22.0″ _module_preset=”default” custom_margin=”0px||0px||false|false” custom_padding=”0px||0px||false|false” locked=”off” global_colors_info=”{}”][et_pb_row column_structure=”1_4,3_4″ _builder_version=”4.27.0″ _module_preset=”default” custom_padding=”50px||||false|false” custom_css_main_element=”z-index:0!important;” global_colors_info=”{}”][et_pb_column type=”1_4″ disabled_on=”on|on|off” _builder_version=”4.22.0″ _module_preset=”default” custom_padding=”||||false|false” sticky_position=”top” sticky_offset_top=”100px” sticky_limit_bottom=”section” motion_trigger_start=”top” global_colors_info=”{}”][et_pb_text admin_label=”indice” _builder_version=”4.27.6″ _module_preset=”default” custom_margin=”||0px||false|false” custom_padding=”||14px||false|false” link_option_url=”#1″ global_colors_info=”{}”]<\/p>\n
Sections<\/strong><\/p>\n [\/et_pb_text][\/et_pb_column][et_pb_column type=”3_4″ _builder_version=”4.27.0″ _module_preset=”default” custom_css_main_element=”z-index:0!important;” global_colors_info=”{}”][et_pb_text admin_label=”seccion” module_id=”1″ module_class=”ittopicscontent” _builder_version=”4.27.6″ _module_preset=”default” z_index=”0″ custom_margin=”0px||0px||true|false” custom_padding=”0px||0px||false|false” hover_enabled=”0″ custom_css_main_element=”font-family:%22Pandora-Light%22;” locked=”off” global_colors_info=”{}” sticky_enabled=”0″]In the good old days, as some would say, when you did not need a mortgage to buy one, the GPU (Graphics Processing Unit) was that component that only mattered to those who wanted to play Crysis at full resolution. Today, it is the antimatter reactor powering half the industry<\/strong>: AI model training, inference, machine learning, simulation, rendering, data analysis and HPC (High-Performance Computing). Without GPUs there is no AI<\/strong>, and without AI, according to the visionaries (who sell it), there is no future. One that, according to those same people, may end up destroyed anyway by that very artificial intelligence. GPU monitoring is the continuous supervision of the health, utilization, memory, temperature, power consumption, availability, and errors of your Graphics Processing Units. Documented like that, it sounds like the control panel of a starship… and we aren’t far off. GPUs are used to accelerate an organization\u2019s critical operations, but since nothing in life comes for free, they also introduce new operational risks. This makes them both the engine of any AI project and its Achilles\u2019 heel. The conclusion is that leaving the GPU out of the monitoring family picture is madness in these times we are living through.<\/p>\n There is a widespread belief that AI lives in the cloud, in a data center managed by someone else and devastating an ecosystem far removed from our own. However, many organizations operate GPUs in their own data centers, in hybrid environments, research centers, industrial settings, cloud instances with GPU… We need to keep an eye on things, fine, but what exactly? A couple of nuances, because they hide the details that will save us from unpleasant surprises. We have already seen the what, now it is time to see how we monitor<\/strong> and also to differentiate between tools, because they are easily confused. Leaving the engine unattended has disastrous consequences, such as:<\/p>\n It is the plot of so many disaster movies: the signs were there, blinking on some panel, but nobody was looking at them.<\/p>\n Isolated GPU monitoring is a thermometer that, with the operational correlation derived from integrating it into global monitoring<\/strong>, becomes diagnosis and operational knowledge. In technology, you are either at the cutting edge or you are not really there. In addition, the information it issues is critical and necessary for the optimal management of an asset more expensive than adamantium. Pandora FMS is here to prevent disaster. To do so, it lifts every last rug and connects the dots to give you something as close as possible to Atreides prescience in Dune. Hence its threshold and alert system. Let\u2019s remember that this critical error makes sense on data center cards with ECC, where any uncorrected error must be treated as a priority, with no middle ground. On consumer GPUs, this does not apply. \u201cPromises that mean nothing…\u201d, as the song says. At Pandora FMS, we are not fond of them, so to avoid making claims that later do not hold up, it is worth being clear about what the plugin does and does not do<\/Strong>:<\/p>\n And one operational nuance. The plugin is designed for individual hosts or medium-sized environments<\/strong>. Considering the number of firstborns we must sacrifice to the Machine God to afford a GPU, everything above is more than justified<\/Strong>. Either we monitor, or the most critical point becomes a blind spot. People usually talk about GPUs because, these days, they have climbed to the top of many IT infrastructures, but they do not play the game alone<\/Strong>. We have covered a lot, but let\u2019s focus on three ideas to take home.<\/p>\n NVIDIA tools provide the reading, but a command bridge is needed to interpret it<\/Strong>. Let\u2019s compile the most important questions about GPU monitoring and their answers.<\/p>\n GPU monitoring is the continuous supervision of the status, utilization, memory, temperature, power consumption and errors of Graphics Processing Units in professional environments<\/Strong>. Because GPUs are what keep the heart beating and, moreover, very expensive and critical resources<\/Strong> for inference, training and machine learning workloads. The main ones are utilization, used, free and total memory, percentage of memory used, temperature, instant power consumption, power limit, fan speed, status, critical ECC errors, GPU model, driver version and CUDA version. nvidia-smi<\/span> is a command-line utility for querying specific metrics from NVIDIA GPUs. Based on our experience embedded in the Pandora FMS GPU monitoring plugin, a temperature from 70 \u00b0C is considered warning and from 90 \u00b0C, critical<\/Strong>. ECC (Error-Correcting Code) errors are memory errors detected in GPUs with ECC support<\/Strong>. Yes. Pandora FMS has a local agent plugin based on nvidia-smi<\/span> that allows NVIDIA GPUs to be monitored on on-premise servers<\/Strong>. Historical utilization, memory, temperature and power consumption data makes it possible to identify GPUs that are recurrently saturated, detect underutilization and redistribute<\/Strong> workloads. If nvidia-smi<\/span> is not available, the Pandora FMS plugin issues the GPU_Count<\/span> module with a value of 0 in critical status. No<\/Strong>. It is part of a broader whole. Habla con el equipo de ventas, pide presupuesto, o resuelve tus dudas sobre nuestras licencias<\/p>\n ” content_phone=”<\/p>\n Habla con el equipo de ventas, resuelve tus dudas sobre nuestras licencias o pide presupuesto<\/p>\n ” content_last_edited=”on|tablet” _builder_version=”4.22.0″ _module_preset=”default” header_2_font_size=”2em” text_orientation=”center” module_alignment=”left” custom_margin=”0px||20px||false|false” custom_padding=”0px||0px||true|false” global_colors_info=”{}”]<\/p>\n Habla con el equipo de ventas, pide presupuesto, [\/et_pb_text][et_pb_code _builder_version=”4.27.4″ _module_preset=”default” width=”200px” module_alignment=”center” custom_margin=”40px||0px||false|false” custom_padding=”0px||0px||true|false” locked=”off” global_colors_info=”{}”]<\/p>\n\n
\nBut in many cases, the diamond heart of the infrastructure remains unsupervised<\/strong>.
\nServers, CPUs, networks, services… are monitored with a hawk-like intensity… o closely scrutinized, but GPUs remain the most expensive blind spot<\/strong>, one that, if left unattended, can turn into can turn into a massive financial drain.
\nThat is why the real challenge is not just measuring the GPU, but integrating it into the monitoring of on-premise, hybrid and cloud infrastructures<\/strong>.
\nHere we will see how.<\/p>\nWhat is GPU monitoring?<\/h2>\n
\nIt is obvious given the introduction, but to be clear, we are not talking about overclocking or squeezing out extra frames here. We are talking about monitoring critical resources used by AI workloads, HPC, or intensive processing within a professional infrastructure monitoring<\/a> strategy.<\/p>\nWhy GPU monitoring matters in AI infrastructures<\/h2>\n
\nWhy?<\/p>\n\n
The real gap: GPU monitoring for on-premise, hybrid and enterprise environments<\/h2>\n
\nAnd many of these organizations operate under confidentiality, data sovereignty or data legislation requirements<\/strong> where a 100% cloud solution is not viable, under penalty of fines or the risk of dangerous leaks.
\nIn these cases, it is not enough to install Ollama or similar tools and forget about it, nor is the current documentation from a cloud provider enough to operate with confidence and efficiency.
\nWe need a platform capable of watching over everything, GPUs and the rest of the infrastructure<\/strong> as a whole, whether on bare-metal servers<\/a> or virtual environments<\/A>.
\nThe never-ending debate between on-premise and SaaS<\/a>, or the requirements of hyperconverged infrastructures<\/a>, reinforces our need for control. Because a GPU may live on-premise, in a hybrid ecosystem or in a cloud instance, but its impact affects services, network, processes, APIs, applications<\/strong>….
\nHence the need to get everyone a little closer together so the GPU fits into that family picture and include it in global monitoring<\/strong>, not as a stray metric in a tab nobody opens.
\nAnd speaking of them…<\/p>\nKey metrics in GPU monitoring<\/h2>\n
\nThe metrics of any GPU monitoring strategy<\/strong> worthy of the name are, in addition to their number and names:<\/p>\n\n
\nCritical errors are not a universal metric<\/strong>. In a data center with ECC support, such as A100, H100, Tesla or RTX PRO, they are relevant, but on consumer-grade cards, such as GeForce RTX or similar models, NVIDIA\u2019s command-line utility nvidia-smi<\/span> may return N\/A for ECC errors.
\nTherefore, our Pandora FMS plugin, for example, will output 0 by design. That eternal zero does not mean the card is immortal, it simply means there is nothing to count.
\nOn the other hand, fan speed only makes sense on cards with active cooling<\/strong> (D\u2019oh, as Homer Simpson would say). On server GPUs or fanless environments, it would not apply, of course.
\nThat is why a monitoring plugin like ours allows that module to be skipped with –include-fan=false<\/span>, since advanced telemetry<\/a> is also about knowing what not to measure, so as not to flood everything with noise.<\/p>\nnvidia-smi, DCGM and monitoring platforms: what each one brings<\/h2>\n
\nLet\u2019s analyze what each one brings.
\nOn the one hand, we have nvidia-smi<\/A><\/span>, which has already shown its face and is the command-line utility that allows you to query information from NVIDIA GPUs<\/strong>. We are talking about utilization, memory, temperature, power consumption, processes, driver, CUDA<\/A>\u2026
\nThis is the GPU tricorder: you point it, read a figure and move on. Perfect for one-off diagnostics or the occasional late-night script<\/strong>.
\nNVIDIA DCGM<\/a> is aimed at GPU management in data centers, clusters and advanced deployments<\/strong>. Here we already have broader sensors for our spaceship, because it is useful when there are many GPUs or an integration with observability stacks.
\nBoth are good tools, but the truth is that neither nvidia-smi<\/span> nor DCGM will solve the entire IT operation for us<\/strong>.
\nThey do not centralize the infrastructure, they do not replace corporate dashboards, they do not provide business-oriented historical reporting and they do not correlate GPU with CPU, RAM, disk, network, logs, services or SLA on their own, for example.
\nFor these reasons, proposals such as Datadog<\/a> insist on connecting the GPU with the rest of the AI workload.
\nOr let me put it another way to make it clear. nvidia-smi<\/span> and DCGM can be excellent data sources, but floating alone in a vacuum<\/strong>.
\nIt is a platform such as Pandora FMS that turns them into operational, actionable knowledge<\/strong> with history, alerts, dashboards, reports, correlation and capacity decisions.
\nThe tricorder gives you a reading; an Enterprise command bridge like Pandora FMS tells you what it means in the context of everything else when it integrates that reading.
\nAnd in a crisis, you want to be on the bridge, not decoding loose data in the dark.<\/p>\nThe risks of not monitoring GPUs<\/h2>\n
\n
From isolated metrics to operational correlation<\/h2>\n
\nKnowing that a GPU is at 95% does not say much, just as fever alone does not reveal its cause. What matters is context<\/strong>.
\nFollowing the example, a GPU at 95% for ten minutes may be normal, but the same GPU at 95% for four hours, with memory above 85%, errors appearing in the logs and the latency of an inference service rising, is an operational incident<\/strong>.
\nAnother example is temperature. A high reading together with a fan running at warp 9 and performance through the floor may anticipate a physical failure, a breach in the wall that will bring down the castle if ignored.
\nConnecting those signals is what distinguishes a tool that looks from a platform that understands.<\/strong>
\nThat is the substance of root cause analysis<\/a>, the daily work of any Network Operations Center<\/a> or SRE<\/a> team that takes IT system monitoring<\/a> seriously.<\/p>\nHow Pandora FMS helps with GPU monitoring<\/h2>\n
\nThat is the philosophy at Pandora FMS<\/a>, and that is why the application allows GPU metrics to be integrated<\/strong> into the same console where servers, services, networks, storage, logs, events and availability are already monitored.
\nThis means GPU monitoring is not a footnote, but a key part of machinery optimized for the monitoring and observability that a professional IT infrastructure needs<\/strong>.
\nTo achieve this, Pandora FMS uses a specialist GPU monitoring plugin<\/strong>.
\nIt relies on nvidia-smi<\/Span> and works as a local agent plugin on the host with an NVIDIA GPU, issuing modules that Pandora FMS incorporates into the agent context.
\nThus, these metrics can:<\/p>\n\n
\nFor each detected GPU, it issues, among others: GPU_<i>_Status<\/span> (1 = healthy, 0 = degraded), GPU_<i>_Utilization<\/span>, GPU_<i>_Memory_Used<\/span>, GPU_<i>_Memory_Free<\/span>, GPU_<i>_Memory_Total<\/Span> (in MiB, mebibytes, not bytes), GPU_<i>_Memory_Used_Percent<\/Span>, GPU_<i>_Temperature<\/Span> (\u00b0C), GPU_<i>_Power_Draw<\/span> and GPU_<i>_Power_Limit<\/span> (W), GPU_<i>_Fan_Speed<\/Span> (optional), GPU_<i>_Critical_Errors<\/span> and GPU_<i>_Name<\/Span>.
\nAt host scale, it adds three global ones: GPU_Count<\/span>, GPU_Driver_Version<\/span> and GPU_CUDA_Version<\/span>.
\nThe total count comes from a simple formula: 12N + 3<\/strong><\/span> modules with fan speed, or 11N + 3<\/strong><\/span> without it, where N is the number of GPUs on the host.
\nAnd one detail reveals a lot about the design. If nvidia-smi<\/Span> is not available, no need to worry, the plugin does not break or produce invalid XML<\/strong>; instead, it issues GPU_Count = 0<\/span> in critical status.
\nThe issue remains visible as a clean alert, not as that suspicious silence that may be brewing disaster.
\nOur goal, because we have experienced firsthand the costly chaos of fragmentation, is that Pandora FMS allows all these signals to be analyzed on the same platform<\/strong>, with integrated alerts or our metaconsole, for example, to have a single point of visibility and control, instead of juggling between commands, logs and scattered dashboards.<\/p>\nThresholds, alerts and critical statuses<\/h2>\n
\nAnd so that we can get started right away without building everything from scratch, a situation that often leads us to procrastinate on what matters, the plugin includes predefined, disjointed thresholds that do not overlap and are bounded, with a clear floor and ceiling, for the most sensitive metrics:<\/p>\n\n
\nWith these thresholds, we will detect memory pressure, thermal risk and critical errors without falling into alert overload<\/A>, that noise that eventually makes everyone ignore everything.
\nAnd if an organization needs different values, great. We believe every organization is its own world, so they can be adjusted to our specific reality from the console itself.<\/p>\nCompatibility and requirements of the Pandora FMS plugin<\/h2>\n
\n
\nIf your case involves clusters with many GPUs per node or large-scale Kubernetes architectures, it may need to be complemented with approaches such as DCGM, Prometheus or other aggregation solutions.
\nNo tool does everything, and acknowledging that seems honest to us.<\/p>\nGPU monitoring and capacity planning<\/h2>\n
\nBut it is in capacity planning where monitoring translates into hard cash<\/strong>.
\nWith historical data on utilization, memory, temperature and power consumption, it is possible to detect GPUs that are recurrently saturated, identify underused ones, redistribute workloads and justify purchases to the procurement department with data, instead of hunches and pleas, since we will be able to measure the real growth in demand.
\nIn the end, it is data science applied to IT<\/a> and turning the past into decisions.<\/p>\nGPU monitoring as part of AI infrastructure monitoring<\/h2>\n
\nAn AI infrastructure also depends on CPU, RAM, disk, storage, network, APIs, services, logs, processes, containers, databases and overall availability.
\nThat is why GPU monitoring must be part of a broader AI infrastructure monitoring strategy<\/strong>.
\nWatching only the GPU is like piloting the Enterprise while paying attention only to what is happening in the antimatter engine, while ignoring shields, life support or even the course.
\nThat is where Pandora FMS becomes that legendary ship computer that knew everything.
\nPandora integrates that visibility into a complete IT operation, where AI applied to management<\/a>, generative AI<\/a> and deep learning<\/a> provide the analysis that, let us admit it as a species, no human could sustain manually, nor would they want to, because there is no need.
\nThis is especially relevant in data centers<\/a>, where availability is taken for granted until it is no longer there, as any good uptime monitoring<\/a> strategy knows well.<\/p>\nConclusion<\/h2>\n
\n
\nPandora FMS aims to be that bridge to bring GPU monitoring to on-premise, hybrid and enterprise environments. Because in the gaps of IT live the gremlins of the machine, the ones that cause trouble precisely where you are not watching<\/Strong>.<\/p>\nFrequently asked questions<\/h2>\n
What is GPU monitoring?<\/h4>\n
\nIt is used to control GPUs used in AI, HPC, inference, model training or intensive processing.
\nAs I mentioned at the beginning, it should not be confused with gaming, overclocking or graphic tuning tools.<\/p>\nWhy is it important to monitor GPUs in AI infrastructures?<\/h4>\n
\nWithout monitoring, we are blind in our most critical asset<\/Strong>, leaving it defenseless against saturation, overheating, errors or underutilization, which have a habit of attacking from the shadows and gaps we leave unwatched.
\nThis increases operational risk and makes it harder to get new hardware investments approved by the finance department.<\/p>\nWhich metrics should be monitored on a GPU?<\/h4>\n
\nAlmost nothing, but isolated they give us an unfinished puzzle, so in professional environments, these metrics must be analyzed together with the rest<\/Strong> of the infrastructure.<\/p>\nWhat is the difference between nvidia-smi, DCGM and a monitoring platform?<\/h4>\n
\nDCGM is aimed at GPU management and monitoring in data centers and clusters.
\nA platform such as Pandora FMS centralizes those metrics together with the rest of the infrastructure<\/Strong>, with history, alerts, dashboards and reports.<\/p>\nAt what temperature is a server GPU considered critical?<\/h4>\n
\nThese thresholds serve as an operational reference, although they may vary depending on the GPU model, manufacturer and thermal conditions of the environment.<\/p>\nWhat are ECC errors in a GPU and why do they matter?<\/h4>\n
\nUncorrected errors may indicate hardware failures and must be treated as critical incidents.
\nThey are especially relevant in data center GPUs such as A100, H100, Tesla or RTX PRO. In consumer-grade GPUs, the module is still issued, but with a value of 0 in the Pandora FMS plugin, because there are no ECC errors to count.<\/p>\nCan GPUs be monitored in on-premise environments with Pandora FMS?<\/h4>\n
\nIt can also be used in hybrid environments or cloud instances if the NVIDIA GPU is exposed to the operating system and the driver is correctly installed.<\/p>\nHow does GPU monitoring help with capacity planning?<\/h4>\n
\nIt also helps justify hardware purchases<\/Strong> or postpone unnecessary expansions with objective usage and capacity data in hand.<\/p>\nWhat happens if nvidia-smi is not installed on the server?<\/h4>\n
\nThis allows an alert to be generated without breaking the agent execution or producing invalid XML<\/Strong>.
\nThe issue remains visible as the absence of GPU or a failure in the availability of the data source.<\/p>\nIs GPU monitoring enough to manage an AI infrastructure?<\/h4>\n
\nThe GPU is a critical component, but an AI infrastructure also depends on CPU, RAM, storage<\/Strong>, network, services…
\nThat is why GPU monitoring must be integrated into a broader AI infrastructure monitoring strategy. Without that, it will be impossible to have a complete operational view.
\n[\/et_pb_text][et_pb_button button_url=”@ET-DC@eyJkeW5hbWljIjp0cnVlLCJjb250ZW50IjoicG9zdF9saW5rX3VybF9wYWdlIiwic2V0dGluZ3MiOnsicG9zdF9pZCI6IjM2MjI3MCJ9fQ==@” button_text=”\u2190 Back to IT Topics” button_alignment=”left” _builder_version=”4.22.0″ _dynamic_attributes=”button_url” _module_preset=”default” custom_button=”on” button_text_size=”1em” button_text_color=”#0C312F” button_bg_color=”#FFFFFF” button_bg_color_gradient_direction=”90deg” button_bg_color_gradient_stops=”#82B92E 0%|#3CB92E 100%” button_bg_color_gradient_start=”#82B92E” button_bg_color_gradient_end=”#3CB92E” button_border_width=”1px” button_border_color=”#eaeaea” button_border_radius=”100px” button_use_icon=”off” z_index=”0″ custom_margin=”60px||0px||false|false” custom_padding=”10px|50px|10px|50px|true|true” custom_padding_tablet=”” custom_padding_phone=”10px|20px|10px|20px|true|true” custom_padding_last_edited=”on|phone” custom_css_main_element=”right:0!important;||font-family:%22Pandora-Bold%22!important;” global_module=”367749″ locked=”off” global_colors_info=”{}” button_bg_color__hover_enabled=”on|desktop” button_bg_color_gradient_start__hover=”#eaeaea” button_bg_color_gradient_end__hover=”#f4f4f4″ button_bg_color__hover=”#eaeaea” button_bg_enable_color__hover=”on” button_bg_use_color_gradient__hover=”on” button_bg_color_gradient_stops__hover=”#eaeaea 0%|#f4f4f4 100%”][\/et_pb_button][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ custom_padding_last_edited=”on|desktop” admin_label=”Final CTA” _builder_version=”4.27.0″ _module_preset=”default” background_color=”#161327″ use_background_color_gradient=”on” background_color_gradient_stops=”rgba(22,19,39,0.5) 17%|rgba(22,19,39,0.5) 100%” background_color_gradient_overlays_image=”on” background_image=”https:\/\/pandorafms.com\/wp-content\/uploads\/2023\/12\/banner-contacta-it-topics.webp” background_size=”custom” background_image_width=”121%” background_image_height=”192%” background_position=”top_left” z_index=”1″ max_width=”1080px” max_width_tablet=”98%” max_width_phone=”98%” max_width_last_edited=”on|tablet” module_alignment=”center” custom_margin=”80px||80px||true|false” custom_padding=”40px|20px|120px|20px|false|true” custom_padding_tablet=”40px|0px|120px|0px|false|true” custom_padding_phone=”40px|0px|120px|0px|false|true” scroll_scaling=”40|55|85|100|100%|120%|100%” motion_trigger_start=”top” background_last_edited=”on|phone” background_size_tablet=”cover” background_size_phone=”cover” background_position_phone=”top_center” border_radii=”off|20px|20px|20px|20px” border_color_all=”#ffffff” box_shadow_style=”preset1″ box_shadow_vertical=”0px” box_shadow_blur=”80px” box_shadow_color=”#506da0″ saved_tabs=”all” global_colors_info=”{}”][et_pb_row use_custom_gutter=”on” gutter_width=”2″ make_equal=”on” _builder_version=”4.22.0″ _module_preset=”default” max_width=”750px” module_alignment=”center” custom_margin=”0px||0px||true|false” custom_padding=”0px|0px|0px|0px|true|true” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.22.0″ _module_preset=”default” global_colors_info=”{}”][et_pb_text content_tablet=”<\/p>\n
o resuelve tus dudas sobre nuestras licencias<\/p>\n