AI Insight Troubleshooting

This document summarizes common issues related to AI Insight and how to resolve them.

Resource Appears as Agent Missing
No Data to Display Appears
GPU Resource Is Not Detected
GPU Appears in Warning Status
GPU Appears in Critical Status
Throttle Event Is Displayed
XID Event Code Appears Differently Than Expected
All MIG Instances Appear in the Same Warning or Critical Status
MIG Instances Are Not Displayed
GPU Metrics Are Not Displayed in Virtual Machine
Error Occurs While Running Installation Commands

Resource Appears as Agent Missing

If a resource appears in Agent Missing status in AI Insight, the metric collection component may not be installed or may not be working properly in the target environment. In Agent Missing status, GPU metrics are not collected and data may not be displayed on the page.

Cause 1: Metric Exporter or Monitoring Agent Is Not Installed

Solution: Install the metric collection component for the target environment.

Environment	Solution
Kubernetes Engine	See Metric Exporter Installation
Virtual Machine	See the VM installation procedure in Metric Exporter Installation

Cause 2: GPU Operator Pods Are Not Running Properly

Solution: Check Pod status in the GPU Operator namespace.

Check Pod status
kubectl get pods -n gpu-operator

If a Pod is not in Running status, check the events and logs for that Pod.

Check Pod events
kubectl describe pod POD_NAME -n gpu-operator

Check DCGM Exporter logs
kubectl logs -n gpu-operator daemonset/nvidia-dcgm-exporter

Cause 3: dcgm-exporter hostNetwork Is Not Configured

Solution: Check whether hostNetwork and dnsPolicy are applied to the nvidia-dcgm-exporter Pod.

Check hostNetwork settings
kubectl get pod -n gpu-operator -l app=nvidia-dcgm-exporter -o yaml | grep -E "hostNetwork|dnsPolicy"

If the settings are applied properly, the following values are displayed.

Expected configuration example
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

If the settings are missing, run the patch command in Metric Exporter Installation again.

No Data to Display Appears

If No data to display appears in a chart area, there may be no metrics for the selected query conditions, or metric collection may not be working properly.

Cause 1: No Data Was Collected in the Selected Time Range

Solution: Change the time range at the top of the page and query again.

1 hour
3 hours
12 hours
1 day
7 days

Cause 2: Data Has Not Yet Been Reflected After Collection

Solution: Run manual refresh or configure auto refresh, then check again.

Cause 3: Target Resource Is in Agent Missing Status

Solution: Resolve Agent Missing status and query again. For more information, see Resource Appears as Agent Missing.

GPU Resource Is Not Detected

If GPU resources are not displayed in AI Insight for Kubernetes Engine, check whether GPU resources are recognized properly on cluster nodes.

Check GPU resource recognition
kubectl describe node GPU_NODE_NAME | grep nvidia.com/gpu

If GPU resources are not displayed, check GPU node configuration, NVIDIA driver installation, and GPU Operator installation status.

GPU Appears in Warning Status

If a GPU appears in Warning status, abnormal signs have been detected on the GPU. This does not necessarily mean an immediate failure, but the GPU must be monitored continuously.

Cause	How to Check	Action
Increased GPU temperature	Check whether GPU Temperature Trend remains at 85°C or higher	Check cooling status, workload load, and node status
SBE ECC Error occurs	Check GPU ECC Error count	Check whether it occurs repeatedly, and inspect GPU status if needed
Thermal Throttling occurs	Check whether Thermal Violation increased by 30 seconds or more in the last 5 minutes	Check temperature cause and cooling status
Power Throttling occurs	Check whether Power Violation increased by 30 seconds or more in the last 5 minutes	Check power limits or instance specifications

GPU Appears in Critical Status

If a GPU appears in Critical status, it is in a severe state that requires immediate inspection or action.

Cause	How to Check	Action
Excessive GPU temperature increase	Check whether GPU Temperature Trend remains at 90°C or higher for 2 minutes or longer	Consider stopping workloads, and check cooling and node status
DBE ECC Error occurs	Check GPU ECC Error count	Inspect GPU status because an uncorrectable memory error may have occurred
Severe Thermal Throttling occurs	Check whether Thermal Violation increased by 180 seconds or more in the last 5 minutes	Check temperature cause, workload load, and cooling status
Reliability Violation persists	Check GPU Throttling	Check whether stability protection limits persist, and inspect node/GPU status

Caution

Critical status indicates a high possibility of failure. Check workload impact and, if needed, stop workloads, isolate the node, and check driver and hardware status.

Throttle Event Is Displayed

Throttle Event means the GPU lowered its clock due to constraints such as power, temperature, board, or reliability. It may not be the same as a simple occurrence count and may be displayed based on accumulated time or occurrence status.

Type	Meaning	Metric to Check
Thermal Violation	Clock reduction due to temperature limits	GPU Temperature Trend
Power Violation	Clock reduction due to power limits	GPU utilization, workload load
Sync Boost Violation	Clock reduction due to Sync Boost limits	GPU Throttling
Board Limit Violation	Clock reduction due to board limits	GPU Throttling
Low Utilization Violation	Constraint related to low utilization	GPU Utilization Trend
Reliability Violation	Reliability protection constraint	GPU Throttling, XID Event Code

If Throttle Event is displayed, check GPU Temperature Trend and GPU Utilization Trend together. XID Event Code can be used as an informational metric.

XID Event Code Appears Differently Than Expected

XID Event Code is the last detected GPU error event code. In AI Insight, XID is provided for informational purposes and is not reflected in the conditions used to determine Warning or Critical GPU status.

Symptom	Cause	Description
XID code remains displayed	Last XID code value is retained	The last detected XID code based on the recent query period may be displayed on the page
XID exists but ECC Error count is displayed as 0	Different metrics	ECC Error count does not include XID Event

If XID is displayed, check temperature, ECC Error, and Throttling together on the GPU detail page. However, XID display alone does not classify the GPU as Warning or Critical.

All MIG Instances Appear in the Same Warning or Critical Status

In MIG environments, GPU utilization and memory usage may be collected by MIG instance. Therefore, Idle or Active status may differ by MIG instance.

In contrast, GPU temperature, ECC Error, and Throttling may be collected at the physical GPU level. In this case, if a Warning or Critical condition occurs on one physical GPU, all MIG instances on that GPU may appear in the same Warning or Critical status. XID Event Code may also be displayed at the physical GPU level, but it is not reflected in status determination.

Warning condition occurs on GPU 0

GPU 0
├─ MIG 0: Warning
├─ MIG 1: Warning
└─ MIG 2: Warning

This behavior may be the result of the same physical GPU status being applied to MIG instances. On the GPU detail page, check temperature, ECC Error, and Throttling, and use XID Event Code as reference information.

MIG Instances Are Not Displayed

If MIG instances are not displayed in an environment that uses MIG, check the mig.strategy setting when installing GPU Operator.

GPU Operator installation option example
--set mig.strategy=none
--set mig.strategy=single
--set mig.strategy=mixed

Value	Description
`none`	Disable MIG
`single`	Apply the same MIG profile to all GPUs
`mixed`	Allow different MIG profiles for each GPU

If you use MIG, configure single or mixed according to your operation policy.

GPU Metrics Are Not Displayed in Virtual Machine

In Virtual Machine environments, check the installation status of DCGM, DCGM Exporter, and the monitoring agent.

Complete the VM installation procedure in Metric Exporter Installation.
Check whether the nvidia-dcgm, dcgm-exporter, and kic_monitor_agent services are running properly.
Check whether DCGM metrics are exposed at http://localhost:9400/metrics.
Check whether [[inputs.prometheus]] input was added to the monitoring agent configuration.
Change the time range or refresh the AI Insight page, then query again.

Error Occurs While Running Installation Commands

If an error occurs while running GPU Operator installation commands, check the following items.

Check Item	Description
Helm Repository	Check whether `helm repo add nvidia https://helm.ngc.nvidia.com/nvidia` was executed
Chart version	Check the `--version` value in the installation command
CSV file location	Check whether `dcgm-exporter-metrics.csv` exists in the current directory
Cluster permission	Check whether you can access the target cluster with the `kubectl` command
Namespace	Check whether the `gpu-operator` namespace was created

Resource Appears as Agent Missing​

Cause 1: Metric Exporter or Monitoring Agent Is Not Installed​

Cause 2: GPU Operator Pods Are Not Running Properly​

Cause 3: dcgm-exporter hostNetwork Is Not Configured​

No Data to Display Appears​

Cause 1: No Data Was Collected in the Selected Time Range​

Cause 2: Data Has Not Yet Been Reflected After Collection​

Cause 3: Target Resource Is in Agent Missing Status​

GPU Resource Is Not Detected​

GPU Appears in Warning Status​

GPU Appears in Critical Status​

Throttle Event Is Displayed​

XID Event Code Appears Differently Than Expected​

All MIG Instances Appear in the Same Warning or Critical Status​

MIG Instances Are Not Displayed​

GPU Metrics Are Not Displayed in Virtual Machine​

Error Occurs While Running Installation Commands​