AI Insight Metrics

This document describes the key metrics and status information provided by KakaoCloud AI Insight. AI Insight metrics are available on the Overview page and the Cluster, Node, and GPU detail pages in GPU Explorer.

Note

If Metric Exporter or the monitoring agent is not installed or is not working properly, some metrics may not be collected and No data to display or Agent Missing may appear on the page.

Resource Summary Metrics

Resource Summary summarizes the overall status of GPU resources at the top of the AI Insight Overview and GPU Explorer detail pages.

Metric	Description	Unit
Total GPUs	Total number of GPUs in the query scope	count
Total Clusters	Total number of clusters in the query scope	count
Total Nodes	Total number of nodes in the query scope	count
Average GPU Load	Average GPU compute utilization in the query scope	%
Average GPU Memory Usage	Average GPU memory usage in the query scope	%
Average GPU Temperature	Average GPU temperature in the query scope	°C
ECC Errors	Sum of SBE and DBE ECC Errors that occurred in the last 24 hours	count

Caution

ECC Error count does not include XID Event, Throttling, or Overheat events. XID Event Code and Throttle Event must be checked as separate metrics.

GPU Status Metrics

AI Insight displays the status of GPUs or MIG instances as follows. If multiple conditions are met at the same time, the status with the highest severity is displayed first.

Status	Description	Main Criteria
Active GPU	GPU in normal operation with compute or memory in use	GPU compute utilization is 10% or higher, or GPU memory usage is 20% or higher
Warning GPU	GPU requiring enhanced monitoring because abnormal signs are detected	One or more of the following occurs: GPU temperature remains at 85°C or higher for 3 minutes, SBE ECC Error, or minor Thermal/Power Throttling
Critical GPU	GPU in severe status requiring immediate inspection or action	One or more of the following occurs: GPU temperature remains at 90°C or higher for 2 minutes, DBE ECC Error, severe Thermal Throttling, or Reliability Violation
Pending GPU	GPU waiting because the node is inactive	The node is in an inactive lifecycle state such as stopped, booting, rebooting, resizing, or powered off
Idle GPU	Idle GPU where both compute and memory usage are low	GPU compute utilization is below 10% and GPU memory usage is below 20%
Agent Missing	GPU whose metric collection component is missing or not working properly	Collection data from Metric Exporter or the monitoring agent is not found

Status Priority

Even if Idle or Active conditions are met, a higher-severity status is displayed if Warning or Critical conditions are also detected. For example, if GPU utilization is low and the GPU meets Idle conditions, but a DBE ECC Error occurs on the same physical GPU, it may be displayed as Critical.

GPU Utilization and Memory Usage

Metric	Description	Use in Status Determination
GPU utilization	Ratio of time the GPU compute engine is active.	Used as one Idle condition if below 10%, and one Active condition if 10% or higher.
GPU memory usage	Ratio of used GPU memory to total GPU memory.	Used as one Idle condition if below 20%, and one Active condition if 20% or higher.
GPU idle ratio	Ratio of time the GPU is not used.	Used to identify idle GPUs.

GPU memory usage is calculated by dividing used GPU memory by the sum of used memory and free memory.

GPU Metrics

GPU Metrics provides time-series metric trends by GPU as charts.

Metric	Description	Unit	Recommended Use
GPU Utilization Trend	GPU utilization trend	%	Check GPU compute load
GPU Memory Usage Trend	GPU memory usage trend	%	Check GPU memory usage
GPU Temperature Trend	GPU temperature trend	°C	Check abnormal GPU temperature
GPU Idle Trend	GPU idle ratio trend	%	Identify idle GPUs
GPU Throttling	Accumulated time or occurrence status where GPU clocks were lowered due to constraints	sec or status	Check whether performance constraints occur
GPU ECC Error count	Trend of ECC Errors that occurred in the last 24 hours	count	Check GPU memory errors

Node Status Metrics

Node Status shows system resource status for a specific node. Displayed metrics and metric names may differ depending on whether the node is based on Virtual Machine (VM) or Kubernetes Engine (KE).

Note

VM nodes display usage and throughput metrics based on the operating system and instance. KE nodes also display Limit, Request, and Usage-based metrics collected from Kubernetes nodes.

VM Node Status Metrics

For VM nodes, you can check CPU, memory, disk, and network usage or throughput trends.

Metric	Description	Unit
Overall CPU usage	Overall CPU usage trend of the node	%
CPU usage by core	CPU usage trend by CPU core on the node	%
Overall memory usage	Overall memory usage trend of the node	%
Disk read bytes	Disk read throughput trend of the node	Bytes/s or KB/s
Disk write bytes	Disk write throughput trend of the node	Bytes/s or KB/s
Network receive bytes	Network receive throughput trend of the node	Bytes/s or KB/s
Network transmit bytes	Network transmit throughput trend of the node	Bytes/s or KB/s

KE Node Status Metrics

For KE nodes, you can check CPU, memory, and disk usage based on Limit, Request, and Usage from a Kubernetes resource perspective, as well as network throughput trends.

Metric	Description	Unit
CPU usage by metric (millicores)	Node CPU usage trend by Limit, Request, and Usage	millicores
Memory usage by metric	Node memory usage trend by Limit, Request, and Usage	Bytes or GB
Disk usage by metric	Node disk usage trend compared with Limit	Bytes or GB
Network receive bytes	Node network receive throughput trend	KB/s
Network transmit bytes	Node network transmit throughput trend	KB/s

Differences Between VM and KE Node Status Metrics

Category	VM	KE
CPU metrics	Overall usage, usage by core	Usage by Limit, Request, and Usage
Memory metrics	Overall usage	Usage by Limit, Request, and Usage
Disk metrics	Read/write throughput	Usage compared with Limit
Network metrics	Receive/transmit throughput	Receive/transmit throughput
Main interpretation criteria	Check system resource usage of the instance	Compare Kubernetes node allocation and actual usage

Caution

Node Status metric names and units are displayed differently depending on node type. When comparing VM node and KE node metrics, check the metric names and units on the page because metrics with the same name may have different collection criteria and units.

GPU Outlier Detection Metrics

GPU Outlier Detection provides a table of average values, maximum values, and peak times for key metrics by GPU. You can use it to check whether a specific GPU shows a different pattern from other GPUs.

Metric	Description	Unit
GPU	GPU or MIG instance name	-
Avg Util	Average GPU utilization	%
Max Util	Maximum GPU utilization	%
Peak Time	Time when the maximum value occurred	timestamp
Avg Mem	Average GPU memory usage	%
Max Mem	Maximum GPU memory usage	%
Avg Temp	Average GPU temperature	°C
Max Temp	Maximum GPU temperature	°C

GPU Correlation Metrics

GPU Correlation provides scatter plots showing relationships between GPU metrics.

Chart	Description	Example Use
GPU Utilization vs Temperature	Displays the relationship between GPU utilization and temperature	Identify GPUs with abnormally high temperature relative to utilization
GPU Utilization vs Idle time	Displays the relationship between GPU utilization and idle time	Identify GPUs that have been idle for a long time

Event and Error Metrics

AI Insight event and error metrics are used for GPU status determination and failure cause analysis.

ECC Error

ECC Error is an error detected in GPU memory. AI Insight aggregates and displays SBE and DBE ECC Errors that occurred in the last 24 hours.

Type	Description	Status Impact
SBE	Single Bit Error. This is a correctable ECC Error.	May be included in Warning conditions.
DBE	Double Bit Error. This is an uncorrectable ECC Error.	May be included in Critical conditions.

Note

ECC Error is collected at the physical GPU level. If MIG is enabled, the same ECC Error value may be displayed for MIG instances on the same physical GPU.

XID Event Code

XID Event Code is an internal error event code generated by NVIDIA GPUs. AI Insight displays the last detected XID code value as an informational metric. XID Event Code is not reflected in the conditions used to determine whether the current GPU status is Warning or Critical.

Item	Description
XID Event Code	Last detected GPU error event code
Recent XID code	Reference value for the last detected XID code in the last 24 hours
Caution	XID Event Code is an informational metric and is not used as an occurrence count or GPU status determination criterion.

Caution

XID Event Code indicates the last detected code value. Even if an XID code is displayed, the code itself does not mean Warning or Critical status. When determining GPU status, check ECC Error, GPU Temperature, and GPU Throttling metrics together.

Throttle Event

Throttle Event means the GPU lowered its clock due to a specific constraint. In AI Insight, the value displayed as Throttle Event may be based on accumulated time during which GPU clocks were constrained or whether the event occurred, rather than a simple occurrence count.

Type	Description	Status Impact
Thermal Violation	GPU clocks are lowered due to temperature limits	May be included in Warning conditions if it increases by 30 seconds or more in the last 5 minutes, or Critical conditions if it increases by 180 seconds or more.
Power Violation	GPU clocks are lowered due to power limits	May be included in Warning conditions if it increases by 30 seconds or more in the last 5 minutes.
Sync Boost Violation	GPU clocks are lowered due to Sync Boost limits	Can be used as a cause analysis metric.
Board Limit Violation	GPU clocks are lowered due to board limits	Can be used as a cause analysis metric.
Low Utilization Violation	Constraint applied due to low utilization	Can be used as a cause analysis metric.
Reliability Violation	Constraint applied to protect GPU reliability	May be included in Critical conditions if it persists.

Note

If Throttle Event is displayed, check GPU temperature, power limits, and reliability limits together. Thermal Violation is especially useful when checked together with GPU temperature trends.

Overheat

Overheat means GPU temperature exceeded the threshold or Thermal Throttling occurred due to temperature limits. AI Insight can identify abnormal temperature through GPU temperature trends and Thermal Violation.

Criteria	Description
Warning level	GPU temperature remains at 85°C or higher for 3 minutes or longer
Critical level	GPU temperature remains at 90°C or higher for 2 minutes or longer
Thermal Violation	GPU clocks are lowered due to temperature limits

Caution

The number of Overheat occurrences may differ from the actual number of hardware events. Check GPU Temperature Trend and Throttle Event together to determine whether temperature is abnormal.

Metric Display Criteria in MIG Environments

In environments where MIG is enabled, some metrics are collected by MIG instance and others are collected at the physical GPU level.

Metric or Status	Display Criteria
GPU utilization	May be displayed differently by MIG instance.
GPU memory usage	May be displayed differently by MIG instance.
Idle / Active status	May be displayed differently depending on utilization and memory usage of each MIG instance.
ECC Error	Collected at the physical GPU level, so the same value may be displayed for MIG instances on the same physical GPU.
GPU temperature	Collected at the physical GPU level, so the same value may be displayed for MIG instances on the same physical GPU.
XID Event Code	Collected at the physical GPU level, so the same value may be displayed for MIG instances on the same physical GPU.
Throttling	Collected at the physical GPU level, so the same value may be displayed for MIG instances on the same physical GPU.
Warning / Critical status	If physical GPU-level conditions are met, the same status may be applied to MIG instances on the same physical GPU.

Metric Interpretation Considerations

Metric	Consideration
ECC Error count	Sum of SBE and DBE ECC Errors that occurred in the last 24 hours. XID, Throttling, and Overheat are not included.
XID Event Code	Last detected code value. It is an informational metric and is not used as an occurrence count or status determination criterion.
Throttle Event	May be displayed based on accumulated time or occurrence status rather than a simple occurrence count.
Overheat	Determined by temperature threshold excess or Thermal Violation rather than a direct event count.
MIG instance	Idle/Active may differ by instance, while Warning/Critical may be applied at the physical GPU level.