Key Concepts
KakaoCloud AI Insight is a GPU monitoring service that lets you explore GPU resources by cluster, node, and GPU and check key metrics. You can check the overall GPU status in Overview and analyze detailed status for a specific cluster, node, or GPU in GPU Explorer.
AI Insight Configuration
AI Insight collects status information from GPUs and nodes through metric collection components and visualizes the collected data in the console.
| Component | Description |
|---|---|
| Metric Exporter | Component for collecting GPU and node metrics |
| Overview | Page for checking summary status and status of all GPU resources |
| GPU Explorer | Menu for checking detailed information by cluster, node, and GPU |
| GPU Map | Map that visually displays GPU resource status and relationships |
| GPU Metrics | Trend charts for GPU utilization, memory usage, temperature, idle ratio, error events, and other metrics |
Pages and Resource Units
Overview
Overview is the entry page for AI Insight. It provides summary information such as total GPUs, clusters, nodes, average GPU load, average GPU memory usage, average GPU temperature, and ECC Error count. You can also quickly identify resources with abnormal status using GPU counts by status and GPU Map.
GPU Explorer
GPU Explorer is a menu for exploring GPU resources in detail. It consists of the following submenus.
| Menu | Description |
|---|---|
| Cluster | Check status and metrics for GPU resources in a specific cluster |
| Node | Check system metrics for a specific node and the status of GPUs on that node |
| GPU | Check detailed metrics and events for an individual GPU or MIG instance |
Cluster
Cluster is a Kubernetes Engine cluster or logical operation unit that contains GPU resources. On the Cluster page, you can check GPU count, average/maximum GPU load, average/maximum GPU memory usage, average/maximum GPU temperature, average/maximum GPU idle ratio, ECC Error count, and other information for the cluster.
Node
Node is a server or virtual machine unit with GPUs attached. On the Node page, you can check GPU status as well as node system metrics such as CPU, memory, disk, and network.
GPU
GPU is the most detailed resource unit that AI Insight can display. On the GPU page, you can check utilization, memory usage, temperature, idle ratio, throttling, and ECC Error trends for an individual GPU or MIG instance.
MIG
MIG (Multi-Instance GPU) is a feature that partitions one physical GPU into multiple GPU instances. AI Insight can distinguish and display GPUs configured with MIG in GPU Map and detail pages.
In MIG environments, status display methods may differ depending on the collection unit.
| Status or Metric | Collection Unit | Display Method |
|---|---|---|
| GPU compute utilization | MIG instance | Can be displayed differently by MIG instance |
| GPU memory usage | MIG instance | Can be displayed differently by MIG instance |
| Idle / Active | MIG instance | Status may differ by MIG instance within the same physical GPU |
| GPU temperature | Physical GPU | Same criteria are applied to MIG instances on the same physical GPU |
| ECC Error | Physical GPU | Same value may be displayed for MIG instances on the same physical GPU |
| XID Event Code | Physical GPU | Same value may be displayed for MIG instances on the same physical GPU |
| Throttling | Physical GPU | Same criteria are applied to MIG instances on the same physical GPU |
| Warning / Critical | Physical GPU | All MIG instances on the same physical GPU may be displayed with the same status |
For example, if a Warning condition occurs on one physical GPU, all MIG instances on that GPU may be displayed as Warning.
Warning condition occurs on GPU 0
GPU 0
├─ MIG 0: Warning
├─ MIG 1: Warning
└─ MIG 2: Warning
GPU Status
AI Insight displays the status of each GPU or MIG instance based on collected GPU metrics. If multiple status conditions are met at the same time, the status with the highest severity is displayed first.
Idle -> Active -> Warning -> Critical
Agent Missing means metric collection is unavailable and may be displayed separately from the status priority above.
| Status | Meaning | Display Criteria |
|---|---|---|
| Agent Missing | Status where metrics cannot be collected because Metric Exporter or the monitoring agent is not installed or is not working properly | Metric collection data is not found from the target resource |
| Idle | Idle status where both GPU compute and memory usage are low | GPU compute utilization is below 10% and GPU memory usage is below 20% |
| Active | Normal operating status where GPU compute or memory is in use | GPU compute utilization is 10% or higher, or GPU memory usage is 20% or higher |
| Warning | Status requiring enhanced monitoring because abnormal signs are detected | GPU temperature remains at 85°C or higher for 3 minutes or longer, SBE ECC Error occurs, or minor Thermal/Power Throttling occurs |
| Critical | Severe status requiring immediate inspection or action | GPU temperature remains at 90°C or higher for 2 minutes or longer, DBE ECC Error occurs, severe Thermal Throttling occurs, or Reliability Violation occurs |
| Pending | Status where the GPU is classified as waiting because the node containing it is inactive | The node is in an inactive lifecycle state such as stopped, booting, rebooting, resizing, or powered off |
Warning and Critical status include metrics collected at the physical GPU level. If MIG is enabled, MIG instances on the same physical GPU may display the same Warning or Critical status.
Warning Status
Warning does not necessarily mean an immediate failure, but it requires enhanced monitoring. A GPU may be displayed as Warning if one or more of the following conditions apply.
| Condition | Description |
|---|---|
| Increased GPU temperature | GPU temperature remains at 85°C or higher for 3 minutes or longer |
| SBE ECC Error occurs | A correctable ECC Error occurs |
| Thermal Throttling occurs | Thermal Violation increases by 30 seconds or more within the last 5 minutes |
| Power Throttling occurs | Power Violation increases by 30 seconds or more within the last 5 minutes |
Critical Status
Critical is a severe status requiring immediate inspection or action. A GPU may be displayed as Critical if one or more of the following conditions apply.
| Condition | Description |
|---|---|
| Excessive GPU temperature increase | GPU temperature remains at 90°C or higher for 2 minutes or longer |
| DBE ECC Error occurs | An uncorrectable ECC Error occurs |
| Reliability Violation persists | Restrictions for GPU stability protection occur continuously |
| Severe Thermal Throttling occurs | Thermal Violation increases by 180 seconds or more within the last 5 minutes |
GPU Events
AI Insight provides GPU event information for GPU status determination and detailed analysis.
| Event | Description | Status Impact |
|---|---|---|
| ECC Error | Error detected in GPU memory | SBE may be included in Warning conditions, and DBE may be included in Critical conditions |
| XID Event Code | Internal error event code generated by NVIDIA GPUs | Informational metric and not reflected in Warning/Critical status determination |
| Throttle Event | Status where the GPU lowered its clock due to constraints such as power, temperature, board, or reliability | Thermal/Power Throttling may be included in Warning conditions, and severe Thermal Throttling or Reliability Violation may be included in Critical conditions |
| Overheat | Status where GPU temperature exceeds the threshold | May be included in Warning conditions if 85°C or higher persists for 3 minutes, or Critical conditions if 90°C or higher persists for 2 minutes |
Throttle Event may not be the same as a simple occurrence count. It may be displayed based on accumulated time during which GPU clocks were lowered by constraints or based on whether it occurred. Check GPU temperature and power status together.
Data Query Criteria
On AI Insight pages, you can select a time range such as 1 hour, 3 hours, 12 hours, 1 day, or 7 days to query metrics. When you change the time range, data displayed in Resource Summary, GPU Metrics, Outlier Detection, Correlation, and other areas is refreshed based on the selected period.
However, some event metrics, such as ECC Error count, XID Event Code, and Throttling, may be aggregated or displayed based on the last 24 hours. XID Event Code is an informational metric and is not reflected in GPU status determination. For criteria for each metric, see AI Insight Metrics.