Skip to main content

Use GPU Resource Preemption

Use Kueue WorkloadPriorityClass to assign priorities to GPU workloads and allow higher-priority workloads to preempt resources from lower-priority workloads.

Preemption overview​

Kueue manages GPU resource contention and preemption. Each tenant namespace corresponds to one ClusterQueue, and preemption occurs only within the same ClusterQueue. A workload cannot preempt another tenant's workload, and another tenant's workload cannot preempt it.

ItemDescription
Preemption scopeClusterQueue of the same tenant
Preemption targetA running workload with a lower WorkloadPriorityClass value than the new workload
Preemption across tenantsNot supported
Admission order across LocalQueuesDetermined by GPU usage-based Fair Sharing and LocalQueue weight

The tenant's ClusterQueue uses quota that matches its allocated resources and applies the following policies.

SettingValueDescription
preemption.withinClusterQueueLowerPriorityPreempt only lower-priority workloads in the same ClusterQueue
preemption.reclaimWithinCohortNeverDo not preempt workloads from other tenants
admissionScope.admissionModeUsageBasedAdmissionFairSharingAdmit the LocalQueue with lower GPU usage history first
queueingStrategyBestEffortFIFOAdmit a later workload if it can run, even when an earlier workload is waiting for resources

Preemption conditions​

Kueue evicts a lower-priority running workload and admits a new workload when all of the following conditions are met:

  • The new workload cannot fit within the tenant ClusterQueue quota.
  • A running workload with a lower WorkloadPriorityClass value exists in the same ClusterQueue.
  • Evicting the lower-priority workload allows the new workload to fit within the quota.

Workloads with the same or higher priority are not preempted. If there is no workload that can be preempted, the new workload waits in the Queue until resources become available.

Priority classes​

You can use the following WorkloadPriorityClass values.

NameValueUse
low100Low-priority jobs that can be preempted
normal500Default priority for general jobs
high1000Jobs that require high priority
critical1500Urgent jobs that must run immediately
Caution

Use high and critical only when operationally necessary. A higher priority does not allow a workload to exceed the quota allocated to its tenant, and it can preempt lower-priority workloads running in the same tenant.

Assign priority to a workload​

Specify a priority class name in the workload's kueue.x-k8s.io/priority-class label.

For a TrainJob, add the label to metadata.labels.

trainjob.yaml
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
name: finetune-01
namespace: <user-namespace>
labels:
kueue.x-k8s.io/queue-name: training
kueue.x-k8s.io/priority-class: high
spec:
runtimeRef:
apiGroup: trainer.kubeflow.org
kind: ClusterTrainingRuntime
name: torch-distributed
trainer:
numNodes: 2
numProcPerNode: 8
resourcesPerNode:
requests:
nvidia.com/gpu: "8"
limits:
nvidia.com/gpu: "8"

For workloads managed by Kueue at the Pod level, such as Deployment, LeaderWorkerSet, and DynamoGraphDeployment, add the label to metadata.labels in the Pod template.

In kueue.x-k8s.io/queue-name, specify the LocalQueue created in the user namespace. We recommend specifying the Queue and priority labels together when creating a workload.

Create a tenant-specific priority class​

To add a priority level between the defaults, create a CustomWorkloadPriorityClass in the user namespace. The platform creates a WorkloadPriorityClass with the same name.

Use the <user-namespace>-<priority-name> format for the name. Specify the same name in the workload's kueue.x-k8s.io/priority-class label.

custom-priority-class.yaml
apiVersion: kakaocloud.io/v1alpha1
kind: CustomWorkloadPriorityClass
metadata:
name: <user-namespace>-urgent
namespace: <user-namespace>
spec:
value: 1200
description: Retraining before the quarterly deadline

Run the following command to check the creation status.

kubectl get cwpc -n <user-namespace>

When READY is True, the class is ready to use. If a WorkloadPriorityClass with the same name already exists, READY is False and reason is shown as NameConflict. In this case, create the class again with a different name.

metadata:
labels:
kueue.x-k8s.io/queue-name: training
kueue.x-k8s.io/priority-class: <user-namespace>-urgent
Caution

Deleting a CustomWorkloadPriorityClass also deletes the associated WorkloadPriorityClass. New workloads that refer to a deleted class are not admitted and receive a WorkloadPriorityClassNotFound warning. Before deleting a class, check that no workloads refer to it.

Adjust admission order across LocalQueues​

You can create LocalQueues for different teams in the same tenant to adjust workload admission order. All LocalQueues must use the tenant's ClusterQueue, so specify the same value as the user namespace in spec.clusterQueue.

localqueue.yaml
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
name: training
namespace: <user-namespace>
spec:
clusterQueue: <user-namespace>
fairSharing:
weight: "2"

When GPU usage history is otherwise equal, a higher fairSharing.weight value gives the LocalQueue earlier admission. This value affects only LocalQueue admission order. WorkloadPriorityClass determines whether a workload is preempted.

For information about creating a LocalQueue and assigning it to a workload, see Use Queues and Schedulers.

Check preempted workloads​

Run the following commands to check the status of a Kueue Workload.

kubectl get workloads -n <user-namespace>
kubectl describe workload -n <user-namespace> <workload-name>

A preempted Workload has the following conditions:

  • Evicted is True and reason is Preempted.
  • The reason of the Preempted condition is InClusterQueue.
  • message shows the UID and JobUID of the workload that preempted it.

Use the JobUID shown in message to find the workload that performed the preemption.

kubectl get workloads -n <user-namespace> -l kueue.x-k8s.io/job-uid=<JobUID>

A preempted TrainJob changes to the Suspend state and waits for resources in the Queue again.

Considerations​

  • Do not add the scheduling.volcano.sh/queue-name annotation or kai.scheduler/queue label to a TrainJob. The TrainJob is managed and preempted by Kueue, and adding either value causes creation to be rejected.
  • WorkloadPriorityClass is separate from the Pod PriorityClass. The WorkloadPriorityClass value determines workload preemption order.
  • A higher priority does not allow a workload to exceed the quota allocated to its tenant. Preemption adjusts workload execution order within the same tenant's quota.

Gang Scheduling​

Gang Scheduling overview​

Distributed training and multi-node serving require multiple Pods to run at the same time. If some Pods start first, they can occupy GPUs while waiting for the remaining Pods. Gang Scheduling starts a workload only when all required Pods can be placed at once.

The cluster divides responsibilities as follows.

StageOwnerRole
admissionKueueDetermines whether a workload fits within the tenant quota. Jobs and TrainJobs remain suspended and do not create Pods until admitted. Deployments, LeaderWorkerSets, and DynamoGraphDeployments create Pods, but scheduling gates prevent placement.
placementVolcano or KAI SchedulerPlaces all Pods of an admitted workload on nodes at once. If there is not enough capacity, all Pods wait.

When you use TrainJob, LLMInferenceService, or DynamoGraphDeployment, the platform settings determine whether Gang Scheduling is used and which scheduler is used. You do not need to create a PodGroup.

When you directly use a Kubernetes Job, JobSet, Deployment, or Pod, the platform does not create a Gang. You must define the Gang group yourself.

See Configure Gang Scheduling directly (Job, JobSet, Deployment, Pod) for instructions.

WorkloadSchedulerGang unitWhat you specify
TrainJobKAI or Volcano, depending on the RuntimeAll Pods for the TrainJob nodesSelect a Runtime with runtimeRef
LLMInferenceService (with workers)VolcanoOne LWS group (one leader and workers)
The prefill Deployment is outside the Gang
schedulerName: volcano-scheduler
DynamoGraphDeploymentKAI (Grove PodGang)The entire DGD. Each component contributes the number of Pods specified by minAvailable; a multi-node worker consists of one leader and nodeCount - 1 workers.schedulerName: kai-scheduler and the kai.scheduler/enabled label
Job, JobSet, Deployment, Pod (direct configuration)Choose Volcano or KAIA manually created PodGroup or batch-min-member for a Job or JobSetConfigure a PodGroup or Gang annotation directly. See Configure Gang Scheduling directly.

TrainJob​

The ClusterTrainingRuntime specified by spec.runtimeRef determines the scheduler and Gang policy for a TrainJob.

The platform provides the following Runtimes.

Runtime nameSchedulerPlacement policy
torch-distributed-kai-schedulerKAIPlaces Pods on the same switch when possible, and places them across switches if needed
torch-distributed-volcanoVolcanoUses soft network topology: tries to place Pods on the same switch, but distributes them if needed
torch-distributed-volcano-hardVolcanoUses hard network topology with highestTierAllowed: 1. This cluster has one HyperNode at tier 1. The workload runs only when all Pods fit on the same switch; otherwise, it continues to wait.

All three Runtimes group the numNodes Pods into one PodGroup and start only when all Pods can be placed. Some Pods do not start ahead of the others.

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
name: pretrain-01
namespace: kbm-g-np-example
labels:
kueue.x-k8s.io/queue-name: training
kueue.x-k8s.io/priority-class: normal
spec:
runtimeRef:
apiGroup: trainer.kubeflow.org
kind: ClusterTrainingRuntime
name: torch-distributed-kai-scheduler
trainer:
numNodes: 4
numProcPerNode: 8
resourcesPerNode:
requests:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"
limits:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"

LLMInferenceService (multi-node serving)​

An LLMInferenceService with spec.worker is deployed as a LeaderWorkerSet, and its leader and workers are grouped into one Gang. The LWS controller creates a Volcano PodGroup for each group. The group size (number of Pods) is determined by parallelism.data and parallelism.dataLocal. parallelism.data is the number of data-parallel (DP) ranks, and dataLocal is the number of DP ranks assigned to one Pod. parallelism.tensor is the number of GPUs (TP) used by one DP rank. TP is performed within a Pod and does not span Pods.

Prefill follows the same rules. If you specify spec.prefill.worker, a separate LeaderWorkerSet is created for prefill and forms a Gang within its own group. If not specified, prefill is created as a Deployment and is not part of a Gang. In either case, prefill and decode use separate LWS or Deployment resources, so they are not grouped into the same Gang.

Set schedulerName: volcano-scheduler in spec.template, spec.worker, and spec.prefill.template so Volcano places the Pods. For multi-node prefill, also set it in spec.prefill.worker.

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
name: glm-5-3
namespace: kbm-g-np-example
labels:
kueue.x-k8s.io/queue-name: serving
kueue.x-k8s.io/priority-class: high
spec:
replicas: 1
parallelism:
tensor: 8
data: 2
dataLocal: 1
template:
schedulerName: volcano-scheduler
worker:
schedulerName: volcano-scheduler
prefill:
replicas: 1
parallelism:
tensor: 8
template:
schedulerName: volcano-scheduler

In this example, decode has two DP ranks (data: 2), with one DP rank assigned to each Pod (dataLocal: 1). The LWS group therefore has two Pods: one leader and one worker. Each Pod uses eight GPUs with TP 8, so decode requires 16 GPUs in total. Volcano groups the two Pods into one PodGroup (minMember: 2) and schedules them together. Because spec.prefill.worker is not specified, prefill runs as one Deployment Pod (TP 8, eight GPUs) and is not part of a Gang. Therefore, Gang Scheduling applies only within the decode group; it does not guarantee that prefill and decode are placed at the same time.

spec.worker is merged into the KServe decode worker template by using strategic merge. The template's worker container has no probe. For example, specifying only startupProbe.failureThreshold creates a probe without a handler (such as httpGet), and the API server rejects worker Pod creation. Without a worker, the LWS group does not meet minMember: 2, so the leader also remains Pending. When overriding an object that is not present in the template, specify the entire object instead of only one field.

DynamoGraphDeployment (multi-node serving)​

Grove groups all DGD components, including the Frontend, into one PodGang, and KAI places them. The Gang includes the number of Pods specified by minAvailable for each component, so capacity must be available for Frontend, prefill, and decode together. This also applies to single-node components that do not specify multinode.nodeCount. When multinode.nodeCount is specified for a worker, it expands to one leader and nodeCount - 1 workers, all included in the same Gang. Add the kai.scheduler/enabled: "true" label and schedulerName: kai-scheduler to the Pod template.

apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
name: glm-5-3-nodecount4
namespace: kbm-g-np-example
spec:
backendFramework: vllm
components:
- name: VllmWorker
type: worker
replicas: 1
multinode:
nodeCount: 4
podTemplate:
metadata:
labels:
kai.scheduler/enabled: "true"
kueue.x-k8s.io/queue-name: serving
kueue.x-k8s.io/priority-class: high
spec:
schedulerName: kai-scheduler

Use nodeCount only when a model does not fit on one node. For models that fit on one node, using replicas is simpler and more resilient. Whether Pods are placed on different nodes depends on the GPU request per Pod (for example, eight GPUs); nodeCount itself does not force placement across nodes.

Changing resources, image, or args for a multi-node worker does not make the Dynamo operator perform a rolling update. The PodCliqueSet template is updated with a RollingUpdateNotSupported warning. Grove's default RollingRecreate strategy then replaces Pods one PCSG replica at a time. If there is one replica, the entire multi-node worker is interrupted during replacement, so this is not a zero-downtime update. The Gang may not fill if replacement is delayed. In production, deleting and recreating the DGD is more predictable.

Configure Gang Scheduling directly (Job, JobSet, Deployment, Pod)​

You can use Gang Scheduling when creating workloads directly with a Kubernetes Job, JobSet, Deployment, or a single Pod instead of using TrainJob, LLMInferenceService, or DynamoGraphDeployment. In this case, define the Gang group yourself and choose either Volcano or KAI. Do not mix both schedulers in one workload.

The Kueue admission process is the same for either scheduler. Jobs, JobSets, Deployments, and Pods all require the kueue.x-k8s.io/queue-name label. Without it, Pods are not created or remain blocked by a scheduling gate.

Configure Volcano directly​

Create a PodGroup first, then specify its name in the scheduling.k8s.io/group-name annotation on the Pod templates in the Gang. Set schedulerName to volcano-scheduler. This cluster does not use Volcano queues, so do not specify spec.queue.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
name: allreduce-test
namespace: kbm-g-np-example
spec:
minMember: 4
---
apiVersion: batch/v1
kind: Job
metadata:
name: allreduce-test
namespace: kbm-g-np-example
labels:
kueue.x-k8s.io/queue-name: training
kueue.x-k8s.io/priority-class: normal
spec:
parallelism: 4
completions: 4
completionMode: Indexed
template:
metadata:
annotations:
scheduling.k8s.io/group-name: allreduce-test
spec:
schedulerName: volcano-scheduler
restartPolicy: Never
containers:
- name: main
image: pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime
command: ["sleep", "600"]
resources:
requests:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"
limits:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"

minMember is the number of Pods that must be scheduled together. It is usually set to the same value as parallelism. For a Deployment or single Pod, specify the annotation and schedulerName in the Pod template in the same way.

If you specify only schedulerName: volcano-scheduler without the annotation, Volcano automatically creates a PodGroup with minMember: 1 for each Pod. This does not provide Gang Scheduling, so always specify the annotation.

Configure KAI directly​

Jobs and JobSets do not require you to create a PodGroup directly. Specify the minimum Gang size with the kai.scheduler/batch-min-member annotation on the resource, and KAI creates the PodGroup. In the Pod template, specify the kai.scheduler/enabled: "true" and kai.scheduler/queue: default-queue labels, and set schedulerName: kai-scheduler.

apiVersion: batch/v1
kind: Job
metadata:
name: allreduce-test
namespace: kbm-g-np-example
labels:
kueue.x-k8s.io/queue-name: training
kueue.x-k8s.io/priority-class: normal
annotations:
kai.scheduler/batch-min-member: "4"
spec:
parallelism: 4
completions: 4
completionMode: Indexed
template:
metadata:
labels:
kai.scheduler/enabled: "true"
kai.scheduler/queue: default-queue
spec:
schedulerName: kai-scheduler
restartPolicy: Never
containers:
- name: main
image: pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime
command: ["sleep", "600"]
resources:
requests:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"
limits:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"

For a Job, if kai.scheduler/batch-min-member is not set, KAI creates a PodGroup with minMember: 1, so Gang Scheduling is not applied. Always specify the annotation. Its value is usually the same as parallelism.

For a JobSet, the annotation's meaning depends on its location. JobSet.metadata.annotations specifies how many replicatedJobs must start at the same time. replicatedJobs[].template.metadata.annotations specifies how many Pods in that replicatedJob must be placed at the same time. If the annotation is omitted, JobSet uses the Job's parallelism as the default.

To group a non-Job resource, such as a Deployment or single Pod, into a Gang, create a PodGroup directly and attach the Pods to it. Since a PodGroup would otherwise be created for each Deployment Pod, add the kai.scheduler/skip-podgrouper: "true" annotation to the resource to prevent KAI from creating PodGroups. Then specify the PodGroup name in the pod-group-name annotation in the Pod template.

apiVersion: scheduling.run.ai/v2alpha2
kind: PodGroup
metadata:
name: serving-group
namespace: kbm-g-np-example
spec:
minMember: 2
queue: default-queue
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: serving-group
namespace: kbm-g-np-example
annotations:
kai.scheduler/skip-podgrouper: "true"
spec:
replicas: 2
selector:
matchLabels:
app: serving-group
template:
metadata:
labels:
app: serving-group
kai.scheduler/enabled: "true"
kueue.x-k8s.io/queue-name: serving
kueue.x-k8s.io/priority-class: normal
annotations:
pod-group-name: serving-group
spec:
schedulerName: kai-scheduler
containers:
- name: main
image: docker.io/vllm/vllm-openai:v0.28.0
resources:
requests:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"
limits:
cpu: "8"
memory: 32Gi
nvidia.com/gpu: "8"

Check status​

If a workload remains Pending, first determine whether it is waiting for Kueue admission or scheduler placement.

kubectl get workloads -n kbm-g-np-example
kubectl describe workload -n kbm-g-np-example <workload-name>

If the Workload's Admitted condition is False, it is waiting in Kueue because the tenant quota is insufficient. Pods have not yet been created for Jobs and TrainJobs. Pods for Deployments, LeaderWorkerSets, and DynamoGraphDeployments appear in the SchedulingGated state.

If Admitted is True but a Pod is Pending, the workload is at the Gang placement stage. Check the PodGroup.

kubectl get podgroups.scheduling.run.ai -n kbm-g-np-example
kubectl describe podgroups.scheduling.run.ai -n kbm-g-np-example <podgroup-name>
kubectl get podgroups.scheduling.volcano.sh -n kbm-g-np-example
kubectl describe podgroups.scheduling.volcano.sh -n kbm-g-np-example <podgroup-name>

For workloads that use KAI (TrainJob with a KAI Runtime, Dynamo, or direct KAI configuration), check scheduling.run.ai. For workloads that use Volcano (TrainJob with a Volcano Runtime, LLMInferenceService, or direct Volcano configuration), check scheduling.volcano.sh. minMember is the required number of Pods. Events in the describe output show why capacity is unavailable.

kubectl get events -n kbm-g-np-example --sort-by=.lastTimestamp
kubectl describe pod -n kbm-g-np-example <pod-name>

Common reasons a Gang cannot be filled:

  • The required number of GPU nodes for minMember are not available at the same time.
  • With a hard topology Runtime, there is not enough room to place all Pods within the same HyperNode (IB spine).
  • Pod creation failed, leaving too few Gang members (for example, a partially overridden worker template, a missing image, or a rejected admission webhook).
  • The Pod did not request nvidia.com/mlnxnics along with its other requests, so the IB network is not attached and the Pod remains in ContainerCreating.

Considerations​

  • Gang Scheduling works only during placement after Kueue admission. Kueue determines priority and preemption with WorkloadPriorityClass; the scheduler does not preempt workloads.
  • Do not add the scheduling.volcano.sh/queue-name annotation or kai.scheduler/queue label to a TrainJob. Adding either causes creation to be rejected. For directly configured Jobs, JobSets, Deployments, and Pods, use only the KAI queue label kai.scheduler/queue: default-queue. Do not use Volcano queue annotations.
  • The training code's world size must match numNodes. Gang Scheduling guarantees the number of Pods, but does not replace the training framework's rendezvous process.

Autoscaling​

Autoscaling overview​

KEDA reads metrics such as request volume or queue length to adjust the number of workload replicas automatically. Create a ScaledObject in your Namespace and specify the scaling target, metrics, and minimum and maximum replica counts. This guide covers traffic-based scaling for Deployments and DynamoGraphDeployments.

OwnerRole
KEDA / HPADetermines the required number of replicas based on metrics
Kueue / resource quotaControls whether additional workloads can run within the tenant resource limit
SchedulerPlaces Pods on nodes

The Prometheus endpoint is http://prom-stack-kube-prometheus-prometheus.monitoring.svc:9090. Specify it in the ScaledObject's serverAddress.

Increasing the number of replicas does not make additional Pods run immediately if GPU resources or quota are insufficient. Existing LocalQueue, scheduler, and resource request settings are still required for workloads.

Scope and permissions​

A ScaledObject is a namespaced resource, and its scaleTargetRef target must be in the same Namespace. You cannot target a Deployment or Dynamo component in another Namespace. User permissions are scoped by the RoleBinding for that Namespace.

First, check your permissions for the Namespace. In the following examples, replace kbm-g-np-example with your Namespace.

kubectl auth can-i create scaledobjects.keda.sh -n kbm-g-np-example
kubectl auth can-i create triggerauthentications.keda.sh -n kbm-g-np-example

If the result is no, ask the operator to configure permissions for that Namespace. You do not need to create a ClusterRole or ClusterRoleBinding yourself.

Configure a Deployment​

Attach a ScaledObject to an existing Deployment. The following example adjusts the number of replicas for a Deployment named inference from 1 to 4 based on the total number of active requests.

inference_active_requests is an example metric name. Update the query to use the metrics and labels actually collected in Prometheus. Check whether the service label exists. The threshold of 8, maximum of 4 replicas, and 300-second scale-down delay are examples and are not recommended values that guarantee model performance.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: inference-autoscaler
namespace: kbm-g-np-example
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: inference
pollingInterval: 30
minReplicaCount: 1
maxReplicaCount: 4
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 1
periodSeconds: 60
triggers:
- type: prometheus
metricType: AverageValue
metadata:
serverAddress: http://prom-stack-kube-prometheus-prometheus.monitoring.svc:9090
query: sum(inference_active_requests{namespace="kbm-g-np-example", service="inference"})
threshold: "8"
ignoreNullValues: "false"

Save the file as scaledobject.yaml, then apply it.

kubectl apply -f scaledobject.yaml
FieldDescription
scaleTargetRef.nameName of the scaling target in the same namespace
minReplicaCount / maxReplicaCountMinimum and maximum number of replicas. The GPU serving example keeps at least one replica running.
pollingIntervalInterval at which KEDA checks the trigger. The HPA polling interval also affects scaling when there is at least one replica.
queryPromQL query that returns the service load as a single value
metricType: AverageValue / thresholdDetermines scaling by dividing total load by the target value per replica
scaleDown.stabilizationWindowSecondsStabilizes scale-down decisions to avoid reducing replicas immediately when load temporarily decreases
scaleDown.policiesIn this example, scales down by at most one Pod every 60 seconds
ignoreNullValues: falseTreats an empty query result as an error rather than as zero load

For example, if the query returns 24 active requests and the threshold is 8, the simple target calculation is 3 replicas. Actual scaling depends on HPA tolerance, stabilization policies, minimum and maximum values, and Pod status. cooldownPeriod applies when scaling down to zero. Because this example keeps at least one replica, it uses HPA behavior to control scale-down delay.

Configure Dynamo​

For Dynamo, use a DynamoGraphDeploymentScalingAdapter as the target instead of directly scaling the generated child Deployment. In the v1beta1 CRD for Dynamo 1.4.2 in production, adding scalingAdapter: {} to a component enables autoscaling, creates a DGDSA, and transfers replica management to the DGDSA.

The following partial example shows only the field to add to the relevant component in an existing DGD. Do not replace the entire components list with this snippet; retain the existing image, resources, podTemplate, and other settings.

spec:
components:
- name: VllmWorker
scalingAdapter: {}

After checking the name of the generated DGDSA, change scaleTargetRef in the ScaledObject above as follows. If you do not have permission to view DGDSAs, ask the operator to check the target name and creation status.

scaleTargetRef:
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentScalingAdapter
name: <generated-DGDSA-name>

Complete example (separate prefill and decode)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: glm-5-3-disagg-prefill
namespace: kbm-g-np-example
spec:
scaleTargetRef:
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentScalingAdapter
name: glm-5-3-disagg-vllmprefillworker
minReplicaCount: 1
maxReplicaCount: 3
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 30
policies:
- type: Pods
value: 1
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 300
triggers:
- type: prometheus
metricType: AverageValue
metadata:
serverAddress: http://prom-stack-kube-prometheus-prometheus.monitoring.svc:9090
query: sum(vllm:num_requests_waiting_by_reason{dynamo_namespace="kbm-g-np-example-glm-5-3-disagg", dynamo_component_type="prefill", reason="capacity"})
threshold: "4"
ignoreNullValues: "false"
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: glm-5-3-disagg-decode
namespace: kbm-g-np-example
spec:
scaleTargetRef:
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentScalingAdapter
name: glm-5-3-disagg-vllmdecodeworker
minReplicaCount: 1
maxReplicaCount: 3
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 30
policies:
- type: Pods
value: 1
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 300
triggers:
- type: prometheus
metricType: AverageValue
metadata:
serverAddress: http://prom-stack-kube-prometheus-prometheus.monitoring.svc:9090
query: sum(vllm:num_requests_waiting_by_reason{dynamo_namespace="kbm-g-np-example-glm-5-3-disagg", dynamo_component_type="decode", reason="capacity"})
threshold: "8"
ignoreNullValues: "false"
  • Create the ScaledObject in the same namespace as the DGDSA. Replace the query used in the Deployment example with the metrics collected for the actual Dynamo model and component.
  • Start with a minReplicaCount of at least 1 for Dynamo workers. This guide does not cover scaling Dynamo down to zero.
  • Do not control the replicas of a component that uses DGDSA with a manual setting or another autoscaler at the same time. If you already use Planner, do not attach KEDA to the same target.
  • For multi-node components, one replica consists of multiple Pods and uses multiple GPUs. Set maxReplicaCount based on the total GPU requirement per replica and the tenant quota.

Check status​

kubectl get scaledobjects.keda.sh -n kbm-g-np-example
kubectl describe scaledobject inference-autoscaler -n kbm-g-np-example
kubectl get hpa -n kbm-g-np-example
kubectl describe hpa keda-hpa-inference-autoscaler -n kbm-g-np-example
kubectl get deployment inference -n kbm-g-np-example
kubectl get events -n kbm-g-np-example --sort-by=.lastTimestamp
StatusWhat to check
ScaledObject Ready=FalseTarget name and API version, Prometheus connection, query errors, authentication settings, and events
Active=FalseThe trigger may be inactive. This is not necessarily an error; check Ready and events as well.
HPA TARGETS shows unknownCheck metric query errors and HPA events.
Desired replica count increased, but Ready Pods did notCheck quota, Kueue Workload, Pod events, GPU availability, and model loading status.
ForbiddenCheck your ScaledObject permissions in your namespace. If the controller event shows Forbidden, ask the operator to check KEDA service account permissions.

Pause and resume autoscaling​

To temporarily pause KEDA autoscaling while keeping the current number of replicas, use the paused annotation.

kubectl annotate scaledobject inference-autoscaler -n kbm-g-np-example autoscaling.keda.sh/paused=true --overwrite

Remove the annotation to resume autoscaling.

kubectl annotate scaledobject inference-autoscaler -n kbm-g-np-example autoscaling.keda.sh/paused-

To remove autoscaling, delete the ScaledObject. With the default settings, the original replica count is not restored automatically after deletion. Check and set the desired fixed replica count separately.

kubectl delete scaledobject inference-autoscaler -n kbm-g-np-example

Reference metrics​

TypeLink
vLLMvLLM Grafana Dashboard Metrics
DynamoNVIDIA Dynamo Metric Labels
SGLangSGLang Production Metrics
EnvoyEnvoy Configuration Stats

Configure metrics​

When running vLLM or SGLang in separate Pods or Deployments without Dynamo, configure the labels and container port name on the actual inference server Pod as shown below. The platform Prometheus scrapes /metrics directly from the Pod IP, so you do not need to create a separate Service, Ingress, ServiceMonitor, or PodMonitor just for metrics collection.

vLLM configuration example​

The following shows some metrics-related fields to merge into an existing Deployment.

For example, if the server is running on port 8000:

vllm serve <model-ID-or-path> --host 0.0.0.0 --port 8000

Apply the following settings to the relevant container in the Deployment.

spec:
template:
metadata:
labels:
natl.kakaocloud.com/endpoint.serving-type: vllm
spec:
containers:
- name: vllm
ports:
- name: modelserver
containerPort: 8000
protocol: TCP

If a port is already declared, set its name to modelserver. If the existing Service's targetPort or probes refer to the previous port name, update those as well.

SGLang configuration example​

Add --enable-metrics to the existing SGLang server arguments. For example, to use port 30000, run:

python3 -m sglang.launch_server --model-path <model-ID-or-path> --host 0.0.0.0 --port 30000 --enable-metrics

Merge the following fields into the existing Deployment's Pod template. Use the actual container name.

spec:
template:
metadata:
labels:
natl.kakaocloud.com/endpoint.serving-type: sglang
spec:
containers:
- name: sglang
ports:
- name: modelserver
containerPort: 30000
protocol: TCP

modelserver is a fixed port name, while 8000 and 30000 are example port numbers. If you use another port number, set containerPort to the port that serves /metrics. Declaring containerPort does not change the server's listening port or enable metrics.

The server must listen on an address reachable through the Pod IP. Prometheus cannot collect metrics if the server is bound only to 127.0.0.1.