Manage node

A node is a server where container applications run, and in Kubernetes Engine, nodes are managed in node pools, which are groups of nodes with the same instance type.

Create and manage node pools

Here’s how to manage node pools in the Kubernetes Engine service.

Create node pool

After creating a cluster, you can create a node pool.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.

2. In the Cluster menu, select the cluster to create a node pool.

3. On the cluster details page, click the Node pool tab, then click the [Create node pool] button.

4. In Create node pool, enter the necessary information and click the Create button.

   

   Item	Description
   Node pool type	Select the type of node pool to create - Bare Metal Server type is supported only in the kr-central-2 region
   Basic settings	Set the basic information for the node pool - Node pool name: Enter the name of the node pool - Node pool description (optional): Enter a description within 60 characters
   Image	Select one image to use for the nodes - Available images vary depending on the node pool type
   Instance type	Select the instance type for the node pool ⚠️ For proper Kubernetes Engine service usage, use instances with at least 1 GiB
   Volume	Configure the volume type and size for the instances - Currently, the volume type is fixed as SSD, and the size can be set from 30 to 5,120GB ⚠️ Volume settings are not available for Bare Metal Server type node pools
   Number of nodes	Set the number of nodes in the node pool
   Node pool network settings	Select the VPC and subnets where nodes will run - VPC: Same as the cluster's VPC and cannot be configured separately - Subnet: Select subnets for node execution from those chosen during cluster creation   ㄴ For the kr-central-2 region supporting Multi-AZ, enhance availability by selecting multiple subnets in different availability zones for the node pool's network. - Security group (optional): Security group to apply to nodes
   Key pair	Configure the use of a key pair for SSH access to the node instances in the node pool   ᄂ Select an existing key pair or create a new key pair   ᄂ To create a new key pair: Click Create key pair, enter a key pair name, and click Create and download to download the .pem file - The key pair assigned to the node pool is not displayed in the instance details - The key pair cannot be changed after cluster creation; if needed, create a new node pool to apply new settings
   Network bonding	If creating a Bare Metal Server type node pool, network bonding is automatically applied - Network bonding sets two interfaces with the same IP on each node in the pool based on the subnet selected in the network settings - Only a single availability zone is supported
   Advanced settings (optional)	Set advanced configurations for the node pool - Node label (optional): Specify Kubernetes labels to be applied to all nodes in the pool, which can be used with nodeSelector - Node taint (optional): Specify Kubernetes taints to be applied to all nodes in the pool, which can be used with toleration - CPU multithreading: An option to optimize performance by specifying single threads per CPU core ᄂ some instance types require CPU multithreading   ᄂ Disable for high-performance computing (HPC) workloads ⚠️ Multithreading settings are not available for Bare Metal Server type node pools - User script (optional): Input a shell script to be executed when nodes in the pool are created   ᄂ Use this for additional node configuration, up to 16KB, and note that user scripts cannot be modified after configuration

Configure node pool

Check or modify information and the number of nodes in a node pool.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.

2. Select the cluster with the node pool settings to modify from the Cluster menu.

3. On the cluster detail page, go to the Node pool tab, then click the [More] icon > Configure node pool.

4. Review the information in the popup. If needed, modify the settings and click [Save].

   Item	Description
   Node pool information	Information of the node pool - Node pool name: Cannot be changed - Node pool description (optional): Review or modify the current description within 60 characters
   Number of nodes	The current number of nodes in the node pool - Node count can be changed.

View node pool details

View detailed information about the node pool and the nodes within it.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.

2. Select the cluster from the Cluster menu that contains the node pool to view.

3. On the cluster detail page, go to the Node pool tab, then select the node pool to view.

4. Review the detailed information of the node pool.

   Tab	Item	Description
   Details	Key pair	Information about the key pair assigned to the node pool
   	Created at	The creation date of the node pool
   	Volume type	Volume type settings for the node pool
   	Image	Detailed information about the image installed on the nodes
   	Node label	Labels applied to nodes in the node pool
   	Node taint	Taints applied to nodes in the node pool
   	User script	User script applied to the nodes in the node pool
   Network	VPC	Cluster's VPC information
   	Subnet	Subnet information where the node pool's nodes are running
   	Security group	Security group assigned to the node pool's nodes
   Scaling	Resource-based auto scaling	Create and manage Resource-based auto scaling policies
   	Schedule-based auto scaling	Create and manage schedule-based auto scaling policies - View schedule-based auto scaling events
   Node	Node	Information about the nodes - Click on the node name to view detailed node information
   	Node status	Node status information - Running: The node is ready and running - Running (Scheduling Disable): The node is no longer scheduling new pods (pods already assigned are still running) - Provisioned: Node provisioning is complete - Deleted: Node deletion is complete - Pending: Node provisioning is in progress - Provisioning: Node is being provisioned - Deleting: Node is being deleted - Failed: Requires user intervention
   	Node pool	Information about the node pool the node belongs to
   	Private IP	The private IP of the node
   	Availability zone	The availability zone of the subnet where the node is running
   	Uptime	The total time elapsed since the node request was made, not the node’s creation date

Configure node labels

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.

2. Select the cluster from the Cluster menu that contains the node pool.

3. On the cluster detail page, go to the Node pool tab, then click the Set node label button in the details.

4. Enter the Key and Value, then click [Save]. The labels will be applied to all nodes in the node pool.

   Item	Description
   Key	A key for identifying the label, up to 50 per node
   Value	The value of the label
   [Trash icon]	Click to delete the label

info

Keywords reserved by KakaoCloud or Kubernetes cannot be used as label keys.

Configure user script

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.
2. Select the cluster from the Cluster menu that contains the node pool.
3. On the cluster detail page, go to the Node pool tab, then click the Set user script button in the details.
4. Upload or enter the User script and click [Save].

info

The user script will only apply to newly created nodes after it is set.

Delete node pool

You can delete a node pool that is no longer needed.

caution

Deleting a node pool will delete all nodes in the pool, and this action cannot be undone.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.
2. Select the cluster from the Cluster menu where the node pool is located.
3. Go to the Node pool tab, click the [More] icon next to the node pool, and select Delete node pool.
4. Enter the information in the Delete node pool popup and click [Delete].

Configure resource-based autoscaling

Resource-based autoscaling is a feature that automatically scales the number of nodes in a node pool based on resource usage. When pods cannot be scheduled due to insufficient available resources on nodes, the number of nodes is automatically increased. Conversely, if resource usage remains below a certain threshold, the number of nodes is automatically decreased.
This feature is based on the Kubernetes project's officially supported Cluster Autoscaler functionality. For more details, refer to Cluster Autoscaler.

info

• Node pools with Bare Metal Server types do not support the autoscaling feature.

caution

• Resource-based autoscaling operates based on the request values defined for pod resources.
  
• If the request values for pod resources are not defined, the autoscaling feature will not function.

1. Go to the KakaoCloud Console and navigate to Container Pack > Kubernetes Engine.

2. In the Cluster menu, select the cluster for which you want to configure autoscaling.

3. On the cluster details page, go to the Node Pool tab and select the node pool to configure.

4. In the Node Pool details page, switch to the Scaling tab and click the [Configure resource-based autoscaling] button.

5. In the Resource-based autoscaling configuration popup, review the information. To modify settings, enter the desired values and click the [Save] button.

   Category	Item	Description
   Resource-based autoscaling	Node count	Current number of nodes in the node pool - Node count can be changed.
   	Minimum node count	Minimum number of nodes allowed during automatic scaling down
   	Maximum node count	Maximum number of nodes allowed during automatic scaling up
   Automatic scaling-down rules	Scaling-down threshold condition	Threshold for node resource usage (CPU/Memory) that triggers automatic scaling down - Range: 1 ~ 100 - Default: 50%
   	Threshold duration	Time for which resource usage must remain below the scaling-down threshold condition - Range: 1 ~ 86400 seconds, 1 ~ 1440 minutes - Default: 10 minutes
   	Exclusion time for scaling-down monitoring	Time during which newly added nodes are excluded from scaling-down monitoring after being scaled up - Range: 1 ~ 86400 seconds, 1 ~ 1440 minutes - Default: 10 minutes

Node pool scaling management

info

• The previous node pool autoscaling setting is now called Resource-based auto scaling.
• Both Resource-based auto scaling and schedule-based auto scaling are available. You can only set one type of autoscaling at a time. ᄂ Future improvements will allow both autoscaling types to be used simultaneously.

Configure resource-based auto scaling

Resource-based auto scaling adjusts the number of nodes based on the resource usage of the node pool. If available resources are insufficient to schedule pods, nodes are automatically added, and when resource utilization remains low, nodes are automatically removed.
This feature is based on the Kubernetes Cluster Autoscaler project.

info

• The autoscaling feature is not available for Bare Metal Server type node pools.

caution

• Resource-based auto scaling operates based on the request value defined for pod resources.
  
• If pod resource request values are not defined, the autoscaling feature will not function.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.

2. Select the cluster from the Cluster menu for autoscaling configuration.

3. On the cluster detail page, go to the Node pool tab and select the node pool.

4. In the Scaling tab of the Node pool details, click the Resource-based auto scaling button.

5. Review the information in the popup. Modify the settings as needed and click [Save].

   Category	Item	Description
   Resource-based auto scaling	Number of nodes	Current number of nodes in the node pool
   	Minimum number of nodes	Minimum number of nodes when automatically scaling down
   	Maximum number of nodes	Maximum number of nodes when automatically scaling up
   Auto-scaling rules	Threshold condition for scaling down	Threshold for node resources (CPU/Memory) usage to trigger automatic scaling down - Allowed range: 1–100 - Default: 50%
   	Threshold duration	Duration for which node resource usage remains below the scaling-down threshold condition - Allowed range: 1–86400 seconds, 1–1440 minutes - Default: 10 minutes
   	Monitoring exclusion time for scaling down	Time excluded from monitoring nodes for scaling down after scaling up - Allowed range: 1–86400 seconds, 1–1440 minutes - Default: 10 minutes

Configure HPA and load testing

You can set up HorizontalPodAutoscaler (HPA) with Cluster Autoscaler for more efficient resource management. This example demonstrates how to test if automatic scaling works as expected by applying HPA.

info

HPA monitors resource usage (such as CPU) and automatically adjusts the number of pods in a workload (such as Deployment, StatefulSet). For more details, refer to the Kubernetes documentation on HPA.

1. Install Helm client before configuring HPA. For installation instructions by operating system, refer to the Helm official documentation > Installing Helm.

2. Install metrics-server to monitor pod metrics for HPA load testing.

   • Add the metrics-server chart repository, then install metrics-server.
     
   metrics-server installation command

   helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
   helm upgrade --install metrics-server metrics-server/metrics-server --set hostNetwork.enabled=true --set containerPort=4443

3. Check if the resource usage of the nodes is being monitored correctly. It may take up to 5 minutes to collect monitoring information after installing the metrics-server.

   Check node resource usage

   kubectl top node

4. Deploy the php-server for load testing after configuring HPA and Cluster Autoscaler.

   php-server App deployment command

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: php-apache
   spec:
     selector:
       matchLabels:
         run: php-apache
     template:
       metadata:
         labels:
           run: php-apache
       spec:
         containers:
         - name: php-apache
           image: ike-controlplane-provider.kr-central-1.kcr.dev/ike-cr/hpa-example:latest
           ports:
           - containerPort: 80
           resources:
             limits:
               cpu: 500m
             requests:
               cpu: 500m
   
   --- 
   apiVersion: v1
   kind: Service
   metadata:
     name: php-apache
     labels:
       run: php-apache
   spec:
     ports:
     - port: 80
     selector:
       run: php-apache
   Check deployment of php-server app

   kubectl apply -f php-apache.yaml

5. Create an HPA for load testing.

   Create HPA

      kubectl autoscale deployment php-apache --cpu-percent=10 --min=1 --max=10  // Create HPA
      kubectl get hpa  // Check HPA settings
   Result

      NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
      php-apache   Deployment/php-apache   46%/50%   1         10        5          28m

6. Run a load-generating pod to test the HPA and auto-scaling configuration.

   Execute pod

    kubectl run -i --tty load-generator --rm --image=ike-controlplane-provider.kr-central-1.kcr.dev/ike-cr/busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- HTTP://php-apache; done"

7. Confirm that the number of pods and nodes increases appropriately as the load increases.

   • The HPA for the php-apache server is triggered, and some pods that cannot be scheduled due to insufficient resources enter the Pending state.

   • Since some pods cannot be scheduled, the number of nodes is automatically scaled to three to add resources.

     Check the results of the HPA and auto-scaling operations.

     kubectl get pods -w  # Check for changes in the number of pods
     kubectl get nodes -w # Check for changes in the number of nodes
     Result

     NAME                             READY   STATUS    RESTARTS      AGE
     php-apache-766d5cdd5b-2t5p8      0/1     Pending   0             44s
     php-apache-766d5cdd5b-5mhlk      0/1     Pending   0             29s
     php-apache-766d5cdd5b-5vjt6      0/1     Pending   0             14s
     php-apache-766d5cdd5b-74z87      1/1     Running   0             44s
     php-apache-766d5cdd5b-d49g9      0/1     Pending   0             29s
     php-apache-766d5cdd5b-fnlld      1/1     Running   0             44s
     php-apache-766d5cdd5b-nr5f2      0/1     Pending   0             29s
     php-apache-766d5cdd5b-t7zr8      0/1     Pending   0             29s
     php-apache-766d5cdd5b-vjjlz      1/1     Running   0             2m49s
     php-apache-766d5cdd5b-whjhw      0/1     Pending   0             14s
     
     NAME                STATUS     ROLES    AGE   VERSION
     host-10-187-5-177   Ready      <none>   51s    v1.24.6
     host-10-187-5-189   Ready      <none>   9m5s   v1.24.6
     host-10-187-5-98    Ready      <none>   69s    v1.24.6

8. When a node is added, all pods that were in the Pending state are changed to the Running state.

Set scheduled-based auto scaling

Scheduled-based auto scaling is a feature that adjusts the number of nodes in a node pool at scheduled times. By using scheduled-based auto scaling, you can automate the adjustment of the number of nodes in the node pool based on predicted increases or decreases in traffic. For example, workloads that regularly experience a pattern of decreased traffic on weekends and increased traffic during the week can be managed by creating rules such as the following:

• Scheduled-based auto scaling
  • Rule 1: Every Monday at 8:30 AM | Maximum node count to increase
  • Rule 2: Every Friday at 7:30 PM | Minimum node count to decrease

This allows the application to have enough nodes to handle peak traffic during the week and reduce unnecessary nodes during the relatively low traffic on weekends. This scheduled-based auto scaling can optimize costs and performance.

info

• Up to 2 scheduling rules can be created for scheduled-based auto scaling.
• Since a maximum of 2 rules can be used, it is recommended to have one rule for scaling up at a specific time and another rule for scaling back to the original number of nodes.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.
2. Select the cluster where to check the node pool settings in the Cluster menu.
3. Choose the node pool to configure in the Node pool tab on the cluster's detail page.
4. In the Node pool details, click the Create rule button under the Scaling tab in the scheduled-based auto scaling section.
5. In the Create rule pop-up window, enter the rule name and description, then click the [Create] button.

   Item	Description
   Name	The name of the scheduled-based auto scaling rule
   Rule	Desired number of nodes - Set the desired number of nodes at the specified time
   Start	The start date and time when the rule operates - The recurrence points are determined based on the start date and time.
   Scheduled execution date	Displays the nearest scheduled date based on the start date and time.

Set recurrence

To create a recurring schedule, you must select [Daily], [Weekly], or [Monthly] in the rule creation. The detailed explanations of the currently available recurrence periods are as follows:

Recurrence Period	Description
Once	Performs once at the start date and time
Daily	Repeats daily based on the time of the start date - Start Date: 2024/05/01 (Wed) 10:00 - Recurrence Point: Daily at 10:00
Weekly	Repeats weekly based on the day of the week and time of the start date - Start Date: 2024/05/01 (Wed) 10:00 - Recurrence Point: Every Wednesday at 10:00
Monthly	Repeats monthly based on the day and time of the start date - Start Date: 2024/05/01 (Wed) 10:00 - Recurrence Point: Every 1st of the month at 10:00

Delete scheduled-based auto scaling

To delete scheduled-based auto scaling, follow these steps:

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.
2. Select the cluster where to configure the auto-scaling settings in the Cluster menu.
3. Choose the node pool to configure in the Node pool tab on the cluster's detail page.
4. In the Node pool details, click the Trash can button in the scheduled-based auto scaling rules list under the Scaling tab.
5. In the pop-up window, enter the rule name and click the [Delete] button.

info

• If the node pool's status is changing, you cannot delete the schedule conditions.
• Changing states: ScalingUp, ScalingDown, Updating

Check scheduled-based auto scaling events

You can check the results of scheduled-based auto scaling in the event list. The activities adjusted through scheduled-based auto scaling will retain a maximum of 20 logs per rule. Scheduled-based auto scaling events can also be verified through the Cloud Trail service.

info

• If the status of scheduled-based auto scaling drops to Failed, it indicates that there was an internal issue during auto-scaling, causing the node pool's status to drop to Failed.
• This may be due to exceeding the node pool quota, timeouts during capacity adjustments, etc. For events that customers cannot resolve, please contact the Helpdesk.
• When a rule is deleted, the associated events will also be deleted.
  ᄂ Logs recorded in the Cloud Trail service will be retained.

Category	Description
Event time	The time when the rule was executed
Rule name	The name of the executed rule
Result	The result of the executed rule - [Success], [Failure] Detailed event results can be checked in a pop-up.

Manage nodes

The following methods are used to manage nodes in the Kubernetes Engine service.

View node details

You can check detailed information about a node.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.

2. Select the cluster where the node pool to check is located from the Cluster menu.

3. Click on the Node tab on the cluster's detail page and select the node to check.

   • Alternatively, select the node from the Node pool > Node tab to view the details.

4. Check the information in the Details tab.

   Category	Description
   Instance ID	The VM instance ID information corresponding to the node - Clicking on the instance ID will take you to the VM instance page
   Instance type	The VM instance type corresponding to the node and the node pool type information
   Created at	The creation date of the VM instance corresponding to the node
   Kubernetes version	The Kubernetes version of the node
   Availability Zone	Information about the availability zone of the subnet where the node is running
   Volume	Information about the volume attached to the VM instance corresponding to the node
   Key pair	Information about the key pair set for the node - The key pair specified through the node pool will not be exposed in the instance details
   Private IP	The private IP information of the node
   Image	Detailed information about the image installed on the node
   CPU Multithreading	Whether the CPU multithreading feature is enabled
   Summary of node	Provides performance and status information about the node, and when the [Refresh] icon is clicked, the latest information of the node is updated - Pod: Information about the pods currently running on the node - Node condition: Detailed status information about the current node - Taint: Taint information set on the current node - Label: Label information set on the current node - Annotation: Annotation information set on the current node - Allocatable resource: Current status of allocatable resources on the node - Event: Event information that has occurred on the current node

Node monitoring

You can check monitoring information such as resource usage and trends of nodes in chart format for a specific period.

info

To monitor nodes in Kubernetes Engine, node-exporter is installed on port 59100 of the node. Please note that port 59100, where node-exporter is installed, cannot be used separately.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.
2. Select the cluster where the node pool to check is located from the Cluster menu.
3. Click on the Node tab on the cluster's detail page and select the node to check.
   • Alternatively, select the node from the Node pool > Node tab to view the details.
4. In the Monitoring tab, select the period and Node/Pod, and check the information.

Node monitoring

When you select Node, you can check the resource usage and trend information of that node.

Category	Description
CPU usage (millicores)	The CPU usage of the node
Memory usage (Bytes)	The memory usage of the node
Disk usage (Bytes)	The disk usage of the node
RX Network (byte/s)	The number of bytes received over the network by the node
TX Network (byte/s)	The number of bytes sent over the network by the node
Reserved CPU Capacity (%)	The percentage of CPU reserved for the node's components
Reserved Memory Capacity (%)	The percentage of memory reserved for the node's components
Pods (Count)	The number of pods running on the node
Containers (Count)	The number of containers running on the node

Pod monitoring

When you select Pod, you can check the resource usage and trend information of the pods running on the node.

Category	Description
CPU usage (millicores)	The CPU usage of the pod
Memory usage (Bytes)	The memory usage of the pod
RX Network (byte/s)	The number of bytes received over the network by the pod
TX Network (byte/s)	The number of bytes sent over the network by the pod
Reserved CPU Capacity (%)	The percentage of CPU reserved for the pod
Reserved Memory Capacity (%)	The percentage of memory reserved for the pod

Recreate nodes

You can select and recreate nodes regardless of their current status.

caution

When recreating a node, the selected node will be drained, a new node will be created, and the existing node will be deleted. This process may affect running services, and the deleted node cannot be recovered. Additionally, note that the IP address of the new VM associated with the recreated node will change. Proceed with caution.

1. Go to the KakaoCloud Console and navigate to Container Pack > Kubernetes Engine.
2. In the Cluster menu, select the cluster containing the node pool with the node you want to recreate.
3. On the cluster details page, go to the Node tab, click the [More] icon for the node to be restored, and select Recreate node.
4. In the Recreate node popup, enter the required information and click the [Recreate] button.

Update nodes

When the cluster has been updated to the latest Kubernetes version or when the latest node component updates can use the latest images, you can update the nodes.

When executing a node update, a rolling update will be performed as follows:

1. A new node with the latest image version is created.

2. Pods running on the existing node are evicted, and the node is switched to an unschedulable state.

3. The evicted pods run on the new node, and once evictions are complete, the existing node is deleted.

4. This process is repeated sequentially for all existing nodes.

Considerations before updating

If any of the following conditions are not met, the update will not proceed.

Condition	Description
Cluster status	Provisioned status - If in any other status, the update button will not be displayed
Node pool status	Running status - If in any other status, the update button will not be displayed

Update procedures

If the node pool meets the conditions to start the update, you can update the node pool.

caution

Once the update starts, it cannot be canceled, and you cannot revert to the previous state.

1. Go to KakaoCloud Console > Container Pack > Kubernetes Engine.
2. Select the cluster where the node pool to delete is located from the Cluster menu.
3. Click on the Node pool tab, and then click the Kubernetes version > [Update] button for the node pool to update.
4. In the pop-up window, check the information and click the [Update] button.
5. Once the node pool update starts, the status of the node pool will change to Updating. Once the node pool update is complete, it will change to Running. During the update, adding new node pools and configuring existing node pools will not be possible.

Check for update failures

During the rolling update, if the eviction of pods fails due to PDB (Pod Disruption Budget) settings, the update may fail. In case of an update failure, you can try the following methods. For more detailed explanations, refer to the official Kubernetes documentation.

• Modify the Min Available and Max Unavailable values of the PDB to successfully evict the pods. Be aware that if the Max Unavailable value is 0, the eviction of nodes for updates will fail.
• Back up the PDB, then delete it. After the update is complete, reset the PDB.
• If the pods are managed by a Deployment, StatefulSet, etc., and the number of pods is adjusted through a ReplicaSet, eviction of pods may fail. In this case, back up and delete the Deployment, StatefulSet, etc., in advance.
• Additionally, you can find guidelines for safely draining nodes in the official Kubernetes documentation: Safely Drain a Node.
• Node pool updates are conducted in a rolling update manner, and you must be able to create the same number of nodes as the current nodes. Therefore, if there are insufficient VM and IaaS resources available for your project, the update may fail.
• If a node transitions to the Failed status during the update, making the Updating status Pending, you can proceed with Node recovery. Once the node is recovered, the update will proceed normally again. If the node pool remains in the Updating state for an extended period, please contact the Helpdesk > Technical Support.