Install and manage Node Problem Detector

Node Problem Detector runs on each worker node and detects problems in node components, including the kernel and container runtime. It reports detected problems to the Kubernetes API server as events or node conditions, which you can inspect with kubectl get events and kubectl describe node. For more information, see the official Node Problem Detector documentation.

Node Problem Detector operates independently of Node Local DNS Cache. If you need node-level DNS caching, see Install and manage Node Local DNS Cache.

Node Problem Detector does not replace node monitoring, which uses node-exporter to collect node CPU and memory usage.

info

• Kubernetes 1.32 and 1.33: You can install and remove Node Problem Detector as needed by following this guide. The installed instance is a user-managed Helm release.
• Kubernetes 1.34 or later: Node Problem Detector is installed by default and managed by Kubernetes Engine. Do not install another instance or delete it manually. Because it is not managed as a Helm release, it might not appear in helm list output. If you delete the DaemonSet, automatic recovery might be delayed. Check its status, and then contact the Helpdesk if necessary.
• If you installed Node Problem Detector by following this guide on a Kubernetes 1.32 or 1.33 cluster, remove the Helm release as described in Step 5. Uninstall Node Problem Detector before upgrading to Kubernetes 1.34 or later. If the existing release remains, the user-installed and default instances can run simultaneously, causing the same problem to be reported multiple times or causing duplicate or conflicting events and node conditions.

Step 1. Prerequisites

The installation procedure in this guide applies only to Kubernetes 1.32 and 1.33 clusters. For Kubernetes 1.34 or later, do not perform the installation procedure. See Step 3. Verify the deployment and Step 4. Monitor operations.

Prepare the cluster and tools

1. Create a Kubernetes Engine cluster in which to install Node Problem Detector.
2. Configure kubectl so that you can run commands against the cluster.
3. Install Helm 3 by following the official Helm documentation.

Set the kubeconfig for the target cluster as an environment variable.

Set kubeconfig

export KUBE_CONFIG=/path/to/cluster-kubeconfig.yaml

Check the Kubernetes version of the cluster.

Check the Kubernetes version

kubectl --kubeconfig=$KUBE_CONFIG version

Set environment variables for the Kubernetes minor version. The following example is for a Kubernetes 1.33 cluster.

Set the Helm chart version

export K8S_MINOR=1.33
export CHART_VERSION=${K8S_MINOR}.0

Kubernetes version	Helm chart version
1.32	1.32.0
1.33	1.33.0

The Node Problem Detector Helm chart is provided through the KakaoCloud public OCI registry. You can install it without registering a separate Helm repository.

Check for an existing deployment and registry access

Check whether Node Problem Detector is already installed.

Check for an existing deployment

helm --kubeconfig=$KUBE_CONFIG list -n kube-system -a

kubectl --kubeconfig=$KUBE_CONFIG get daemonset -n kube-system \
    | grep -E 'node-problem-detector' || true

If a deployment exists, do not install a duplicate. First identify the Helm release name, applied values, and managing entity. Deploying multiple instances can cause the same problem to be reported more than once.

The helm list output can include system releases managed by Kubernetes Engine, such as ke-cilium and ke-tigera-operator. Do not modify or delete system releases.

Verify that worker nodes can access the following registry.

ke-container-registry.kr-central-2.kcr.dev

Before deploying to a production cluster, test installation and rollback in a development or staging cluster that uses the same Kubernetes version, CNI, node OS, and admission policies.

Step 2. Install Node Problem Detector

This procedure applies only to Kubernetes 1.32 and 1.33 clusters. Do not run it on Kubernetes 1.34 or later.

Review the Helm chart defaults and the resources that the chart will create.

Check the Helm chart defaults

helm show values \
    oci://ke-container-registry.kr-central-2.kcr.dev/ke-helm-public/node-problem-detector \
    --version $CHART_VERSION

Validate Node Problem Detector before installation

helm template node-problem-detector \
    oci://ke-container-registry.kr-central-2.kcr.dev/ke-helm-public/node-problem-detector \
    --version $CHART_VERSION \
    --namespace kube-system \
    | kubectl --kubeconfig=$KUBE_CONFIG apply --dry-run=server -f -

By default, Node Problem Detector uses a privileged container, hostPath volumes, a ServiceAccount, and RBAC resources. If the cluster uses Pod Security Admission, Kyverno, or Gatekeeper policies, review the dry-run results and apply the required policies first.

After validation succeeds, install Node Problem Detector.

Install Node Problem Detector

helm --kubeconfig=$KUBE_CONFIG upgrade --install node-problem-detector \
    oci://ke-container-registry.kr-central-2.kcr.dev/ke-helm-public/node-problem-detector \
    --version $CHART_VERSION \
    --namespace kube-system \
    --wait \
    --timeout 5m

Step 3. Verify the deployment

Kubernetes 1.32 and 1.33

Check the status of the Helm release installed by following this guide.

Check the Helm release status

helm --kubeconfig=$KUBE_CONFIG status node-problem-detector -n kube-system

Verify that the DaemonSet and pods are running normally on every target worker node.

Check the DaemonSet and pods

kubectl --kubeconfig=$KUBE_CONFIG get daemonset \
    node-problem-detector -n kube-system

kubectl --kubeconfig=$KUBE_CONFIG get pod -n kube-system -o wide \
    -l app.kubernetes.io/instance=node-problem-detector

kubectl --kubeconfig=$KUBE_CONFIG rollout status \
    daemonset/node-problem-detector -n kube-system

Kubernetes 1.34 or later

Node Problem Detector on a Kubernetes 1.34 or later cluster is not managed as a Helm release. Skip the Helm status check and inspect the DaemonSet managed by Kubernetes Engine.

Check the platform-managed DaemonSet

kubectl --kubeconfig=$KUBE_CONFIG get daemonset -n kube-system \
    | grep node-problem-detector

Verify node events and conditions

Check the node events and conditions reported by Node Problem Detector. Replace {node-name} with an actual worker node name.

Check node events

kubectl --kubeconfig=$KUBE_CONFIG get events -A \
    --field-selector involvedObject.kind=Node

Check node conditions

kubectl --kubeconfig=$KUBE_CONFIG describe node {node-name}

The deployment is operating normally when all of the following conditions are met:

• The DaemonSet Ready count matches the number of target worker nodes.
• Worker nodes remain in the Ready state.
• Node Problem Detector does not generate incorrect node conditions or excessive events.
• On Kubernetes 1.32 and 1.33, the user-installed Helm release is in the deployed state.

Step 4. Monitor operations

Monitor the following items after deployment:

• Changes to the Ready state of worker nodes
• Events and node conditions generated by Node Problem Detector
• Abnormal increases in Kubernetes events
• Restart counts and error logs for Node Problem Detector pods

If you installed Node Problem Detector by following this guide on Kubernetes 1.32 or 1.33, use the following commands to inspect the applied values and logs.

Check the applied values and logs

helm --kubeconfig=$KUBE_CONFIG get values node-problem-detector \
    -n kube-system -a

kubectl --kubeconfig=$KUBE_CONFIG logs -n kube-system \
    -l app.kubernetes.io/instance=node-problem-detector \
    --tail=100

If a problem persists with the Kubernetes Engine-managed Node Problem Detector on Kubernetes 1.34 or later, inspect the DaemonSet, node conditions, and events, and then contact the Helpdesk.

Step 5. Uninstall Node Problem Detector

This procedure applies only to a Helm release installed by following this guide on Kubernetes 1.32 or 1.33. You cannot use this procedure to remove the default Node Problem Detector on Kubernetes 1.34 or later.

Identify the Helm release to remove. Because helm list displays releases only in the current namespace, specify the kube-system namespace.

Identify the Helm release to remove

helm --kubeconfig=$KUBE_CONFIG list -n kube-system -a \
    | grep -E '^node-problem-detector[[:space:]]'

If necessary, back up the current values before removal.

Back up the applied values

helm --kubeconfig=$KUBE_CONFIG get values node-problem-detector \
    -n kube-system -a -o yaml > node-problem-detector-values-backup.yaml

Uninstall Node Problem Detector.

Uninstall Node Problem Detector

helm --kubeconfig=$KUBE_CONFIG uninstall node-problem-detector \
    --namespace kube-system \
    --wait \
    --timeout 5m

Verify that the Helm release and Kubernetes resources were removed.

Verify removal

helm --kubeconfig=$KUBE_CONFIG list -n kube-system -a \
    | grep -E '^node-problem-detector[[:space:]]' || true

kubectl --kubeconfig=$KUBE_CONFIG get daemonset -n kube-system \
    | grep '^node-problem-detector[[:space:]]' || true

Existing Kubernetes events generated by Node Problem Detector can remain until the event retention period expires, even after the Helm release is removed. After removal, verify that no new related events or node conditions continue to appear.

Step 6. Recover from a failure

If a worker node enters the NotReady state or abnormal node conditions or excessive events occur, first inspect the status.

Check the failure status

kubectl --kubeconfig=$KUBE_CONFIG get daemonset -n kube-system \
    | grep node-problem-detector

kubectl --kubeconfig=$KUBE_CONFIG get events -A \
    --field-selector involvedObject.kind=Node

If you installed Node Problem Detector by following this guide on Kubernetes 1.32 or 1.33, inspect the logs and Helm release history.

Check logs and Helm release history

kubectl --kubeconfig=$KUBE_CONFIG logs -n kube-system \
    -l app.kubernetes.io/instance=node-problem-detector \
    --tail=200

helm --kubeconfig=$KUBE_CONFIG history node-problem-detector -n kube-system

If a previously working revision exists, roll back to that revision.

Roll back Node Problem Detector

helm --kubeconfig=$KUBE_CONFIG rollback node-problem-detector {revision} \
    --namespace kube-system \
    --wait \
    --timeout 5m

If the initial installation failed or the problem persists after rollback, remove the Helm release as described in Step 5. Uninstall Node Problem Detector.

The Kubernetes Engine-managed Node Problem Detector on Kubernetes 1.34 or later is not subject to Helm history inspection or rollback. If the problem persists after you inspect the DaemonSet status, pod logs, node conditions, and events, contact the Helpdesk.

After recovery, verify the following items again:

• Worker nodes remain in the Ready state.
• Node Problem Detector pods are running normally.
• Abnormal node conditions or events no longer increase.