Install and manage Node Local DNS Cache

Node Local DNS Cache is a DNS cache that runs as a DaemonSet on each worker node. It handles pod DNS requests on the same node and forwards only cache misses to cluster DNS, reducing DNS lookup latency and CoreDNS load. For more information, see the official Kubernetes NodeLocal DNSCache documentation.

You can install and manage Node Local DNS Cache independently of Node Problem Detector. If you need node problem detection, see Install and manage Node Problem Detector.

info

• You can manually install Node Local DNS Cache on clusters running a supported Kubernetes version of 1.32 or later. It is not installed automatically.
• A Node Local DNS Cache installed by following this guide is a user-managed Helm release. You are responsible for installation, configuration, upgrades, removal, and failure recovery.
• On a Cilium 1.14.4 cluster, installing only the Helm chart might not enable the cache for regular pods. If you cannot change the pod specifications and DNS egress policies of workloads that will use the cache, installing it is not recommended because it is unlikely to reduce CoreDNS load.

Consider introducing Node Local DNS Cache if any of the following DNS performance problems persist:

• CoreDNS pods experience sustained CPU throttling or high CPU usage.
• DNS lookup latency remains high after adjusting the number of CoreDNS replicas.
• Applications repeatedly experience DNS request timeouts.

You do not need to install it if no DNS performance problem exists. Before production deployment, compare CoreDNS CPU usage and throttling, DNS latency and request timeouts, and Node Local DNS Cache request metrics before and after deployment in a development or staging cluster.

By default, Node Local DNS Cache requests 25m CPU and 128Mi memory per target worker node and sets a 128Mi memory limit. It uses the system-node-critical priority and tolerations for all NoSchedule and NoExecute node taints, so it can preempt lower-priority pods on resource-constrained nodes.

Step 1. Prerequisites

Prepare the cluster and tools

1. Create a Kubernetes Engine cluster in which to install Node Local DNS Cache.
2. Create a node pool in the cluster and verify that the worker nodes are Ready.
3. Configure kubectl so that you can run commands against the cluster.
4. Install Helm 3 by following the official Helm documentation.
5. Install jq to inspect the ports of pods that use the host network.

caution

Worker nodes in a private subnet cannot directly access the internet. Configure an outbound communication path, such as a NAT instance, so that the nodes can pull the Node Local DNS Cache container image and diagnostic images. Verify that worker nodes can access the image registries. For more information, see Use a NAT instance.

Set environment variables for the target cluster's kubeconfig and diagnostic images. Prepare the diagnostic images in a user-managed registry that the worker nodes can access. The following example assumes that busybox:1.36.1 is mirrored to that registry.

Set environment variables

export KUBE_CONFIG=/path/to/cluster-kubeconfig.yaml
export CUSTOMER_REGISTRY=registry.example.com/project
export NODE_DEBUG_IMAGE=${CUSTOMER_REGISTRY}/busybox:1.36.1
export DNS_TEST_IMAGE=${CUSTOMER_REGISTRY}/busybox:1.36.1

Replace KUBE_CONFIG and CUSTOMER_REGISTRY with values for your environment. NODE_DEBUG_IMAGE must include the sh and chroot commands, and DNS_TEST_IMAGE must include the nslookup and sleep commands.

Check the Kubernetes version of the cluster.

Check the Kubernetes version

kubectl --kubeconfig=$KUBE_CONFIG version

Set environment variables for the Kubernetes minor version. The following example is for a Kubernetes 1.33 cluster.

Set the Helm chart version

export K8S_MINOR=1.33
export CHART_VERSION=${K8S_MINOR}.0

Kubernetes version	Helm chart version
1.32	1.32.0
1.33	1.33.0
1.34	1.34.0
1.35	1.35.0

The Node Local DNS Cache Helm chart is provided through the KakaoCloud public OCI registry. You can install it without registering a separate Helm repository.

Check the CNI and existing deployments

Identify the CNI used by the cluster. Cilium is deployed in kube-system and Calico in calico-system, so search all namespaces.

Check the CNI

kubectl --kubeconfig=$KUBE_CONFIG get daemonset -A \
    | grep -E 'cilium|calico' || true

In the Cilium 1.14.4 environment validated by Kubernetes Engine, DNS requests from regular pods do not pass through the NOTRACK path configured by Node Local DNS Cache, even when kubeProxyReplacement=false and kube-proxy is operating normally. To use Node Local DNS Cache with Cilium, change the workload configuration as described in Configure DNS for Cilium workloads.

Check whether Node Local DNS Cache is already installed.

Check for an existing deployment

helm --kubeconfig=$KUBE_CONFIG list -n kube-system -a

kubectl --kubeconfig=$KUBE_CONFIG get daemonset -n kube-system \
    | grep -E 'node-local-dns' || true

If a deployment exists, do not install a duplicate. First identify the Helm release name, applied values, and managing entity. Multiple Node Local DNS Cache deployments modify the same node DNS IP, ports, and iptables rules. When one deployment terminates, it can remove network settings still used by another deployment.

The helm list output can include system releases managed by Kubernetes Engine, such as ke-cilium and ke-tigera-operator. Do not modify or delete system releases.

Check for registry access and network conflicts

Verify that worker nodes can access the following registry.

ke-container-registry.kr-central-2.kcr.dev

Check whether another node interface or route uses the IP address reserved for Node Local DNS Cache. Replace {node-name} with an actual worker node name.

Check for a Node Local DNS IP conflict

kubectl --kubeconfig=$KUBE_CONFIG debug node/{node-name} -it \
    --image=$NODE_DEBUG_IMAGE -- \
    chroot /host sh -c 'ip addr show | grep 169.254.20.25 || true'

kubectl --kubeconfig=$KUBE_CONFIG debug node/{node-name} -it \
    --image=$NODE_DEBUG_IMAGE -- \
    chroot /host sh -c 'ip route get 169.254.20.25 || true'

Node Local DNS Cache uses the NET_ADMIN capability and the host network to modify node network settings. Do not deploy it if an existing network interface or host agent uses 169.254.20.25.

Check whether ports used by Node Local DNS Cache conflict on worker nodes.

DNS	Health check	Metrics
TCP/UDP 53	TCP 8080	TCP 9253, TCP 9353

The following command lists ports configured for pods that use the host network. Because containerPort is optional, this result alone cannot prove that no port conflict exists.

Check ports of host-network pods

kubectl --kubeconfig=$KUBE_CONFIG get pod -A -o json \
    | jq -r '.items[] | select(.spec.hostNetwork == true) | [.metadata.namespace, .metadata.name, ([.spec.containers[].ports[]?.containerPort] | join(","))] | @tsv'

If you can access a worker node, also inspect whether the default ports are in use.

Check ports on a worker node

ss -lntup | grep -E ':(53|8080|9253|9353)\b' || true

If health check port 8080 conflicts, select an unused port and enter it as config.healthPort in the values file. This Helm chart does not allow you to change DNS port 53 or metrics ports 9253 and 9353. If any of those ports conflict, resolve the cause before installation.

Step 2. Install Node Local DNS Cache

Create a values file

Get the cluster IP of the cluster's CoreDNS service, kube-dns.

Get the cluster DNS service IP

kubectl --kubeconfig=$KUBE_CONFIG \
    -n kube-system get service kube-dns \
    -o jsonpath='{.spec.clusterIP}'
echo

Create node-local-dns-values.yaml.

node-local-dns-values.yaml

config:
  bindIp: true
  dnsServer: '<CLUSTER_DNS_IP>'
  localDns: 169.254.20.25
  healthPort: 8080 # If you selected another port after checking for conflicts, enter it here
  enableLogging: false

Replace <CLUSTER_DNS_IP> with the service IP returned by the preceding command. Shell environment variables are not automatically expanded in a YAML file, so verify that no placeholder remains. If you enter another IP, the cache might not work or traffic for the service at that IP might be directed to Node Local DNS Cache.

The default Helm chart value for config.enableLogging is true. Query logging records successful DNS queries as well, increasing log usage and potentially retaining queried domain names in logs. This guide recommends false as the production default. Do not install the chart with its default settings; always apply the preceding node-local-dns-values.yaml file. Enable query logging temporarily only when needed for troubleshooting, as described in Step 6. Recover from a failure.

Setting config.bindIp to false does not disable node network configuration. After installation, always validate metrics as described in Step 3. Verify the deployment.

Check and install the Helm chart

Review the Helm chart defaults and the resources that the chart will create.

Check the Helm chart defaults

helm show values \
    oci://ke-container-registry.kr-central-2.kcr.dev/ke-helm-public/node-local-dns \
    --version $CHART_VERSION

Validate Node Local DNS Cache before installation

helm template node-local-dns \
    oci://ke-container-registry.kr-central-2.kcr.dev/ke-helm-public/node-local-dns \
    --version $CHART_VERSION \
    --namespace kube-system \
    --values node-local-dns-values.yaml \
    | kubectl --kubeconfig=$KUBE_CONFIG apply --dry-run=server -f -

The Helm chart defaults, config.setupInterface: true and config.setupIptables: true, configure the local DNS IP and cluster DNS service IP on the node and add the related network interface and iptables rules. In a Calico cluster that uses the kube-proxy iptables path, applying network rules after the Node Local DNS Cache pod becomes ready can take from several tens of seconds to approximately one minute, depending on the environment. After the rules are applied, existing ClusterFirst pods on that node use Node Local DNS Cache and do not need to be restarted.

Because Node Local DNS Cache is deployed as a DaemonSet, DNS paths on all target worker nodes can change at approximately the same time. In a production cluster, install it during a period of low service impact.

During installation, configuration changes, rollback, and removal, Node Local DNS Cache pods can be created, replaced, or deleted. DNS lookups on the affected node can temporarily fail or be delayed. Perform these operations during a period of low service impact and verify the deployment immediately afterward.

Install Node Local DNS Cache

helm --kubeconfig=$KUBE_CONFIG upgrade --install node-local-dns \
    oci://ke-container-registry.kr-central-2.kcr.dev/ke-helm-public/node-local-dns \
    --version $CHART_VERSION \
    --namespace kube-system \
    --values node-local-dns-values.yaml \
    --wait \
    --timeout 5m

Configure DNS for Cilium workloads

In a Cilium cluster, apply the DNS egress policy before changing pod specifications for workloads that will use Node Local DNS Cache. Skip this procedure on a Calico cluster.

Before changing pod specifications, workloads with restricted DNS egress must be allowed to communicate with TCP and UDP port 53 at 169.254.20.25/32. Add the following rule to egress in the existing CiliumNetworkPolicy and verify that the policy is applied.

Example CiliumNetworkPolicy egress configuration

egress:
  - toCIDR:
      - 169.254.20.25/32
    toPorts:
      - ports:
          - port: '53'
            protocol: TCP
          - port: '53'
            protocol: UDP

When using a toFQDNs policy

If an existing CiliumNetworkPolicy restricts egress with toFQDNs, the Cilium DNS proxy must inspect DNS responses to learn the mapping between domain names and IP addresses. If you add only a toCIDR rule and then change the workload DNS server to 169.254.20.25, DNS requests might bypass the Cilium DNS proxy and connections to destinations allowed by toFQDNs can fail.

For a workload that uses toFQDNs, add rules.dns to toPorts in the toCIDR rule for Node Local DNS Cache, as shown in the following example. Keep the existing toFQDNs rule as a separate egress item.

Example DNS proxy configuration with toFQDNs

egress:
  - toCIDR:
      - 169.254.20.25/32
    toPorts:
      - ports:
          - port: '53'
            protocol: TCP
          - port: '53'
            protocol: UDP
        rules:
          dns:
            - matchPattern: '*'

After verifying that the policy is applied, add the following configuration to spec.template.spec of a Deployment, StatefulSet, or other workload.

Example workload DNS configuration

dnsPolicy: None
dnsConfig:
  nameservers:
    - '169.254.20.25'
    - '<CLUSTER_DNS_IP>'
  searches:
    - '<NAMESPACE>.svc.cluster.local'
    - 'svc.cluster.local'
    - 'cluster.local'
  options:
    - name: ndots
      value: '5'

Replace <CLUSTER_DNS_IP> with the actual CoreDNS service IP and <NAMESPACE> with the workload namespace. The first DNS server is Node Local DNS Cache, and the second is a fallback path to CoreDNS. If the first server does not respond, the DNS resolver waits for a response before querying the second server, which can delay the lookup. Do not treat the second server as an immediate failover path.

Changing the pod template triggers a rolling restart of the target workload's pods. Review the workload update strategy and PDB (PodDisruptionBudget), and apply the change during a period of low traffic impact. Regular pods without this configuration continue to use the existing CoreDNS path and do not benefit from the cache.

Step 3. Verify the deployment

Check the Helm release and DaemonSet

Check the Helm release status.

Check the Helm release status

helm --kubeconfig=$KUBE_CONFIG status node-local-dns -n kube-system

Verify that the DaemonSet and pods are running normally on every target worker node.

Check the DaemonSet and pods

kubectl --kubeconfig=$KUBE_CONFIG get daemonset \
    node-local-dns -n kube-system

kubectl --kubeconfig=$KUBE_CONFIG get pod -n kube-system -o wide \
    -l k8s-app=node-local-dns

kubectl --kubeconfig=$KUBE_CONFIG rollout status \
    daemonset/node-local-dns -n kube-system

Verify DNS resolution

Test DNS resolution for an internal cluster service and an external domain.

Test the default DNS path

kubectl --kubeconfig=$KUBE_CONFIG run dns-test \
    --rm -it \
    --restart=Never \
    --image=$DNS_TEST_IMAGE \
    -- nslookup kubernetes.default.svc.cluster.local

kubectl --kubeconfig=$KUBE_CONFIG run dns-test-external \
    --rm -it \
    --restart=Never \
    --image=$DNS_TEST_IMAGE \
    -- nslookup kakao.com

Query Node Local DNS Cache directly by specifying its address.

Query Node Local DNS Cache directly

kubectl --kubeconfig=$KUBE_CONFIG run dns-test-node-local \
    --rm -it \
    --restart=Never \
    --image=$DNS_TEST_IMAGE \
    -- nslookup kubernetes.default.svc.cluster.local 169.254.20.25

A successful direct query confirms only that Node Local DNS Cache responds at 169.254.20.25. Verify separately in the next section whether DNS requests from regular pods actually pass through the cache.

Verify the DNS path of a regular pod

On a Calico cluster, create a regular ClusterFirst pod.

Create a pod to validate the Calico DNS path

kubectl --kubeconfig=$KUBE_CONFIG run dns-path-test \
    --restart=Never \
    --image=$DNS_TEST_IMAGE \
    --command -- sleep 3600

On a Cilium cluster, replace <CLUSTER_DNS_IP> in the following file with the actual CoreDNS service IP. Replace <DNS_TEST_IMAGE> with the image shown by echo $DNS_TEST_IMAGE. Then create a validation pod with the same configuration described in Configure DNS for Cilium workloads.

dns-path-test.yaml

apiVersion: v1
kind: Pod
metadata:
  name: dns-path-test
spec:
  restartPolicy: Never
  dnsPolicy: None
  dnsConfig:
    nameservers:
      - '169.254.20.25'
      - '<CLUSTER_DNS_IP>'
    searches:
      - 'default.svc.cluster.local'
      - 'svc.cluster.local'
      - 'cluster.local'
    options:
      - name: ndots
        value: '5'
  containers:
    - name: dns-path-test
      image: '<DNS_TEST_IMAGE>'
      command: ['sleep', '3600']

Create a pod to validate the Cilium DNS path

kubectl --kubeconfig=$KUBE_CONFIG apply -f dns-path-test.yaml

After the validation pod is ready, identify the Node Local DNS Cache pod running on the same node.

Check the validation pod and Node Local DNS Cache pod

kubectl --kubeconfig=$KUBE_CONFIG wait pod/dns-path-test \
    --for=condition=Ready --timeout=2m

export DNS_TEST_NODE=$(kubectl --kubeconfig=$KUBE_CONFIG \
    get pod dns-path-test -o jsonpath='{.spec.nodeName}')

export NODE_LOCAL_DNS_POD=$(kubectl --kubeconfig=$KUBE_CONFIG \
    get pod -n kube-system \
    -l k8s-app=node-local-dns \
    --field-selector spec.nodeName=$DNS_TEST_NODE \
    -o jsonpath='{.items[0].metadata.name}')

echo $DNS_TEST_NODE
echo $NODE_LOCAL_DNS_POD

Forward the selected Node Local DNS Cache pod's metrics port to the local host, and compare the counter before and after generating DNS requests from the validation pod.

Check Node Local DNS Cache request metrics

kubectl --kubeconfig=$KUBE_CONFIG port-forward -n kube-system \
    pod/$NODE_LOCAL_DNS_POD 9253:9253 \
    > /tmp/node-local-dns-port-forward.log 2>&1 &
export PORT_FORWARD_PID=$!

curl --retry 10 --retry-delay 1 --retry-connrefused -fsS \
    http://127.0.0.1:9253/metrics >/dev/null

curl -s http://127.0.0.1:9253/metrics \
    | grep '^coredns_dns_requests_total' || true

for i in 1 2 3 4 5; do
    kubectl --kubeconfig=$KUBE_CONFIG exec dns-path-test -- \
        nslookup kubernetes.default.svc.cluster.local >/dev/null
done

curl -s http://127.0.0.1:9253/metrics \
    | grep '^coredns_dns_requests_total' || true

kill $PORT_FORWARD_PID
kubectl --kubeconfig=$KUBE_CONFIG delete pod dns-path-test

If the second coredns_dns_requests_total value is greater than the first, DNS requests from the validation pod passed through Node Local DNS Cache. Validate with a regular ClusterFirst pod on Calico and with a pod configured with dnsConfig on Cilium.

Do not apply the configuration to production if no metrics are returned or the counter does not increase. On Cilium, first remove dnsPolicy and dnsConfig from workloads and verify the default CoreDNS path, and then uninstall Node Local DNS Cache. On Calico, proceed directly to the removal procedure.

Verification criteria

The deployment is complete only when all of the following conditions are met:

• The Helm release is in the deployed state.
• The DaemonSet Ready count matches the number of target worker nodes.
• DNS resolution succeeds for internal cluster services and external domains.
• A direct query using 169.254.20.25 as the DNS server succeeds.
• After generating DNS requests from a regular pod on Calico or a pod configured with dnsConfig on Cilium, coredns_dns_requests_total increases on the same node.
• config.dnsServer matches the actual cluster DNS service IP.
• CoreDNS and Node Local DNS Cache logs contain no persistent DNS errors.
• Worker nodes remain in the Ready state.

Step 4. Monitor operations

Monitor the following items after deployment:

• DNS lookup latency and failure rate
• SERVFAIL, timeout, and connection refused errors in CoreDNS and Node Local DNS Cache logs
• Restart counts of Node Local DNS Cache pods
• Changes to the Ready state of worker nodes

To verify that Node Local DNS Cache is used in the actual DNS path, generate DNS requests from a regular pod on Calico or a pod configured with dnsConfig on Cilium, and confirm that coredns_dns_requests_total increases.

Use the following commands to inspect the applied values and logs.

Check the applied values and logs

helm --kubeconfig=$KUBE_CONFIG get values node-local-dns \
    -n kube-system -a

kubectl --kubeconfig=$KUBE_CONFIG logs -n kube-system \
    -l k8s-app=node-local-dns --tail=100

Step 5. Uninstall Node Local DNS Cache

Identify the Helm release to remove. Because helm list displays releases only in the current namespace, specify the kube-system namespace.

Identify the Helm release to remove

helm --kubeconfig=$KUBE_CONFIG list -n kube-system -a \
    | grep -E '^node-local-dns[[:space:]]'

If necessary, back up the current values before removal.

Back up the applied values

helm --kubeconfig=$KUBE_CONFIG get values node-local-dns \
    -n kube-system -a -o yaml > node-local-dns-values-backup.yaml

If you applied dnsPolicy and dnsConfig to Cilium workloads, remove those settings before uninstalling Node Local DNS Cache to restore the default CoreDNS path. Wait for the rolling restart to complete and verify DNS resolution through the default CoreDNS path before removal. If a workload or kubelet directly uses 169.254.20.25 as its DNS server, DNS failures or lookup delays can occur after removal.

Uninstall Node Local DNS Cache

helm --kubeconfig=$KUBE_CONFIG uninstall node-local-dns \
    --namespace kube-system \
    --wait \
    --timeout 5m

Verify that the Helm release and Kubernetes resources were removed.

Verify removal

helm --kubeconfig=$KUBE_CONFIG list -n kube-system -a \
    | grep -E '^node-local-dns[[:space:]]' || true

kubectl --kubeconfig=$KUBE_CONFIG get daemonset -n kube-system \
    | grep '^node-local-dns[[:space:]]' || true

Verify that internal services and external domains resolve correctly through the default cluster DNS path.

Verify DNS resolution after removal

kubectl --kubeconfig=$KUBE_CONFIG run dns-test-after-uninstall \
    --rm -it \
    --restart=Never \
    --image=$DNS_TEST_IMAGE \
    -- nslookup kubernetes.default.svc.cluster.local

kubectl --kubeconfig=$KUBE_CONFIG run dns-test-external-after-uninstall \
    --rm -it \
    --restart=Never \
    --image=$DNS_TEST_IMAGE \
    -- nslookup kakao.com

Step 6. Recover from a failure

If DNS resolution fails, latency increases, or a worker node enters the NotReady state, first inspect the status and logs.

Check the failure status

kubectl --kubeconfig=$KUBE_CONFIG get daemonset \
    node-local-dns -n kube-system

kubectl --kubeconfig=$KUBE_CONFIG logs -n kube-system \
    -l k8s-app=node-local-dns --tail=200

kubectl --kubeconfig=$KUBE_CONFIG logs -n kube-system \
    -l k8s-app=kube-dns --tail=200

Enable query logging only when you need to inspect detailed DNS requests. Because it also records successful queries, it can increase log usage. Enable it only for the required period and disable it when the investigation is complete.

Enable query logging

helm --kubeconfig=$KUBE_CONFIG upgrade node-local-dns \
    oci://ke-container-registry.kr-central-2.kcr.dev/ke-helm-public/node-local-dns \
    --version $CHART_VERSION \
    --namespace kube-system \
    --reuse-values \
    --set config.enableLogging=true \
    --wait \
    --timeout 5m

After the investigation, change the value to false in the same command to disable query logging again.

Check the Helm release history.

Check the Helm release history

helm --kubeconfig=$KUBE_CONFIG history node-local-dns -n kube-system

If a previously working revision exists, roll back to that revision. During rollback, DNS lookups on the affected node can temporarily fail or be delayed, so perform the rollback during a period of low service impact.

Roll back Node Local DNS Cache

helm --kubeconfig=$KUBE_CONFIG rollback node-local-dns {revision} \
    --namespace kube-system \
    --wait \
    --timeout 5m

If the initial installation failed or the problem persists after rollback, remove the Helm release as described in Step 5. Uninstall Node Local DNS Cache. After rollback or removal, verify that internal cluster services and external domains resolve correctly through the default cluster DNS path, and check the verification criteria again.