Skip to main content

Troubleshooting Kubernetes Engine

This document summarizes major issues and corresponding solutions related to the Kubernetes Engine service.

Cluster Failed: Insufficient resources​

The cluster status changes to Failed, and the following error message appears:

Error message
Error from server (Forbidden): the targeted Availability Zone "region-name", does not currently have sufficient capacity to support the cluster.

This error occurs when one or more of the specified availability zones lack sufficient capacity to support the cluster during creation.

▶️ Solution: Some availability zones cannot host the cluster. Verify the availability zones of the associated subnets and recreate the cluster.

Cluster Failed: Internal processing error​

The cluster status changes to Failed, and the following error message appears:

Error message
Error from server (Forbidden): Cluster processing has failed because of an internal error, exception or failure

This error occurs due to an internal system error or exception during cluster creation.

▶️ Solution: Contact the Help Desk with cluster information.

kubectl: Namespace is forbidden​

Even after completing the kubectl configuration, running a kubectl command shows the following error:

Error message
Error from server (Forbidden): namespaces is forbidden: User "<user_email@example.com>" cannot list resource "namespaces" in API group "" at the cluster scope

This error occurs when the project specified during access key generation does not match the project of the target cluster.

▶️ Solution: When generating the access key, ensure the project associated with the target cluster is selected. Refer to Access key issuance for detailed steps.

Kubelet: Nameserver limits exceeded​

After creating a node pool, the following error may appear in kubelet logs:

Jul 15 16:42:22 {Node-name} kubelet[6020]: E0715 16:42:22.417041 6020 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: ~ ~ ~"

KakaoCloud Kubernetes Engine nodes are Linux (Ubuntu) based and support up to 3 DNS nameserver records.
Kubernetes requires 1 DNS nameserver.
If 3 nameservers are already defined, the Nameserver limit exceeded error will occur.

▶️ Solution: Check if additional nameservers were added via [advanced settings] > [user script] during node pool creation and reduce the number of nameserver records in resolv.conf.

Reference

For details on DNS nameserver limitations, refer to the official Kubernetes guide.

kubectl: Please enter Username​

Even after configuring kubectl, the following prompt appears:

Error message
Please enter Username:

This occurs when the user name in contexts > context > cluster > user does not match the name in users > name in the kubeconfig file.

▶️ Solution: Ensure the user name in contexts > context > cluster > user matches the value in users > name.

kubectl: Unable to connect to the server​

kubectl command execution results in the following error:

Error message
Unable to connect to the server: getting credentials: exec: executable kic-iam-auth failed with exit code 1

This indicates that the access key information in users > user > exec > env in the kubeconfig file is incorrect or mismatched.

▶️ Solution: Refer to User authentication settings to reconfigure the authentication.

CSI Provisioner deployment, pod​

After installing the cinder-csi, the pod shows a CrashLoopBackOff status and the following error:

Error message
E0123 07:54:11.985138       1 openstack.go:102] Failed to open OpenStack configuration file: open /etc/kubernetes/cloud.conf: no such file or directory
E0123 07:54:11.985145       1 openstack.go:144] GetConfigFromFiles [/etc/kubernetes/cloud.conf] failed with error: open /etc/kubernetes/cloud.conf: no such file or directory

This error occurs when required parameters are not provided during Helm chart installation.

▶️ Solution: Reinstall cinder-csi with required parameters:

helm install cinder-csi cpo/openstack-cinder-csi \
--version 2.3.0 \
--set secret.enabled=true \
--set secret.name=cloud-config \
--namespace kube-system

helm: kubernetes cluster unreachable​

When executing Helm CLI commands, the following error appears:

Error message
Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

This error occurs when Helm cannot find the kubeconfig file containing cluster information.

▶️ Solution: Register the kubeconfig file using the $KUBECONFIG environment variable or the --kubeconfig option:

  1. Register the environment variable:

    export KUBECONFIG="{kubeconfig path}"

  2. Use the --kubeconfig option:

    helm --kubeconfig={Download_Path/kubeconfig.yaml} list

Istio: failed to call Webhook​

This error occurs when Istio is deployed but network communication between master and worker nodes fails:

Error message
Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

Master nodes are not part of the container network, so communication with https://istiod for webhook validation fails.

▶️ Solution: Add hostNetwork: true to the istiod deployment to allow direct node communication. Also, set dnsPolicy: ClusterFirstWithHostNet explicitly when using hostNetwork: true.

kubectl edit deployment -n istio-system istiod

***
spec:
    hostNetwork: true # added
    dnsPolicy: ClusterFirstWithHostNet
containers:
***

dnsPolicy options​

  1. dnsPolicy: Default
    Uses the DNS configuration from the node's /etc/resolv.conf.

  2. dnsPolicy: ClusterFirst
    Routes queries for non-cluster domains to upstream servers.

  3. dnsPolicy: ClusterFirstWithHostNet
    Required when using hostNetwork: true to ensure cluster DNS is used.
    If dnsPolicy: ClusterFirst is used instead, it defaults to Default behavior.

  4. dnsPolicy: None
    Ignores cluster DNS settings; used for custom DNS configurations.

For more information, refer to the Kubernetes documentation.