Troubleshooting Kubernetes Engine
This document summarizes major issues and corresponding solutions related to the Kubernetes Engine service.
- Cluster Failed: Insufficient resources
- Cluster Failed: Internal processing error
- kubectl: Namespace is forbidden
- Kubelet: Nameserver limits exceeded
- kubectl: Please enter Username
- kubectl: Unable to connect to the server
- CSI Provisioner deployment, pod
- helm: kubernetes cluster unreachable
- Istio: failed to call webhook
Cluster Failed: Insufficient resources
The cluster status changes to Failed, and the following error message appears:
Error from server (Forbidden): the targeted Availability Zone "region-name", does not currently have sufficient capacity to support the cluster.
This error occurs when one or more of the specified availability zones lack sufficient capacity to support the cluster during creation.
▶️ Solution: Some availability zones cannot host the cluster. Verify the availability zones of the associated subnets and recreate the cluster.
Cluster Failed: Internal processing error
The cluster status changes to Failed, and the following error message appears:
Error from server (Forbidden): Cluster processing has failed because of an internal error, exception or failure
This error occurs due to an internal system error or exception during cluster creation.
▶️ Solution: Contact the Helpdesk with cluster information.
kubectl: Namespace is forbidden
Even after completing the kubectl configuration, running a kubectl command shows the following error:
Error from server (Forbidden): namespaces is forbidden: User "<user_email@example.com>" cannot list resource "namespaces" in API group "" at the cluster scope
This error occurs when the project specified during access key generation does not match the project of the target cluster.
▶️ Solution: When generating the access key, ensure the project associated with the target cluster is selected. Refer to Access key issuance for detailed steps.
Kubelet: Nameserver limits exceeded
After creating a node pool, the following error may appear in kubelet logs:
Jul 15 16:42:22 {Node-name} kubelet[6020]: E0715 16:42:22.417041 6020 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: ~ ~ ~"
KakaoCloud Kubernetes Engine nodes are Linux (Ubuntu) based and support up to 3 DNS nameserver records.
Kubernetes requires 1 DNS nameserver.
If 3 nameservers are already defined, the Nameserver limit exceeded error will occur.
▶️ Solution: Check if additional nameservers were added via [advanced settings] > [user script] during node pool creation and reduce the number of nameserver records in resolv.conf.
For details on DNS nameserver limitations, refer to the official Kubernetes guide.
kubectl: Please enter Username
Even after configuring kubectl, the following prompt appears:
Please enter Username:
This occurs when the user name in contexts > context > cluster > user does not match the name in users > name in the kubeconfig file.
▶️ Solution: Ensure the user name in contexts > context > cluster > user matches the value in users > name.
kubectl: Unable to connect to the server
kubectl command execution results in the following error:
Unable to connect to the server: getting credentials: exec: executable kic-iam-auth failed with exit code 1
This indicates that the access key information in users > user > exec > env in the kubeconfig file is incorrect or mismatched.
▶️ Solution: Refer to User authentication settings to reconfigure the authentication.
CSI Provisioner deployment, pod
After installing the cinder-csi, the pod shows a CrashLoopBackOff status and the following error:
E0123 07:54:11.985138 1 openstack.go:102] Failed to open OpenStack configuration file: open /etc/kubernetes/cloud.conf: no such file or directory
E0123 07:54:11.985145 1 openstack.go:144] GetConfigFromFiles [/etc/kubernetes/cloud.conf] failed with error: open /etc/kubernetes/cloud.conf: no such file or directory
This error occurs when required parameters are not provided during Helm chart installation.
▶️ Solution: Reinstall cinder-csi with required parameters:
helm install cinder-csi cpo/openstack-cinder-csi \
--version 2.3.0 \
--set secret.enabled=true \
--set secret.name=cloud-config \
--namespace kube-system
helm: kubernetes cluster unreachable
When executing Helm CLI commands, the following error appears:
Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused
This error occurs when Helm cannot find the kubeconfig file containing cluster information.
▶️ Solution: Register the kubeconfig file using the $KUBECONFIG environment variable or the --kubeconfig option:
-
Register the environment variable:
export KUBECONFIG="{kubeconfig path}" -
Use the
--kubeconfigoption:helm --kubeconfig={Download_Path/kubeconfig.yaml} list
Istio: failed to call webhook
This error occurs when the API server cannot communicate with the Admission Webhook endpoint of istiod while Istio is configured on a Kubernetes Engine cluster:
Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded
In Kubernetes 1.32 and 1.33 clusters, control plane nodes are not part of the container network, so they may fail to communicate with the Admission Webhook server.
This solution applies to Kubernetes 1.32 and 1.33 clusters. To allow the API server to access the istiod Pod, add hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet to the Pod spec of the istiod Deployment.
kubectl edit deployment -n istio-system istiod
spec:
template:
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
In Kubernetes 1.34 or later clusters, the API server communicates with the Admission Webhook server through Konnectivity, so you do not need to set hostNetwork: true on the Admission Webhook server Pod.
dnsPolicy options
The following table describes each dnsPolicy value.
| Value | Description |
|---|---|
Default | Uses the DNS configuration from the node's /etc/resolv.conf. |
ClusterFirst | Routes DNS queries that do not match the cluster domain suffix, such as .cluster.local, to upstream name servers. |
ClusterFirstWithHostNet | Allows Pods that use hostNetwork: true to keep using cluster DNS settings. If a Pod with hostNetwork: true uses dnsPolicy: ClusterFirst, it behaves like the Default policy. Therefore, set ClusterFirstWithHostNet explicitly. |
None | Prevents Pods from using Kubernetes DNS settings. Use this value for Pods that require custom DNS servers. |
For more information, see Kubernetes DNS policy.