Skip to main content

Troubleshooting Kubernetes Engine

This document summarizes major issues and corresponding solutions related to the Kubernetes Engine service.

Cluster Failed: Insufficient resources​

The cluster status changes to Failed, and the following error message appears:

Error message
Error from server (Forbidden): the targeted Availability Zone "region-name", does not currently have sufficient capacity to support the cluster.

This error occurs when one or more of the specified availability zones lack sufficient capacity to support the cluster during creation.

▶️ Solution: Some availability zones cannot host the cluster. Verify the availability zones of the associated subnets and recreate the cluster.

Cluster Failed: Internal processing error​

The cluster status changes to Failed, and the following error message appears:

Error message
Error from server (Forbidden): Cluster processing has failed because of an internal error, exception or failure

This error occurs due to an internal system error or exception during cluster creation.

▶️ Solution: Contact the Helpdesk with cluster information.

kubectl: Namespace is forbidden​

Even after completing the kubectl configuration, running a kubectl command shows the following error:

Error message
Error from server (Forbidden): namespaces is forbidden: User "<user_email@example.com>" cannot list resource "namespaces" in API group "" at the cluster scope

This error occurs when the project specified during access key generation does not match the project of the target cluster.

▶️ Solution: When generating the access key, ensure the project associated with the target cluster is selected. Refer to Access key issuance for detailed steps.

Kubelet: Nameserver limits exceeded​

After creating a node pool, the following error may appear in kubelet logs:

Jul 15 16:42:22 {Node-name} kubelet[6020]: E0715 16:42:22.417041 6020 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: ~ ~ ~"

KakaoCloud Kubernetes Engine nodes are Linux (Ubuntu) based and support up to 3 DNS nameserver records.
Kubernetes requires 1 DNS nameserver.
If 3 nameservers are already defined, the Nameserver limit exceeded error will occur.

▶️ Solution: Check if additional nameservers were added via [advanced settings] > [user script] during node pool creation and reduce the number of nameserver records in resolv.conf.

Reference

For details on DNS nameserver limitations, refer to the official Kubernetes guide.

kubectl: Please enter Username​

Even after configuring kubectl, the following prompt appears:

Error message
Please enter Username:

This occurs when the user name in contexts > context > cluster > user does not match the name in users > name in the kubeconfig file.

▶️ Solution: Ensure the user name in contexts > context > cluster > user matches the value in users > name.

kubectl: Unable to connect to the server​

kubectl command execution results in the following error:

Error message
Unable to connect to the server: getting credentials: exec: executable kic-iam-auth failed with exit code 1

This indicates that the access key information in users > user > exec > env in the kubeconfig file is incorrect or mismatched.

▶️ Solution: Refer to User authentication settings to reconfigure the authentication.

CSI Provisioner deployment, pod​

After installing the cinder-csi, the pod shows a CrashLoopBackOff status and the following error:

Error message
E0123 07:54:11.985138       1 openstack.go:102] Failed to open OpenStack configuration file: open /etc/kubernetes/cloud.conf: no such file or directory
E0123 07:54:11.985145       1 openstack.go:144] GetConfigFromFiles [/etc/kubernetes/cloud.conf] failed with error: open /etc/kubernetes/cloud.conf: no such file or directory

This error occurs when required parameters are not provided during Helm chart installation.

▶️ Solution: Reinstall cinder-csi with required parameters:

helm install cinder-csi cpo/openstack-cinder-csi \
--version 2.3.0 \
--set secret.enabled=true \
--set secret.name=cloud-config \
--namespace kube-system

helm: kubernetes cluster unreachable​

When executing Helm CLI commands, the following error appears:

Error message
Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

This error occurs when Helm cannot find the kubeconfig file containing cluster information.

▶️ Solution: Register the kubeconfig file using the $KUBECONFIG environment variable or the --kubeconfig option:

  1. Register the environment variable:

    export KUBECONFIG="{kubeconfig path}"

  2. Use the --kubeconfig option:

    helm --kubeconfig={Download_Path/kubeconfig.yaml} list

Istio: failed to call webhook​

This error occurs when the API server cannot communicate with the Admission Webhook endpoint of istiod while Istio is configured on a Kubernetes Engine cluster:

Error message
Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

In Kubernetes 1.32 and 1.33 clusters, control plane nodes are not part of the container network, so they may fail to communicate with the Admission Webhook server.

This solution applies to Kubernetes 1.32 and 1.33 clusters. To allow the API server to access the istiod Pod, add hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet to the Pod spec of the istiod Deployment.

Edit the istiod Deployment
kubectl edit deployment -n istio-system istiod
Configuration example
spec:
  template:
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
info

In Kubernetes 1.34 or later clusters, the API server communicates with the Admission Webhook server through Konnectivity, so you do not need to set hostNetwork: true on the Admission Webhook server Pod.

dnsPolicy options​

The following table describes each dnsPolicy value.

ValueDescription
DefaultUses the DNS configuration from the node's /etc/resolv.conf.
ClusterFirstRoutes DNS queries that do not match the cluster domain suffix, such as .cluster.local, to upstream name servers.
ClusterFirstWithHostNetAllows Pods that use hostNetwork: true to keep using cluster DNS settings. If a Pod with hostNetwork: true uses dnsPolicy: ClusterFirst, it behaves like the Default policy. Therefore, set ClusterFirstWithHostNet explicitly.
NonePrevents Pods from using Kubernetes DNS settings. Use this value for Pods that require custom DNS servers.

For more information, see Kubernetes DNS policy.

External Secrets Operator (ESO) integration​

Use the following checks to troubleshoot ESO integration. For configuration and supported scope, see Integrate External Secrets Operator.

ImagePullBackOff during Helm installation​

Check the following:

  • Whether worker nodes can connect to ghcr.io:443
  • Whether the image address and tag match the installation guide
  • The cause of the image download failure reported in pod events
Check pod status
kubectl get pods -n external-secrets
kubectl describe pod <POD_NAME> -n external-secrets

Webhook timeout when applying a SecretStore or ExternalSecret​

The following errors indicate an admission webhook communication problem before any IAM or Secrets Manager calls.

failed calling webhook
context deadline exceeded

Check the following:

  • On Kubernetes 1.33, whether webhook.hostNetwork=true and webhook.port=9443 are applied
  • On Kubernetes 1.34 or later, whether the Helm chart's default webhook network settings are used
  • The status of webhook pods, services, and EndpointSlice resources
  • Whether the Kubernetes API server can communicate with the webhook
Check webhook resources
kubectl get pods,svc,endpointslice -n external-secrets
kubectl get validatingwebhookconfiguration

SecretStore reports READY=False​

Diagnose SecretStore
kubectl describe secretstore kakaocloud-secret-store -n eso-demo
kubectl logs deployment/external-secrets -n external-secrets

Check the following:

  • Whether the kakaocloud-credentials Kubernetes Secret containing IAM credentials exists
  • Whether its data keys are accessKeyID and secretAccessKey
  • Whether the IAM access key has been deleted
  • The IAM and Secrets Manager endpoints
  • Whether the IAM user has permission to read Secrets Manager secrets

ExternalSecret reports READY=False​

Diagnose ExternalSecret
kubectl describe externalsecret kakaocloud-external-secret -n eso-demo
kubectl get events -n eso-demo --sort-by=.lastTimestamp
SymptomWhat to check
Authentication failureIAM access key ID and secret access key
403 ForbiddenIAM and Secrets Manager read permissions
Secrets Manager secret not foundWhether remoteRef.key contains the secret ID rather than its name
Secrets Manager secret version not foundremoteRef.version
JSON field errorWhether the Secrets Manager value is a valid JSON object and, if remoteRef.property is specified, whether that field exists
Updated values are not reflected in the Kubernetes SecretrefreshPolicy and refreshInterval