Skip to main content

Troubleshoot Kubernetes Engine

This document summarizes major issues and solutions related to the Kubernetes Engine service.

kubectl: Namespace is forbidden​

Even after completing kubectl control settings, you may encounter the following error when using kubectl commands.

Error Message
Error from server (Forbidden): namespaces is forbidden: User "poby.hyeon@kakaoenterprise.com" cannot list resource "namespaces" in API group "" at the cluster scope

This error occurs when the project selected during access key issuance does not match the project of the cluster you need to control.

▶️ When specifying the project during access key creation, ensure that it is the project containing the cluster to be controlled. For more details on access key creation, refer to Get access key.

Kubelet: Nameserver limits exceeded​

After creating a node pool in a Kubernetes Engine cluster, the following error appears in the kubelet log.

Nodes created through KakaoCloud's Kubernetes Engine can add up to three DNS nameserver records, and Kubernetes uses one DNS nameserver record.
If a node is already using three nameservers, a Nameserver limit exceeded error will occur.

Jul 15 16:42:22 {Node-name} kubelet[6020]: E0715 16:42:22.417041 6020 dns.go:153] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: ~ ~ ~"

▶️ Check if you added nameserver records to the resolv.conf file through [Advanced Settings] > [User script] during node pool creation and adjust the number of nameserver records.

info

For more information on DNS server record limitations, please refer to the Kubernetes official guide.

kubectl: Please enter Username​

Even after completing kubectl control settings, you may encounter the following error when using kubectl commands.

Error Message
Please enter Username:

This error occurs when the cluster name set in contexts > context > cluster > user does not match the name set in users > name in the kubeconfig file.

▶️ Ensure that the cluster name set in contexts > context > cluster > user matches the name set in users > name.

kubectl: Unable to connect to the server​

Even after completing kubectl control settings, you may encounter the following error when using kubectl commands.

Error Message
Unable to connect to the server: getting credentials: exec: executable kic-iam-auth failed with exit code 1

User authentication uses the access key information set in the kubeconfig file under users > user > exec > env. The error occurs when this access key information does not match.

▶️ Refer to User Authentication Settings to reset the user authentication.

CSI Provisioner deployment, pod​

After completing the cinder-csi installation, the pod status is displayed as CrashLoopBackOff, and the following error occurs.

Error Message
E0123 07:54:11.985138       1 openstack.go:102] Failed to open OpenStack configuration file: open /etc/kubernetes/cloud.conf: no such file or directory
E0123 07:54:11.985145       1 openstack.go:144] GetConfigFromFiles [/etc/kubernetes/cloud.conf] failed with error: open /etc/kubernetes/cloud.conf: no such file or directory

This error occurs because necessary parameters were not added during the helm chart installation.

▶️ Reinstall cinder-csi by adding the required parameters during the helm installation process.

$ helm install cinder-csi cpo/openstack-cinder-csi \
--version 2.3.0 \
--set secret.enabled=true \         # Required parameter
--set secret.name=cloud-config \    # Required parameter
--namespace kube-system

helm: kubernetes cluster unreachable​

After installing helm, when executing commands through the CLI, a "Kubernetes cluster unreachable" error occurs.

Error Message
Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

Helm needs to register the information required to connect to the Kubernetes cluster, but the error occurs because it cannot find the kubeconfig file containing the cluster information.

▶️ The methods for registering the kubeconfig.yaml file as the $KUBECONFIG environment variable and using the --kubeconfig option are as follows:

  1. Register the $KUBECONFIG environment variable:

    export KUBECONFIG="{kubeconfig path}

  2. Use the --kubeconfig option:

    $ helm --kubeconfig={Download_Path/kubeconfig.yaml} list

Istio: failed to call Webhook​

This error occurs when network communication between the master and worker nodes is not established while configuring Istio on Kubernetes Engine.

Error Message
Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

Master nodes are not included in the container network, so communication with https://instiod/validate (validation webhook) is not possible, leading to this error.

▶️ Add the hostNetwork: true setting to istiod to enable direct communication between the nodes. Also, when using hostNetwork: true, explicitly set dnsPolicy: ClusterFirstWithHostNet.

$ kubectl edit deployment -n istio-system istiod

***
spec:
    hostNetwork: true # 추가
    dnsPolicy: ClusterFirstWithHostNet
containers:
***

dnsPolicy settings information​

  1. dnsPolicy: Default
    Uses the settings in the node's '/etc/resolv.conf' file to handle DNS queries.

  2. dnsPolicy: ClusterFirst
    All DNS queries that do not match the cluster domain suffix (e.g., '.cluster.local') are forwarded to upstream DNS servers by the DNS server.

  3. dnsPolicy: ClusterFirstWithHostNet
    When using hostNetwork: true in a pod, you must explicitly set dnsPolicy: ClusterFirstWithHostNet.
    If a pod using hostNetwork: true is configured with dnsPolicy: ClusterFirst, it will actually operate as if using the dnsPolicy: Default policy. This setting ensures that pods running in host network mode retain cluster DNS configurations.

  4. dnsPolicy: None
    This policy allows the pod to ignore DNS settings in the Kubernetes environment.
    It is used for pods that require custom DNS servers.