Troubleshooting

This guide explains common issues and resolutions when using the Kubeflow service.

• Resolve KServe authentication issue
• Resolve KServe timeout issue

---

Resolve KServe authentication issue

In Kubeflow environments, KServe requires Dex authentication by default. Therefore, calling an InferenceService endpoint without an authservice_session cookie or appropriate authentication headers may result in a 403 Forbidden error.

▶️ Resolution

To expose a KServe Inference Service as a public API without authentication, modify the existing Istio EnvoyFilter resource as shown below.

Example: Modify authn-filter

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  annotations:
    meta.helm.sh/release-name: <your-auth-release>
    meta.helm.sh/release-namespace: auth
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: authn-filter
  namespace: istio-system
spec:
  configPatches:
    # Add: Configure specific serving API endpoints to bypass authentication
    - applyTo: VIRTUAL_HOST
      match:
        routeConfiguration:
          vhost:
            name: {isvc_name}.{namespace}.{your_domain}:80 # 예: lb-predictor.kbm-u-example.example.com:80
      patch:
        operation: MERGE
        value:
          typed_per_filter_config:
            envoy.filters.http.ext_authz:
              '@type': >-
                type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
              disabled: true
    # Existing auth filter settings (unchanged)
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_authz
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
          transport_api_version: V3
            http_service:
              authorization_request:
                allowed_headers:
                  patterns:
                  - exact: authorization
                  - exact: cookie
                  - exact: x-auth-token
              authorization_response:
                allowed_upstream_headers:
                  patterns:
                  - exact: kubeflow-userid
              server_uri:
                cluster: outbound|8080||authservice.istio-system.svc.cluster.local
                timeout: 10s
                uri: http://authservice.istio-system.svc.cluster.local
  workloadSelector:
    labels:
      istio: ingressgateway

After applying the configuration, verify that it is correctly reflected in the Istio IngressGateway. If necessary, restart the relevant pods to confirm that the changes have taken effect.

▶️ Verify application and apply command

Use the following command to check whether the configuration has been successfully applied to Istio IngressGateway.

kubectl get envoyfilter authn-filter -n istio-system -o json | \
jq '.spec.configPatches[] | select(.applyTo == "VIRTUAL_HOST" and .match.routeConfiguration.vhost.name == "{isvc_name}.{project_id}.kubeflow.{your_domain}:80")'

Sample output:

{
  "applyTo": "VIRTUAL_HOST",
  "match": {
    "routeConfiguration": {
      "vhost": {
        "name": "{isvc_name}.{project_id}.kubeflow.{your_domain}:80"
      }
    }
  },
  "patch": {
    "operation": "MERGE",
    "value": {
      "typed_per_filter_config": {
        "envoy.filters.http.ext_authz": {
          "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute",
          "disabled": true
        }
      }
    }
  }
}

If necessary, you can manually edit the resource using the command below. Be cautious when editing resources deployed with Helm, as future Helm upgrades may overwrite your changes.

kubectl edit envoyfilter authn-filter -n istio-system

If changes are not reflected, restart the istio-ingressgateway pod to confirm the applied state.

kubectl rollout restart deployment istio-ingressgateway -n istio-system

▶️ Test authentication bypass endpoint (sample request)

After modifications are complete, you can call the inference API without authentication using curl as shown below.

curl --insecure --location 'https://{your_domain}/v1/models/{model_name}:predict' \
-H 'Host: {isvc_name}.{namespace}.{your_domain}' \
-H 'Content-Type: application/json' \
-d '{"instances": [[0.1, 0.1, 0.1, 0.1]]}'

info

This example uses a traffic prediction model. Payload format may differ depending on your model. Be sure to check your model’s input specification and adjust the request accordingly.

Resolve KServe timeout issue

When using large models such as LLMs, timeout issues may occur for various reasons.
(For all cases other than the two listed below, check whether the timeout is set on the client side.)

▶️ Resolution

1. Load balancer timeout

Configure timeout settings by referring to KakaoCloud’s Load Balancer documentation.

• Network load balancer > Manage listeners > Set idle timeout

2. InferenceService timeout

Set a timeout when defining the InferenceService (spec.predictor.model.timeout).

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: kanana-isvc
spec:
  predictor:
    model:
      modelFormat:
        ...
      args:
        ...
      resources:
        ...
      timeout: 100 # Set to 100 seconds