Troubleshoot

This guide explains common issues and resolutions when using the Kubeflow service.

• Resolve KServe authentication issue

---

Resolve KServe authentication issue

In Kubeflow environments, KServe requires Dex authentication by default. Therefore, calling an InferenceService endpoint without an authservice_session cookie or appropriate authentication headers may result in a 403 Forbidden error.

▶️ Resolution

To expose a KServe Inference Service as a public API without authentication, modify the existing Istio EnvoyFilter resource as shown below.

Example: Modify authn-filter

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  annotations:
    meta.helm.sh/release-name: <your-auth-release>
    meta.helm.sh/release-namespace: auth
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: authn-filter
  namespace: istio-system
spec:
  configPatches:
    # Add: Configure specific serving API endpoints to bypass authentication
    - applyTo: VIRTUAL_HOST
      match:
        routeConfiguration:
          vhost:
            name: {isvc_name}.{namespace}.{your_domain}:80 # 예: lb-predictor.kbm-u-example.example.com:80
      patch:
        operation: MERGE
        value:
          typed_per_filter_config:
            envoy.filters.http.ext_authz:
              '@type': >-
                type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
              disabled: true
    # Existing auth filter settings (unchanged)
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_authz
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
          transport_api_version: V3
            http_service:
              authorization_request:
                allowed_headers:
                  patterns:
                  - exact: authorization
                  - exact: cookie
                  - exact: x-auth-token
              authorization_response:
                allowed_upstream_headers:
                  patterns:
                  - exact: kubeflow-userid
              server_uri:
                cluster: outbound|8080||authservice.istio-system.svc.cluster.local
                timeout: 10s
                uri: http://authservice.istio-system.svc.cluster.local
  workloadSelector:
    labels:
      istio: ingressgateway

After applying the configuration, verify that it is correctly reflected in the Istio IngressGateway. If necessary, restart the relevant pods to confirm that the changes have taken effect.

▶️ Verify application and apply command

Use the following command to check whether the configuration has been successfully applied to Istio IngressGateway.

kubectl get envoyfilter authn-filter -n istio-system -o json | \
jq '.spec.configPatches[] | select(.applyTo == "VIRTUAL_HOST" and .match.routeConfiguration.vhost.name == "{isvc_name}.{project_id}.kubeflow.{your_domain}:80")'

Sample output:

{
  "applyTo": "VIRTUAL_HOST",
  "match": {
    "routeConfiguration": {
      "vhost": {
        "name": "{isvc_name}.{project_id}.kubeflow.{your_domain}:80"
      }
    }
  },
  "patch": {
    "operation": "MERGE",
    "value": {
      "typed_per_filter_config": {
        "envoy.filters.http.ext_authz": {
          "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute",
          "disabled": true
        }
      }
    }
  }
}

If necessary, you can manually edit the resource using the command below. Be cautious when editing resources deployed with Helm, as future Helm upgrades may overwrite your changes.

kubectl edit envoyfilter authn-filter -n istio-system

If changes are not reflected, restart the istio-ingressgateway pod to confirm the applied state.

kubectl rollout restart deployment istio-ingressgateway -n istio-system

▶️ Test authentication bypass endpoint (sample request)

After modifications are complete, you can call the inference API without authentication using curl as shown below.

curl --insecure --location 'https://{your_domain}/v1/models/{model_name}:predict' \
-H 'Host: {isvc_name}.{namespace}.{your_domain}' \
-H 'Content-Type: application/json' \
-d '{"instances": [[0.1, 0.1, 0.1, 0.1]]}'

info

This example uses a traffic prediction model. Payload format may differ depending on your model. Be sure to check your model’s input specification and adjust the request accordingly.