Skip to main content

Troubleshoot

This guide explains common issues and resolutions when using the Kubeflow service.

Resolve KServe authentication issue​

In Kubeflow environments, KServe requires Dex authentication by default. Therefore, calling an InferenceService endpoint without an authservice_session cookie or appropriate authentication headers may result in a 403 Forbidden error.

▶️ Resolution

To expose a KServe Inference Service as a public API without authentication, modify the existing Istio EnvoyFilter resource as shown below.

Example: Modify authn-filter
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  annotations:
    meta.helm.sh/release-name: <your-auth-release>
    meta.helm.sh/release-namespace: auth
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: authn-filter
  namespace: istio-system
spec:
  configPatches:
    # Add: Configure specific serving API endpoints to bypass authentication
    - applyTo: VIRTUAL_HOST
      match:
        routeConfiguration:
          vhost:
            name: <your-inference-service-domain>:80  # e.g., lb-forecast.example.com:80
      patch:
        operation: MERGE
        value:
          typed_per_filter_config:
            envoy.filters.http.ext_authz:
              '@type': >-
                type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
              disabled: true
    # Existing auth filter settings (unchanged)
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ext_authz
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
          http_service:
            authorization_request:
              allowed_headers:
                patterns:
                - exact: authorization
                - exact: cookie
                - exact: x-auth-token
            authorization_response:
              allowed_upstream_headers:
                patterns:
                - exact: kubeflow-userid
            server_uri:
              cluster: outbound|8080||authservice.istio-system.svc.cluster.local
              timeout: 10s
              uri: http://authservice.istio-system.svc.cluster.local
  workloadSelector:
    labels:
      istio: ingressgateway

After applying the configuration, verify that it is correctly reflected in the Istio IngressGateway. If necessary, restart the relevant pods to confirm that the changes have taken effect.