Skip to main content

Troubleshoot

This guide explains common issues and resolutions when using the Kubeflow service.


Resolve KServe authentication issue

In Kubeflow environments, KServe requires Dex authentication by default. Therefore, calling an InferenceService endpoint without an authservice_session cookie or appropriate authentication headers may result in a 403 Forbidden error.

▶️ Resolution

To expose a KServe Inference Service as a public API without authentication, modify the existing Istio EnvoyFilter resource as shown below.

Example: Modify authn-filter
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
annotations:
meta.helm.sh/release-name: <your-auth-release>
meta.helm.sh/release-namespace: auth
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
name: authn-filter
namespace: istio-system
spec:
configPatches:
# Add: Configure specific serving API endpoints to bypass authentication
- applyTo: VIRTUAL_HOST
match:
routeConfiguration:
vhost:
name: {isvc_name}.{namespace}.{your_domain}:80 # 예: lb-predictor.kbm-u-example.example.com:80
patch:
operation: MERGE
value:
typed_per_filter_config:
envoy.filters.http.ext_authz:
'@type': >-
type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
disabled: true
# Existing auth filter settings (unchanged)
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: envoy.http_connection_manager
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.ext_authz
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
transport_api_version: V3
http_service:
authorization_request:
allowed_headers:
patterns:
- exact: authorization
- exact: cookie
- exact: x-auth-token
authorization_response:
allowed_upstream_headers:
patterns:
- exact: kubeflow-userid
server_uri:
cluster: outbound|8080||authservice.istio-system.svc.cluster.local
timeout: 10s
uri: http://authservice.istio-system.svc.cluster.local
workloadSelector:
labels:
istio: ingressgateway

After applying the configuration, verify that it is correctly reflected in the Istio IngressGateway. If necessary, restart the relevant pods to confirm that the changes have taken effect.

▶️ Verify application and apply command

Use the following command to check whether the configuration has been successfully applied to Istio IngressGateway.

kubectl get envoyfilter authn-filter -n istio-system -o json | \
jq '.spec.configPatches[] | select(.applyTo == "VIRTUAL_HOST" and .match.routeConfiguration.vhost.name == "{isvc_name}.{project_id}.kubeflow.{your_domain}:80")'

Sample output:

{
"applyTo": "VIRTUAL_HOST",
"match": {
"routeConfiguration": {
"vhost": {
"name": "{isvc_name}.{project_id}.kubeflow.{your_domain}:80"
}
}
},
"patch": {
"operation": "MERGE",
"value": {
"typed_per_filter_config": {
"envoy.filters.http.ext_authz": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute",
"disabled": true
}
}
}
}
}

If necessary, you can manually edit the resource using the command below. Be cautious when editing resources deployed with Helm, as future Helm upgrades may overwrite your changes.

kubectl edit envoyfilter authn-filter -n istio-system

If changes are not reflected, restart the istio-ingressgateway pod to confirm the applied state.

kubectl rollout restart deployment istio-ingressgateway -n istio-system

▶️ Test authentication bypass endpoint (sample request)

After modifications are complete, you can call the inference API without authentication using curl as shown below.

curl --insecure --location 'https://{your_domain}/v1/models/{model_name}:predict' \
-H 'Host: {isvc_name}.{namespace}.{your_domain}' \
-H 'Content-Type: application/json' \
-d '{"instances": [[0.1, 0.1, 0.1, 0.1]]}'
info

This example uses a traffic prediction model. Payload format may differ depending on your model. Be sure to check your model’s input specification and adjust the request accordingly.