Skip to main content

Troubleshooting

This document provides guidance on how to troubleshoot issues when using Kubeflow.

Notebook SSH session disconnects periodically​

The currently configured SSH session timeout is 5 hours.

▶️ Extending the SSH session duration to "unlimited" is not supported for security reasons. However, you can configure your SSH settings to increase the session duration as follows:

/etc/ssh/ssh_config
Host ssh-nipagpu.kakaocloud.com
    Port {SSH port}
    ServerAliveInterval 100 # added
    ServerAliveCountMax 10 # added

▶️ Additionally, you can check if the SSH service is running by executing the command below in the notebook terminal:

SSH service is running
$ sudo service ssh status
 * sshd is running

SSH session disconnects while using rsync to transfer files​

The currently configured SSH session timeout is 5 hours.

▶️ Extending the SSH session duration to "unlimited" is not supported for security reasons. However, the issue can be resolved using the following method:

  1. Modify the SSH configuration file.
/etc/ssh/sshd_config
Host ssh-nipahpc.kubeflow.kakaocloud.com
    Port {SSH port}
    ServerAliveInterval 100 # added
    ServerAliveCountMax 10 # added
  1. Use the --partial option to resume the file transfer from the point of failure.
rsync -avz --progress -e 'sshpass -p {ssh pw} ssh -p {ssh port}' . {username}@ssh-nipahpc.kubeflow.kakaocloud.com:/home/sun.day --partial
info

If you have a stable object storage service (S3, MinIO, GCS, etc.), you can download data from object storage using tools like MinIO Client.

▶️ Additionally, check whether the SSH service is running using the following command in the notebook terminal:

SSH service is running
$ sudo service ssh status
 * sshd is running

Forgotten password after changing it via SSH​

If you changed the password after SSH access, it cannot be retrieved.

Also, the SSH password shown in Kubeflow > Manage Account is different from the one you set manually.

▶️ In this case, you should either recreate the notebook, or request a password reset via the Support & Issue Reporting channel.

  • It is recommended not to change the password unless necessary.

CONNECT button is disabled or loading for a long time when creating a Jupyter Notebook​

If the CONNECT button remains disabled or shows a loading state for a long time after requesting notebook creation, it may indicate a notebook creation failure.
Check the creation logs from the Activity section at the top of the Kubeflow dashboard.

When it's a resource issue​

If the Activity log shows the following message, it means resource shortage is the cause:

Error message
Reissued from pod/vscode-gpus-0: 0/84 nodes are available: 2 Insufficient memory, 31 node(s) didn't match Pod's node affinity/selector, 53 Insufficient nvidia.com/mig-7g.80gb, 6 Insufficient cpu. preemption: 0/84 nodes are available: 31 Preemption is not helpful for scheduling, 53 No preemption victims found for incoming pod..

▶️ You can check CPU/Memory/GPU allocation and usage in Grafana Monitoring. For more details, refer to the Monitoring Guide.

If the issue persists even though there are no resource limitations, please contact the NIPA High Performance Computing Support via Support & Issue Reporting.

SSH access works, but CONNECT button leads to error on web access​

If the default account's HOME Directory has changed, errors like the following may appear, making web access to the notebook unavailable.
You can resolve it using the steps below.

Error message
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: delayed connect error: 111
Error message
"Error"
Permission denied: Untitled.ipynb
Error message
"Failed to load the jupyterlab-git server extension"
git command not found - please ensure you have Git > 2 installed
Error message
"File Save Error for Untitled.ipynb"
Permission denied: Untitled.ipynb

Restore the changed HOME Directory​

  1. Check if the permissions for the /home/jovyan directory are set to 777.
Check HOME Directory permissions
sun.day@permission-test-0:~$ ls -al /home
total 16
drwxr-xr-x 1 root    root  4096 Jun  4 10:40 .
drwxr-xr-x 1 root    root  4096 Jun  4 10:40 ..
dr-xr-xr-x 7 root    root  4096 Jun  4 10:40 jovyan
drwxr-xr-x 3 sun.day users 4096 Jun  4 10:42 sun.day
  1. Change the permission of /home/jovyan to 777.
Change HOME Directory permissions
sun.day@permission-test-0:~$ sudo chmod 777 /home/jovyan/ 

sun.day@permission-test-0:~$ ls -al /home
total 16
drwxr-xr-x 1 root    root  4096 Jun  4 10:40 .
drwxr-xr-x 1 root    root  4096 Jun  4 10:40 ..
drwxrwxrwx 7 root    root  4096 Jun  4 10:40 jovyan
drwxr-xr-x 3 sun.day users 4096 Jun  4 10:42 sun.day

If you’re unable to resolve it from the logs​

If your notebook is still not being created successfully or log-based resolution is difficult, please contact Support & Issue Reporting.