Stuck at booting after upgrading

Question

My GPU is NVIDIA - GeForce RTX 3090 Ti, and the OS is Ubuntu 18.04. As my code didn’t work, I checked the versions of python, pytorch, cuda, and cudnn.

Python: 3.6
torch. version : 1.4.0
torch.version.cuda : 10.1 (nvidia-smi shows CUDA version 11.3)
cudnn: 7.6.3

These are not compatible with 3090 Ti, I successfully upgraded Python to 3.9, and Pytorch to 1.12.1+cu102. However, “pip3 install cuda-python” and “pip install nvidia-cudnn” did not work for me. So I followed the steps on the website.

For cuda (tried version 11.8): https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=18.04&target_type=deb_local
For cudnn (tried version 8.6.0, tar file installation): Installation Guide :: NVIDIA Deep Learning cuDNN Documentation

After the installation steps, nvidia-smi shows “Failed to initialize NVML: Driver/library version mismatch”. I found that rebooting would work, but the system is stuck at the rebooting step.

dpkg -l |grep nvidia

iU libnvidia-cfg1-520:amd64 520.61.05-0ubuntu1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-465 465.19.01-0ubuntu1 all Shared files used by the NVIDIA libraries
iU libnvidia-common-520 520.61.05-0ubuntu1 all Shared files used by the NVIDIA libraries
rc libnvidia-compute-465:amd64 465.19.01-0ubuntu1 amd64 NVIDIA libcompute package
iU libnvidia-compute-520:amd64 520.61.05-0ubuntu1 amd64 NVIDIA libcompute package
iU libnvidia-compute-520:i386 520.61.05-0ubuntu1 i386 NVIDIA libcompute package
ii libnvidia-container-tools 1.11.0-1 amd64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container1:amd64 1.11.0-1 amd64 NVIDIA container runtime library
iU libnvidia-decode-520:amd64 520.61.05-0ubuntu1 amd64 NVIDIA Video Decoding runtime libraries
iU libnvidia-decode-520:i386 520.61.05-0ubuntu1 i386 NVIDIA Video Decoding runtime libraries
iU libnvidia-encode-520:amd64 520.61.05-0ubuntu1 amd64 NVENC Video Encoding runtime library
iU libnvidia-encode-520:i386 520.61.05-0ubuntu1 i386 NVENC Video Encoding runtime library
iU libnvidia-extra-520:amd64 520.61.05-0ubuntu1 amd64 Extra libraries for the NVIDIA driver
iU libnvidia-fbc1-520:amd64 520.61.05-0ubuntu1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
iU libnvidia-fbc1-520:i386 520.61.05-0ubuntu1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
iU libnvidia-gl-520:amd64 520.61.05-0ubuntu1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
iU libnvidia-gl-520:i386 520.61.05-0ubuntu1 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
rc nvidia-compute-utils-465 465.19.01-0ubuntu1 amd64 NVIDIA compute utilities
iU nvidia-compute-utils-520 520.61.05-0ubuntu1 amd64 NVIDIA compute utilities
ii nvidia-container-toolkit 1.11.0-1 amd64 NVIDIA Container toolkit
ii nvidia-container-toolkit-base 1.11.0-1 amd64 NVIDIA Container Toolkit Base
rc nvidia-dkms-465 465.19.01-0ubuntu1 amd64 NVIDIA DKMS package
iU nvidia-dkms-520 520.61.05-0ubuntu1 amd64 NVIDIA DKMS package
iU nvidia-driver-520 520.61.05-0ubuntu1 amd64 NVIDIA driver metapackage
rc nvidia-kernel-common-465 465.19.01-0ubuntu1 amd64 Shared files used with the kernel module
iU nvidia-kernel-common-520 520.61.05-0ubuntu1 amd64 Shared files used with the kernel module
iU nvidia-kernel-source-520 520.61.05-0ubuntu1 amd64 NVIDIA kernel source package
iU nvidia-modprobe 520.61.05-0ubuntu1 amd64 Load the NVIDIA kernel driver and create device files
ii nvidia-opencl-dev:amd64 9.1.85-3ubuntu1 amd64 NVIDIA OpenCL development files
ii nvidia-prime 0.8.16~0.18.04.1 all Tools to enable NVIDIA’s Prime
iU nvidia-settings 520.61.05-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
iU nvidia-utils-520 520.61.05-0ubuntu1 amd64 NVIDIA driver support binaries
iU xserver-xorg-video-nvidia-520 520.61.05-0ubuntu1 amd64 NVIDIA binary Xorg driver





  ls -l /usr/lib/x86_64-linux-gnu/libcuda*
    
        lrwxrwxrwx 1 root root 28 Sep 29 05:22 /usr/lib/x86_64-linux-gnu/libcudadebugger.so.1 → libcudadebugger.so.520.61.05
        -rw-r–r-- 1 root root 10934360 Sep 29 01:20 /usr/lib/x86_64-linux-gnu/libcudadebugger.so.520.61.05
        lrwxrwxrwx 1 root root 12 Sep 29 05:22 /usr/lib/x86_64-linux-gnu/libcuda.so → libcuda.so.1
        lrwxrwxrwx 1 root root 20 Sep 29 05:22 /usr/lib/x86_64-linux-gnu/libcuda.so.1 → libcuda.so.520.61.05
        -rw-r–r-- 1 root root 26284256 Sep 29 01:56 /usr/lib/x86_64-linux-gnu/libcuda.so.520.61.05


dkms status

virtualbox, 5.2.42, 5.4.0-126-generic, x86_64: installed
virtualbox, 5.2.42, 5.4.0-72-generic, x86_64: installed

What is the question? You state that your code didn't work and then that you couldn't install `pip` modules and then that the system is stuck rebooting. — Nasir Riley, Oct 20 '22 at 01:26
Please ask here instead https://forums.developer.nvidia.com/c/gpu-graphics/linux/148 — Artem S. Tashkinov, Oct 20 '22 at 09:19

score 0 · Accepted Answer · answered Oct 28 '22 at 09:19

Current driver seems to be causing black screen and freezing machine on boot.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+

I have this issue on bare metal Ubuntu 22.04 after upgrading the driver/cuda packages. However, virtual machines that get similar rtx3090 passthrough GPUs work fine with the same driver and OS versions. Perhaps because they use GPUs only for compute and not for display.

Some people say switching from HDMI input to DP might help. I haven't tested. The fix according to Nvidia rep will be out in the next release, so you can either downgrade to previous version or wait for a fix.

https://forums.developer.nvidia.com/t/nvidia-driver-520-61-05-cuda-11-8-rtx-3090-black-display-and-superslow-modesets/230217/5

Stuck at booting after upgrading

1 Answers1