Restricting user access to nvidia GPU?

Question

On a server with Tesla Nvidia Card we decide to Restrict user access to GPU. In our server 2 GPU.

# ls -las /dev/nvidia*
0 crw-rw-rw-. 1 root root 195,   0 Dec  2 22:02 /dev/nvidia0
0 crw-rw-rw-. 1 root root 195,   1 Dec  2 22:02 /dev/nvidia1

I found this solve Defining User Restrictions for GPUs

I create local group gpu_cuda

sudo groupadd gpu_cuda

after add user to group gpu_cuda

Create a config file at /etc/modprob.d/nvidia.conf with content

#!/bin/bash
options nvidia NVreg_DeviceFileUID=0 NVreg_DeviceFileGID=0 NVreg_DeviceFileMode=0777 NVreg_ModifyDeviceFiles=0

Create script in /etc/init.d/gpu-restriction

#!/bin/bash
### BEGIN INIT INFO
# Provides:          gpu-restriction
# Required-Start:    $all
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
#  permissions if needed.
### END INIT INFO
set -e
start() {
/sbin/modprobe --ignore-install nvidia;
/sbin/modprobe nvidia_uvm;
test -c /dev/nvidia-uvm || mknod -m 777 /dev/nvidia-uvm c $(cat /proc/devices | while read major device; do if [ "$device" == "nvidia-uvm" ]; then echo $major; break; fi ; done) 0 && chown :root /dev/nvidia-uvm; 
test -c /dev/nvidiactl || mknod -m 777 /dev/nvidiactl c 195 255 && chown :root /dev/nvidiactl; 
devid=-1; 
for dev in $(ls -d /sys/bus/pci/devices/*); 
do vendorid=$(cat $dev/vendor); 
if [ "$vendorid" == "0x10de" ]; 
then class=$(cat $dev/class); 
classid=${class%%00}; 
if [ "$classid" == "0x0300" -o "$classid" == "0x0302" ]; 
then devid=$((devid+1)); 
test -c /dev/nvidia${devid} || mknod -m 750 /dev/nvidia${devid} c 195 ${devid} && chown :gpu_cuda /dev/nvidia${devid}; 
fi; 
fi; 
done
}
stop() {
:
}
case "$1" in
    start)
       start
       ;;
    stop)
       stop
       ;;
    restart)
       stop
       start
       ;;
    status)
       # code to check status of app comes here 
       # example: status program_name
       ;;
    *)
       echo "Usage: $0 {start|stop|status|restart}"
esac
exit 0

I reboot server and run

/etc/init.d/gpu-restriction start

check result in first time is good.

# ls -las /dev/nvidia*
0 crw-rw-rw-. 1 root gpu_cuda 195,   0 Dec  2 22:02 /dev/nvidia0
0 crw-rw-rw-. 1 root gpu_cuda 195,   1 Dec  2 22:02 /dev/nvidia1

but in second time, chown group is back to root

# ls -las /dev/nvidia*
0 crw-rw-rw-. 1 root root 195,   0 Dec  2 22:02 /dev/nvidia0
0 crw-rw-rw-. 1 root root 195,   1 Dec  2 22:02 /dev/nvidia1

Why result back? and how to solve this problem?

Answer 1

nvidia provides the way to set the group ID of its special device files without needing to resort to whatever extra somber script :

Whether a user-space NVIDIA driver component does so itself, or invokes nvidia-modprobe, it will default to creating the device files with the following attributes:

  UID:  0     - 'root'
  GID:  0     - 'root'
  Mode: 0666  - 'rw-rw-rw-'

Existing device files are changed if their attributes don't match these defaults. If you want the NVIDIAdriver to create the device files with different attributes, you can specify them with the "NVreg_DeviceFileUID" (user), "NVreg_DeviceFileGID" (group) and "NVreg_DeviceFileMode" NVIDIA Linux kernel module parameters.

The nvidia Linux kernel modue parameters can be set in the /etc/modprobe.d/nvidia.conf file, mine tells :

...
options nvidia \
        NVreg_DeviceFileGID=27 \
        NVreg_DeviceFileMode=432 \
        NVreg_DeviceFileUID=0 \
        NVreg_ModifyDeviceFiles=1\
...

And I indeed can ls -ails /dev/nvidia0 :

3419 0 crw-rw---- 1 root video 195, 0  4 déc.  15:01 /dev/nvidia0

and witness the fact that access to root owned special files is actually restricted to the members of the video group (GID=27 on my system)

Therefore, all you need to do is to get the group id of your gpu_cuda group and modify (or setup) your nvidia.conf accordingly.

---

Credits : /usr/share/doc/nvidia-drivers-470.141.03/html/faq.html (you'll probably need to adapt the path to your driver version).