Is there a tool that could allow the users of our scientific computer to book it exclusively?

Question

We have one beefy ubuntu rig for our research department that can do heavy lifting with its CPUs and GPUs. All of our researchers SSH into the machine and run (machine learning) workloads on the system.

The problem is that we are having collisions with people using the system at the same time, and a simple chatbox where people call dibs hasn't sufficed. Essentially, if researcher A wants to do a time-sensitive benchmark using the GPUs, we don't want anybody else to touch the GPUs in order to maintain validity.

I am wondering if there is a tool available that can schedule and grant users exclusive access to certain commands or devices. All tasks are run via a centralised Conda (python) installation that is accessible via a custom group. Everybody SSHs into the system. Perhaps it would be possible to block SSH access/make GPUs exclusive/block python access?

EDIT: I should have specified earlier that while we do have an active userbase among our research group, we'd prefer to not complicate the setup with a queueing system. A less intrusive (more naive) change to our setup would be heavily preferred. I am sorry for not mentioning this earlier.

Answer 1

The normal way that these things are handled is by using a queuing system and only allowing users to submit jobs through the queue. A commonly used one that I have seen in research institutes I've worked in is TORQUE:

TORQUE is a resource manager providing control over batch jobs and distributed compute nodes. Basically, one can setup a home or small office Linux cluster and queue jobs with this software. A cluster consists of one head node and many compute nodes. The head node runs the torque-server daemon and the compute nodes run the torque-client daemon. The head node also runs a scheduler daemon.

There are several others as well. Talk to your system administrator about setting up a proper queuing system so you can all use the machine effectively. This is the standard setup for accessing shared clusters.

https://help.ubuntu.com/community/TorquePbsHowto

Answer 2

My thoughts on it:

• Disabling simultaneous SSH access doesn't seem right: from my experience people love to open SSH connections and leave them running without having any applications running. Logging in again and again could make many people upset.
• Disabling simultaneous python access doesn't seem right: people may run tasks outside of accessing your CPU/GPU resources.

Here's how I'd approach it:

• There must be a way to monitor your CPU/GPU load.
• You create a task (running under root) which checks these values every second, if they are high enough, you basically chmod 600/700 files_required_to_access_CPU_GPU. Once the load is low or zero, you chmod 644/755 the_same_files.

I could probably program this for you, but I'll need access to the system. I don't see nothing extra complicated.

Answer 3

You can prevent new logins (except by root) by creating a /etc/nologin file. This requires root privileges. BUT we want to be careful to always remove this file when the user is finished with the system. That means we should have some privileged process, that creates the file, awaits some signal or closing, then deletes the file. That's a bit complex using sudo since the process runs in a separate privileged context, and we can't send it signals w/o privilege. We can however communicate from user to privilege process over a pipe.

Consider a bash shell script: _mkexclude

#!/bin/bash
MSG="$1 has exclusive access"
XFILE="/etc/nologin"

[ -e "$XFILE" ] &&  echo "Exclusivity already established, FAIL" >&2  && exit 3
# make exclusive
echo ${MSG} > $XFILE

trap '/bin/rm -rf $XFILE; exit' USR1 QUIT HUP EXIT TERM
# wait to read any line before continuing from stdin cleanup
read f

# make non-exclusive
/bin/rm -rf $XFILE

The script MUST be configured in /etc/sudoers.d as a NOPASSWD command for the proper users. (or group). The script checks to see if there is already a nologin file. If not it creates a new one. Then it WAITS on either a read or a signal. If the originating (bash) shell exits, so will the exclusivity.

A decent way to invoke this script is via bash shell functions ...

exclude() { coproc EXCLUDE (sudo _mkexclusive $(id -un) ); }
disexclude() { echo FOO >&${EXCLUDE[1]} ; }

[give this to users in their ~/.bashrc]

Any user, allowed to sudo the script, can can prevent further logins by executing the $ exclude function. Such a user can, ONLY IN THE CONTEXT of the VERY SAME SHELL execute **disexclude** re-allow others to login.

IF a user closes the shell that creates the exclusion, OR logs-out, then the coproc should remove the $XFILE that prevents others to login.