5

I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up:

Script#1: myCommands

./myscript --input infile.txt --setting 1 --output out1
./myscript --input infile.txt --setting 2 --output out2
./myscript --input infile.txt --setting 3 --output out3
./myscript --input infile.txt --setting 4 --output out4

Script#2: run.sh

#SBATCH --time=00:02:00
#SBATCH --nodes=2
#SBATCH --cpus-per-task=2

cat myCommands | parallel -j 4

This works, however it only uses one node. The two cores on that nodes are split into 4 threads to make room for 4 jobs as requested by parallel. That is not desirable.

My searching indicates I will need a nodefile and a sshloginfile to accomplish this, but I see no examples online that work with Slurm, only with PBS system.

How can I make the script (1) use both nodes, and (2) not split cores into threads?

cryptic0
  • 191
  • 6

2 Answers2

0

You can just do this with a round robin srun (something like):

jobs=({1..4})
nodes=($(scontrol show hostname $SLURM_NODELIST))
for ((n = 0; n < ${#jobs[@]}; n++)); do
  index=$(expr $n % ${#nodes[@]})
  srun --nodes=1 --ntasks=1 --nodelist=${nodes[$index]} \
       --exclusive ./myscript --input infile.txt \
       --setting $n --output out$n &
done
wait

I presume --cpus-per-task=2 will be given to srun. Let me know if you have any issues. I was messing around with parallel this morning, but I don't see how to fix this issue directly. Additionally, I found that if you scancel a job which contains GNU parallel jobs the running processes don't die unless you use srun.

barrymoo
  • 1
  • 1
0

I currently use GNU Parallel to essentially just "batch" my jobs onto a remote cluster:

Here's an exert with script name removed:

parallel --colsep '\t' \
         --shuf \
         --jobs=25% \
         --delay=1s \
         ssh -q ${remote} \
         sbatch --chdir="${remote_dir}" \
         --job-name="my-job-name-{1}-{2}-{4}-{5}_{6}" \
         --output="${OUTPUT_PREFIX}/joblogs/%x.out" \
         --error="${OUTPUT_PREFIX}/joblogs/%x.err" \
         my-job.sh \
         "${OUTPUT_PREFIX}/" "{1}" "{2}" "{4}" "{3}" "{5}" "{6}" \
         :::: "${COMPARISONS}" \
         :::: "${REPORTS}" \
         :::: "${METHODS}"

Each of the ${COMPARISONS}, ${REPORTS}, and ${METHODS} variables are tab-delimited files that are then passed to the sbatch. Essentially, the this uses parallels' cross-product of arguments to submit the different variations and then job control is left to SLURM, not parallel.

my-job.sh is something like the following:

#!/usr/bin/env bash

#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 4
#SBATCH -t 0-01:00:00

exec ~/bin/job.bin $@

If your queue supports it, you should not need to manually specify the nodes (unless there's some other reason to do so).

I hope this helps. I actually came looking for a way to slow this down so that I don't overwhelm the scheduler (I'm already submitting enough jobs that I QOS myself in the queue until the jobs start completing, most of the jobs take a few seconds to 10 minutes). Parallel has a --delay option which can be helpful when submitting hundreds of jobs.

If you want parallel used in your script you originally start and have it kick off the tasks in SLURM, you can probably better adapt @barrymoo's answer with sbatch and drop the node list (again, unless there's a special requirement to use only a subset of the nodes available to you; most HPC's are designed to be as equitable across nodes in their respective queues.)

kballou
  • 132
  • 1
  • 1
  • 8