Batch Processing with one task at a time

Question

I'm used to IBM i (AS/400) batch processing with flexible queueing configuration possibilities. I'm searching a similar facility for Linux Batch processing.

Most important is that one job at a time being taken from the queue and executed. After that, the queue is scanned for more entries. If there are any, the next task will be fetched to run.

at -b comes relatively close but if a job takes so little resources that the load stays down, more tasks get executed in parallel which is not desired.

Run-parts would be another option but it is too static in behavior and doesn't run as a daemon to look for new entries in a given queue. I could run it as a regular cron job but I need to take care of jobs to delete from the queue myself.

Do I have to code my own or extend at to achieve my desired functionality or are there alternate batch queueing mechanisms available?

Have you looked at `ts`/`tsp`? It has some sort of task spooler server and allows you to run commands after another and should offer some advanced functionality. — rudib, Dec 28 '19 at 18:52
Also see: http://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html — rudib, Dec 28 '19 at 18:54
@rudib Thanks a lot! Ts seems to match closely what I need. If you move your comment to an answer, I can mark this as done. — PoC, Dec 29 '19 at 19:43

Abdurrahim · Answer 1 · 2019-12-28T19:50:49.927

4

You can do this just using bash:

1- create fifo using mkfifo (you can create file too)

mkfifo mybuff

2- tail the fifo and pass each line to your executor (this will be kind of task server)

while IFS= read -r line
do
  echo "Running: '$line' with bash"
  bash -c "$line"
  echo "Finished '$line' with exit code: $?"
done < <(tail -f mybuff)

3- Send your commands to queue for example (this will client requesting tasks)

echo 'sleep 10' >> mybuff
echo 'echo "hello world"' >> mybuff
echo 'my fancy command to be executed' >> mybuff
...

PS: You can simplify step 2 like example below but this will finish task listener when you send "exit" to your command buffer

tail -f mybuff | bash -x

edited Dec 28 '19 at 19:50

answered Dec 28 '19 at 19:26

Abdurrahim

196
3

I get an error `Syntax error: redirection unexpected` on line 6 of the "server" part – theV0ID Aug 20 '20 at 00:53
@theV0ID you can use bidirectional pipe for old versions of shell: tail -f mybuff | while read LINE; do bash -c "$LINE"; done – Abdurrahim Aug 20 '20 at 17:05

score 3 · Accepted Answer · answered Dec 29 '19 at 20:12

ts - task spooler. A simple unix batch system should be up for the task. It runs a task spooler/queue server and you can add a new task with a simple command like:

tsp yourcommand

You can specifify the number of slots - aka how many jobs should be executed at a time. By default this is set to one, AFAIK. It also has some advanced functionality, like running a command only if a certain previous command succeeds. You can view the queue and details for each job as well. You can find more details in the manpage.

Batch Processing with one task at a time

2 Answers2