0

I have an AWS EC2 Ubuntu 20.04 instance, accessed via puTTY SSH. Running a Python process via OpenMPI through the following command on non root:

nohup mpirun python3 job.py

When the shell is open it runs normally. Even though using ps and job doesn't show the process I can see via the nohup.out and the changing file system that it's working even hours after it runs.

When I close the shell the nohup process ends. I should probably also note that when I run the nohup command above I cannot enter any more input (the cursor becomes blank). So when I ran ps and job above I had to open a second shell. I never used nohup before and assume this is abnormal.

I investigated the nohup.out file for errors and found nothing written.

So TLDR:

  1. Closing the SSH instance (puTTY) ends the nohup process

  2. ps and job don't list the nohup process even though I know it's running

I can answer any more questions if needed

Sad CRUD Developer
  • 103
  • 1
  • 4
  • can you edit python script `job.py` ? if yes, you can run it in background, close stdin, stdout and stderr (altough I never did it on ec2 instance), then you can close ssh connection. – Archemar Nov 25 '20 at 21:00
  • @Archemar I can edit job.py but what do you mean by that? – Sad CRUD Developer Nov 25 '20 at 21:03
  • use `sys.stdin.close()` (and stdout and stderr) before calling `pid=os.fork()` exit when pid > 0. be sure to check that mpi layer do not use standard I/O (that wasn't obvious in `mpirun` man page. – Archemar Nov 25 '20 at 21:09

1 Answers1

3

jobs can only show you jobs of the current shell.

Bare ps shows processes with the same effective user ID as the current user and the same controlling terminal as the invoker. You need e.g. ps -e to see other processes (not necessarily all processes in the OS though).

If you run

nohup something … &

then you will be able to run jobs or ps in the same shell. Don't expect to see nohup. Its task is to run something in a specific way, it doesn't need to last. Expect to see something.

So it may be your process survived and you only assumed it had died. Or it may be it really died. See Why was process killed with nohup? The answer there mentions two possibilities:

  • KillUserProcesses=yes in logind.conf (systemd);
  • or the spawned process implements its own handler for SIGHUP which overrides the protection of nohup.

More here: Difference between nohup, disown and &. If I understand correctly, disown can help if the spawned process implements its own handler for SIGHUP. Use it like this:

nohup something … &
disown
Kamil Maciorowski
  • 19,242
  • 1
  • 50
  • 94