Run mpirun and ssh-add remotely on the same session

Question

I am developing a program that is running on multiple machines using MPI.

I have machines on Amazon EC2 where I can start mpirun from one of them (the master).

Everything works as I expect if I ssh to the master machine, then trigger ssh-agent and then ssh-add my_rsa_key.

What I want to do, is to be able to have a script on my local machine where I trigger the mpirun on the master remotely with ssh, the problem is that I get Permission denied (Public key) because ssh-add is not receiving requests over that session (I suppose).

Here is a very basic draft of the deploy script I am trying right now.

#!/bin/bash

MACHINES_LIST="M1
M2"     

echo $MACHINES_LIST | tr " " "\n" | while read fn; do
    echo "$fn"

    echo "deploying and compiling to : $fn ..."
    scp -i key "sample.c" user@$fn:/home/user
    ssh $fn 'mpicc sample.c -o sample' &

done

echo "uploading lists of hosts to master M"
scp -i key .hosts user@M:/home/user

echo "starting mpirun on master M"

##### Here mpirun needs to execute after ssh-add
ssh M 'exec ssh-agent bash;ssh-add my_rsa_key;mpirun --hostfile .hosts -np 10 sample'

Is there a way to make mpirun execute (remotely) within a session where ssh-add is running?

Answer 1

I think the problem is in this line:

ssh M 'exec ssh-agent bash;ssh-add my_rsa_key;mpirun --hostfile .hosts -np 10 sample'

There are at least a couple of issues:

1. the exec ssh-agent part will replace the current shell (the remote shell started by ssh) with [ssh-agent], so the following commands are never run.

2. in order for [ssh-add] to talk to [ssh-agent], a few environment variables must be defined, telling the location of agent socket.

So the usual way of starting ssh-agent is via the shell eval command:

eval $(ssh-agent -s)

I would therefore change the last line of your script to:

ssh M 'exec $(ssh-agent); ...(keep the rest unchaged)'

Note that you must use the single quotes ' here, otherwise the $(...) will be expanded by the shell running the script, i.e., an ssh-agent will be started on your local machine.

Alternatively, you could configure all your EC2 hosts (M and M1+M2) and your local ssh client to allow agent forwarding, and you just run the agent locally. Then you would only need to be sure that the key that you add locally is authroized on every remote host.

Answer 2

How about simplify this whole problem by using ssh-agent-forwarding instead? You can pass on credentials from a local ssh-agent for use on a remote machine so you don't have to worry about running a new agent and adding keys there.