I have a cluster with several login nodes and many compute nodes (call it the cluster). Then I have another server with a large shared storage (call it the storage). I need to be able to rsync (i.e. ssh) from any node on the cluster to the storage. For that, I generated a private-public key pair on one of the headnodes of the cluster and registered that key with the storage.
At first, ssh-ing prompts for the password but any subsequent ssh works without the password. However, if I switch hosts on cluster (for example, landing on a different login node or ssh-ing from one of the compute nodes), then storage asks for the password again. Subsequent ssh connections from the same cluster host work, but the moment I switch hosts, the password is prompted again one more time.
I was wondering whether this could be a security setting on storage or there is anything specific to be done when sharing the same keys between multiple nodes to prevent this happening? I don't have root access to either cluster or storage and I cannot decide which nodes on cluster are assigned to me at runtime.
I believe my question is related to this one because my home directory (where the keys are located) is remotely mounted on the cluster nodes.