I've created a systemd service that uses ssh to call home to a central server.
[Unit]
Description=Create a tunnel in the cloud back to SSH on this machine
After=network-online.target
[Service]
User=mindhive
ExecStart=/usr/bin/ssh -o ServerAliveInterval=20 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -i /home/mindhive/.ssh/tunnel.id_rsa -N -T -R1822:localhost:22 tunnel@***server-hidden***
RestartSec=60
Restart=always
[Install]
WantedBy=multi-user.target
I use this on many of our servers, all Ubuntu 16.04. Suddenly one of them has stopped working. I can see from the logs (below) it's to do with access to .ssh in the homedir. The service is set to Restart=always and RestartSec=60, so after a reboot it sits there trying again every minute, failing each time. However, if I manually sudo systemctl restart ssh-tunnel.service then it starts no problem.
You can see from the logs below (output of journalctl) it failing and retrying but when started manually at 14:54:10 it starts fine.
From Googling so far I've tried adding both WorkingDirectory=~ and ProtectHome=off to the service. But that's made no difference.
Why can't ssh access the user's homedir when started by systemd after a reboot, but it can if started manually by systemctl?
-- Reboot --
Feb 24 14:50:12 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:50:12 ***servername*** ssh[1252]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:50:13 ***servername*** ssh[1252]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:50:14 ***servername*** ssh[1252]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:50:15 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:50:15 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:50:15 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:51:15 ***servername*** systemd[1]: ssh-tunnel.service: Service hold-off time over, scheduling restart.
Feb 24 14:51:15 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:51:15 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:51:15 ***servername*** ssh[1367]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:51:16 ***servername*** ssh[1367]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:51:17 ***servername*** ssh[1367]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:51:18 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:51:18 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:51:18 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:52:18 ***servername*** systemd[1]: ssh-tunnel.service: Service hold-off time over, scheduling restart.
Feb 24 14:52:18 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:52:18 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:52:18 ***servername*** ssh[1370]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:52:19 ***servername*** ssh[1370]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:52:20 ***servername*** ssh[1370]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:52:20 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:52:20 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:52:20 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:53:21 ***servername*** systemd[1]: ssh-tunnel.service: Service hold-off time over, scheduling restart.
Feb 24 14:53:21 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:53:21 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:53:21 ***servername*** ssh[1393]: Warning: Identity file /home/mindhive/.ssh/tunnel.id_rsa not accessible: No such file or directory.
Feb 24 14:53:21 ***servername*** ssh[1393]: Could not create directory '/home/mindhive/.ssh'.
Feb 24 14:53:22 ***servername*** ssh[1393]: Failed to add the host to the list of known hosts (/home/mindhive/.ssh/known_hosts).
Feb 24 14:53:23 ***servername*** systemd[1]: ssh-tunnel.service: Main process exited, code=exited, status=255/n/a
Feb 24 14:53:23 ***servername*** systemd[1]: ssh-tunnel.service: Unit entered failed state.
Feb 24 14:53:23 ***servername*** systemd[1]: ssh-tunnel.service: Failed with result 'exit-code'.
Feb 24 14:54:10 ***servername*** systemd[1]: Stopped Create a tunnel in the cloud back to SSH on this machine.
Feb 24 14:54:10 ***servername*** systemd[1]: Started Create a tunnel in the cloud back to SSH on this machine.
