SSH public key login fails without pattern

Question

(previously posted at stackoverflow by error)

I'm running a bunch of servers with Ubuntu 14.04.1 (sun,hyperion,...) all of which use public keys (OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 on all machines) for rsync without problems. Almost all...

One connection fails without any changes in the configs or keys. Then I'll try to re-add the keys, check for ECDSA, reboot/restart ssh and it works again. Or it doesn't. In this case I just wait a random amount of time (1h up to 3 months) an do the same. This time it fixes the problem - for a while.

The relevant parts of a ssh -vvv diff:

Successful connection

debug1: Host 'hyperion.internal' is known and matches the ECDSA host key.
debug1: Found key in /home/bar/.ssh/known_hosts:20
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/bar/.ssh/id_rsa (0x7f..),
debug2: key: /home/bar/.ssh/id_dsa ((nil)),
debug2: key: /home/bar/.ssh/id_ecdsa ((nil)),
debug2: key: /home/bar/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 95:...
debug3: sign_and_send_pubkey: RSA 95:...
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to hyperion.internal ([172.16.0.10]:22).

Failed connection

debug1: Host 'hyperion.internal' is known and matches the ECDSA host key.
debug1: Found key in /home/bar/.ssh/known_hosts:20
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/bar/.ssh/id_rsa (0x7f..),
debug2: key: /home/bar/.ssh/id_dsa ((nil)),
debug2: key: /home/bar/.ssh/id_ecdsa ((nil)),
debug2: key: /home/bar/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/bar/.ssh/id_dsa
debug3: no such identity: /home/bar/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/bar/.ssh/id_ecdsa
debug3: no such identity: /home/bar/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/bar/.ssh/id_ed25519
debug3: no such identity: /home/bar/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

Things that I've checked several times:

permissions for .ssh/ and id_rsa on all machines
that I'm using the right keys
that ssh-copy-id -i /home/bar/.ssh/id_rsa [email protected] copies the right keys to the right authorized_hosts file

What didn't really help but added to the vodoo/heisenbug effect:

rebooting the machines
restarting the ssh service
fiddling with global ssh options

I've pasted the full logs with some redacted info at pastebin: wall of log

Anything in `/var/log/secure` (or whatever Ubuntu's equivalent is?) — Aaron Copley, Oct 31 '14 at 15:39

score 3 · Accepted Answer · edited Jul 07 '16 at 18:20

3

The problem has been resolved, it wasn't ssh-related at all:

hyperion.internal had an encrypted home, so the key-lookup failed when it was not mounted to /home/europe.

In hindsight quite obvious, but it accounts for the heisenbug effect of not failing when observing the logs on the machine (while being logged in, of course...)

Hope this helps at least some else.

edited Jul 07 '16 at 18:20

chicks

3,793
10
27
36

answered Dec 02 '14 at 14:41

St. Hermes

81
1
7

score 2 · Answer 2 · answered Nov 03 '14 at 20:33

debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password

This indicates the server didn't accept your private key. Unfortunately, the server doesn't provide the client with any more detail about why it didn't accept the key, so you really need to troubleshoot this on the server.

I would start by checking the syslogs in /var/log on the server for any messages from sshd indicating why it rejected the authentication attempt.

If you have root access on the remote server, you can run a debugging instance of sshd and then connect to it with a client. On the remote server, become root and run /path/to/sshd -d -p 2222. This will launch an instance of sshd which listens on port 2222. It will accept one connection, and it will print debugging information to your terminal.

Then, on the client, run ssh as normally but include -p 2222 to connect to the correct port. If the login fails, check the debugging output printed by the server.

Here's the funny part. I did this and I **can** log in without password! Switch back to default port, and I can't any more. So it seems the server-side `sshd` (on default port) is not looking at the `authorized_keys` file, but the test `sshd` on 2222 is doing so. I already tried `service sshd restart` but it doesn't help. **How do I force `sshd` to reload or look at `authorized_keys` file?** — ADTC, Dec 11 '15 at 08:25
**HA!** Found the solution (for CentOS at least). I had to run `restorecon -R -v /root/.ssh` on the server side, and it spewed two lines as though it's fixing something. (I ran it again, and it didn't spew anything.) (**Note:** Previously I had tried applying correct permissions on `.ssh`, home folder, `authorized_keys` etc as per various tips online, but it wasn't enough.). The output was something like: `restorecon reset /root/.ssh/authorized_keys context unconfined_u:object_r:admin_home_t:s0->unconfined_u:object_r:ssh_home_t:s0` — ADTC, Dec 11 '15 at 08:36

score 2 · Answer 3 · answered Jun 18 '15 at 20:05

For me, this was a permissions issue involving the Home directory as well. The permissions for the Home directory on the destination server was set to 775. From what I discovered, the Home directory permissions must be set to 755 or less. This sets it to where no user other than the owner of the home directory is allowed to have write permissions.

score 1 · Answer 4 · answered Nov 02 '14 at 00:10

It looks like a server issue:

debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password

The server does not seem to be sending a reply here. I would try cranking the debug up to 11 on the server side and see what it is whining about.

How much time elapses after the publickey packet is sent waiting for the reply when it fails? If it i

Actually, the 'Authentications that can continue' message is logged when the client receives an auth fail message from the server. — Kenster, Nov 03 '14 at 20:25

score 0 · Answer 5 · edited Jan 17 '16 at 08:51

0

I've often had ssh problems due to the same MAC address being associated with multiple computer names and visa-versa. There are command-line options for ssh to handle this. You can also blow away or edit your known-hosts file to eliminate the problem. Not sure if this helps but this covers 90% of my ssh problems.

edited Jan 17 '16 at 08:51

Mateusz Piotrowski

168
9

answered Oct 31 '14 at 15:17

shrewmouse

347
2
7

score 0 · Answer 6 · answered Nov 14 '14 at 03:16

0

Chown the root-created files (not just chmod) back to that original user account. You could also try testing with Userify, see if it works and that you have public key without errors.

answered Nov 14 '14 at 03:16

fatal_error

1,152
1
11
18

SSH public key login fails without pattern

6 Answers6