3

I've got the problem that on a server I maintain, after switching to a Ceph filesystem for the homedirectories, suddenly ssh login through RSA keys stopped working. I've tried checking several things but none of the solutions I found on the internet worked.

So to start off, yes my permissions for my homedir and .ssh/* are correct:

~$ ls -la .ssh
drwx------ 1 johndoe foo         5 10 apr 20:24 .
drwx------ 1 johndoe foo        50  8 aug 19:51 ..
-rw------- 1 johndoe foo      1617  7 aug 18:56 authorized_keys
-rw-r--r-- 1 johndoe foo       680 10 apr 20:24 config
-rw------- 1 johndoe foo      1675 17 dec  2018 id_rsa
-rw-r--r-- 1 johndoe foo       408 17 dec  2018 id_rsa.pub
-rw-r--r-- 1 johndoe foo      4050 10 apr 20:24 known_hosts

Also, it seems like every user on the server has this problem, so those permissions shouldn't be the problem.

Next thing I checked was ssh -vvv server. This shows the following output:

debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:SOMETHING /home/john/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/john/.ssh/id_dsa
debug3: no such identity: /home/john/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/john/.ssh/id_ecdsa
debug3: no such identity: /home/john/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/john/.ssh/id_ed25519
debug3: no such identity: /home/john/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

It surprised me to see that it only mentioned trying private key id_dsa as opposed to id_rsa, but after trying to connect with a server where I do not have my keys copied, it looked like it did the same thing.

I did try copying over the pubkey again with ssh-copy-id server, which asked me for my password, and then claimed I should try ssh server, but that still asks me for my password.

On the server side, I then tried checking if there were any signs of problems. So, I tried running sudo journalctl -u sshd.service -f, which gave the following output while trying to connect, then being asked for a password, and correctly entering the password:

aug 08 20:42:13 server.domain.com sshd[30191]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=sshgateway.otherdomain.com  user=johndoe
aug 08 20:42:14 octiron.liacs.nl sshd[30191]: Accepted password for johndoe from 1.3.3.7 port 43636 ssh2

So it looks like despite the key being correct, the server still gives an authentication failure.

At this point, I got stuck trying to find what I could do. The verbose output from attempting to connect with ssh clearly lists Authentications that can continue: publickey, so I'd assume pubkeys would work the way I'd expect them to. Permissions are all good, but still I can't figure out how to get the keys to work as they should.

I'd appreciate any tips to try figuring this out.

Edit: When trying to do the same with a local user (with a homedir on the local disk as opposed to a homedir on the ceph mount), this all did work just as expected.

When attempting to log in with just the keys, and then ctrl+C'ing out, the following gets added to /var/log/secure:

Aug  8 23:22:34 server sshd[16775]: Connection closed by 132.229.44.36 port 59610 [preauth]
Aug  8 23:22:51 server unix_chkpwd[16780]: password check failed for user (johndoe)
Aug  8 23:22:51 server sudo: pam_unix(sudo:auth): authentication failure; logname=johndoe uid=200104 euid=0 tty=/dev/pts/0 ruser=johndoe rhost=  user=johndoe
Aug  8 23:22:51 server sudo: johndoe : TTY=pts/0 ; PWD=/home/johndoe ; USER=root ; COMMAND=/bin/cat /var/log/secure
Aug  8 23:22:51 server sudo: pam_unix(sudo:session): session opened for user root by johndoe(uid=0)
Joeytje50
  • 165
  • 6
  • Please check `/etc/ssh/sshd_config` on the server. RSA could be disabled there. Any non-default options? – Ned64 Aug 08 '19 at 18:57
  • Compared to https://fishilico.github.io/generic-config/etc/ssh/sshd_config.default.raw.html it has HostKey lines `rsa`, `ecdsa` and `ed25519` uncommented, `GSSAPIAuthentication yes` and `GSSAPICleanupCredentials no`, and some `AcceptEnv` lines. All of those seemed to me that they shouldn't cause this. Also, I'd be surprised if such a thing would only affect users with a dir in the ceph-mounted `/home/` (ie. root@server can login with keys just fine) – Joeytje50 Aug 08 '19 at 19:08
  • Are the home directories automounted or are they always available? Some automounter configurations make it impossible to log in with a key unless you're already logged in, because the SSH daemon sees the mount point as an invalid home directory and doesn't trigger the automounting. I don't know if this could happen with Ceph (I'm not at all familiar with Ceph). Is each home directory mounted separately or are they part of a whole `/home` filesystem? – Gilles 'SO- stop being evil' Aug 08 '19 at 20:06
  • Are security frameworks such as SELinux and AppArmor enabled? Are the home directories under `/home`? On a failed login attempt, what files under `/var/log` get modified and what's in them? Or what systemd journal entries appear on a failed login attempt, not necessarily entries from the sshd service? What are the exact mount options for the Ceph filesystem? – Gilles 'SO- stop being evil' Aug 08 '19 at 20:09
  • If you run out of ideas, try running an SSH server in debug mode (`sshd -d -p 2222`) and see exactly what it tries to access. Note that these logs are likely to contain a lot of private data. – Gilles 'SO- stop being evil' Aug 08 '19 at 20:11
  • @Gilles the Ceph filesystem is mounted through fstab as such: `bar:/FOO/home /home ceph name=foo,secretfile=/etc/ceph/ceph.client.foo.key,_netdev 0 2`, so it doesn't use automounter. Also, if the filesystem would be mounted with automounter, after mounting it should work again for all users that use /home, right? – Joeytje50 Aug 08 '19 at 21:20
  • @Gilles for your second comment, I haven't done anything with SELinux or AppArmor, so I think that's not the case (unless work a previous admin >1 year ago did could affect this). The only `/var/log/` file that gets changed is `secure`, of which I'll add the output to my Q. What should I do to check other system messages? Any specific command you'd like output of? And for your last question, see my previous reaction ;). Thanks for helping with these questions by the way! – Joeytje50 Aug 08 '19 at 21:24
  • @Gilles I'll try the -d -p 2222 tomorrow, thanks for the tip. – Joeytje50 Aug 08 '19 at 21:25
  • 1
    Please post the output of `ls -ld /home/johndoe/.ssh /home/johndoe/.ssh/*` as well as `getenforce` (both on the server, of course) to check access modes and SELinux status. – Ned64 Aug 09 '19 at 11:37
  • @Ned64 The output of the `ls -ld` commands return the same things as listed in the question, except with `.ssh/` in front; ie. the permissions I show in the questions are of those relative to the path `/home/johndoe/.ssh/*`. The output of `getenforce` is `Enforcing`, but after trying around a bit, it turns out that setting `setenforce 0` my problem no longer appears. So, for now I've changed `/etc/selinux/config` to have `SELINUX=permissive`, but I'll have a look if that's a good long-term solution I'd like to have. If you have any recommendations, I'd love to hear them. – Joeytje50 Aug 09 '19 at 17:44
  • OK, thanks. Sorry if I have overlooked the permissions answer previously. – Ned64 Aug 09 '19 at 17:56
  • @Ned64 Based on https://blog.tinned-software.net/ssh-key-authentication-is-not-working-selinux/ I've tried re-enabling `setenforce 0` and running the suggested `semanage`, `chcon` and `restorecon` commands but those do not seem to fix the problem. The same applies to https://www.pyrosoft.co.uk/blog/2013/01/12/setting-up-ssh-authorized_keys-with-selinux-enabled/, since that recommends roughly the same strategy. – Joeytje50 Aug 09 '19 at 18:01
  • The question is: Can your ceph mount store the special attributes needed for SELinux? Please check the mount options and perhaps the FS's manual. If you call `restorecon -rv ...` again, does it try to change the attributes again? – Ned64 Aug 09 '19 at 18:04

1 Answers1

1

In your new tests you say that disabling SELinux (setenforce 0 or ... permissive) on the server results in a working ssh login. Your SELinux settings are likely wrong for the mounted file system's .ssh and .ssh/* for that user.

Try the following:

restorecon -rv /home/*/.ssh

(or use only that one user).

Ned64
  • 8,486
  • 9
  • 48
  • 86