2

Not certain what's happened here, but any SSH connection to my server is failing. Yesterday it was working, today it is not.

Server: Centos 6.10 cPanel v86.0.18
Client: Windows systems using Putty SSH

I am on a Windows machine using Putty. I can connect to the IP on port 22. If I type the wrong password, it gives me the password prompt again. If I provide the correct password it either just closes the session window or gives me an error Server unexpectedly closed network connection. On my local firewall, the logs are showing the session ended due to a tcp-fin packet - meaning the remote server closed the session.

From within cPanel I use the Terminal app to emulate a console session and I am able to get root command line: [root@sX-X-X-X ~]#. From there I then ssh to itself: ssh [email protected] - and I get this error: dup() in/out/err failed

I restarted the sshd and the problem persists. What can I do or look at to get this resolved?

UPDATE:

Ran this debug command from prompt:

[[email protected] ~]# ssh -vvvo PreferredAuthentications=password [email protected]
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to X.X.X.X [X.X.X.X] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/identity-cert type -1
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 864 bytes for a total of 885
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,[email protected]
debug2: kex_parse_kexinit: hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96
debug2: kex_parse_kexinit: hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: found hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 909
debug2: dh_gen_key: priv key bits set: 161/320
debug2: bits set: 1000/2048
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 272 bytes for a total of 1181
debug3: check_host_in_hostfile: host 166.62.85.95 filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: host 166.62.85.95 filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host 'X.X.X.X' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: bits set: 1030/2048
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1197
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 52 bytes for a total of 1249
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/identity ((nil))
debug2: key: /root/.ssh/id_rsa ((nil))
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug3: Wrote 84 bytes for a total of 1333
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred password
debug3: authmethod_lookup password
debug3: remaining preferred:
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[email protected]'s password:

debug3: packet_send2: adding 48 (len 72 padlen 8 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug3: Wrote 148 bytes for a total of 1481
Segmentation fault

UPDATE 2:

I have tried ssh via several different user accounts, even root -- I get the same results for all users. Here is the user output info:

$ grep userID /etc/passwd
$ userID:x:501:501::/home/userID:/bin/bash

$ ls -1 /proc/1121/fd | wc -l
4
$ cat /proc/sys/fs/file-max
262144
$ cat /proc/sys/fs/file-nr
139584  0       262144


$ df -h
Filesystem         Size  Used Avail Use% Mounted on
/dev/ploop34902p1  118G   14G  100G  12% /
none               2.0G  4.0K  2.0G   1% /dev
none               2.0G     0  2.0G   0% /dev/shm

Also reboot did nothing, still unable to ssh.

UPDATE 3:

From my SSH logs, its accepting the password and then immediately closing the session:

Apr 21 11:44:09 sX.X.X.X sshd[23144]: Accepted password for myAdminID from 47.200.121.187 port 4283 ssh2
Apr 21 11:44:09 sX.X.X.X sshd[23144]: pam_unix(sshd:session): session opened for user myAdminID by (uid=0)
Apr 21 11:44:10 sX.X.X.X sshd[23144]: pam_unix(sshd:session): session closed for user myAdminID
Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
rolinger
  • 165
  • 9
  • Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/107254/discussion-on-question-by-rolinger-server-is-failing-ssh-connections). – terdon Apr 27 '20 at 08:18
  • Ok, this is annoying. Why does someone get to downvote my OP without reason or explanation? This is a very valid thread with well documented troubleshooting info. But in the last 7 days....this shows a `-10` against my reputation. – rolinger Feb 16 '21 at 18:01
  • There are no downvotes on this question. But yes, one user decided to remove their upvote (you can see this if you look at [the reputation tab](https://unix.stackexchange.com/users/408543/rolinger?tab=reputation) on your profile; note the "unupvote"). But that's just the way the sites work: people are free to vote as they choose. – terdon Feb 16 '21 at 18:32

1 Answers1

0

I realize that this question is from nearly a year ago, but I too ran into this issue, and I've found this question floating around other places online.

The gist is:

  • You are running CentOS 6
  • You connect the SSH server, client immediately segfaults, sometimes spits out dup() in/out/err failed
  • Logs show successful connection and that the client is closing the connection.

What happened is that your server got a bad OpenSSH update - 5.3p1-269.el6.x86_64

You can verify this by running: yum --showduplicates list openssh-server\*

You need to remove this version, and install the older version over it:

rpm -qa openssh*
rpm -e --nodeps openssh-server-5.3p1-269.el6.x86_64
rpm -e --nodeps openssh-5.3p1-269.el6.x86_64
rpm -e --nodeps openssh-clients-5.3p1-269.el6.x86_64
yum -y update openssh

In my case, the ssh executable was also set as immutable, check for the I flag:

lsattr /usr/sbin/sshd

Then unset it:

chattr -i /usr/sbin/sshd

You can then manually re-install openssh-server:

yum install  openssh
yum install  openssh-server
yum install  openssh-clients

I would reboot, but you can service sshd restart

Version should now be:

[root@server ~]# ssh -version
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013

Sources:
https://www.godaddy.com/community/VPS-Dedicated-Servers/SSH-accepting-connection-then-dropping-connection/m-p/161332
https://forums.cpanel.net/threads/error-rexec-of-usr-sbin-sshd-failed-no-such-file-or-directory.678129/
https://forums.cpanel.net/threads/unable-to-ssh-or-sftp-able-to-access-whm.671209/

Ian M
  • 101
  • 2