Can't ssh, connection terminates immediately with exit status 254

Question

The most recent thing I remember is changing the soft and hard memlock ulimit to unlimited. Now I can't ssh into the machine.

This is the ssh log.

Authenticated to IP ([IP]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LC_CTYPE = 
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Wed Aug  6 07:18:07 2014 from IP-SOURCE
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
Connection to IP closed.
Transferred: sent 4256, received 2504 bytes, in 0.4 seconds
Bytes per second: sent 9616.9, received 5658.0
debug1: Exit status 254

I have tried the following unsuccessfully till now, before posting here:

Trying for a norc noprofile login by ssh user@host 'bash --noprofile'
Forcing a tty by ssh -t user@host
Moved the bash_profile. Tried sshing by ssh user@host.
Renaming the limits.conf file in hopes that it won't be read.
Restarted ssh server.
Run a command via knife as knife ssh "name:server" "come_command"
ssh user@host 'ulimit -l 64', ssh user@host 'ulimit -S -l 64', ssh user@host 'ulimit -H -l 64', ssh user@host 'exec ulimit -H -l 64'

I am not sure if this way of running commands inline: ssh user@host "some_command" works, because I can't get a simple directory listing. I also tried rebooting by ssh user@host 'reboot' but don't think the command was executed. I restarted the machine from AWS also, but unsuccessful.

Is it a lost cause trying to ssh? Is there any way I can ssh into the server?

If the login shell of the remote user is bash, then bash always reads ~/.bashrc over ssh. There's no way around that. Can you sftp there and check/alter your bashrc that way? — Stéphane Chazelas, Aug 06 '14 at 12:07
Tried sftp using CyberDuck, got the `SSH_FXP_INIT` error code. — theTuxRacer, Aug 06 '14 at 12:09
You can also get more detailed output using the `-v` option, or for more, the `-vv` option of for even more, then `-vvv` option. E.g. `ssh -vvv user@host`. That may give you a better idea of where things are going wrong. — Warwick, Aug 06 '14 at 12:43
Do you have any other file (FTP/HTTP...?) or shell (console?) access to the machine which you can use? — Stéphane Chazelas, Aug 06 '14 at 15:14
@StéphaneChazelas Unfortunately, not. Its a machine hosted on AWS, and only SSH access is allowed. I tried SFTP, but I got the error as commented above. — theTuxRacer, Aug 06 '14 at 16:53
If `ssh host 'echo foo >&2'` doesn't output anything, I don't think there's much you can do. — Stéphane Chazelas, Aug 06 '14 at 16:58
Yes, I thought so too. Have started migrating data off that server. Will play with it once I'm done migrating. — theTuxRacer, Aug 07 '14 at 02:26

score 17 · Answer 1 · answered Sep 12 '14 at 08:16

17

Try to change

UsePAM yes

on

UsePAM no

in /etc/ssh/sshd_config (for CentOS)

answered Sep 12 '14 at 08:16

frad sorvensen

171
5

2

It works, but why? – FelikZ Jul 07 '15 at 11:12
1

sorry, but i dont remember the reason) – frad sorvensen Jul 08 '15 at 11:53
Is SELinux in play here ? Wondering if the context of `/etc/security/limits.conf` got hosed, and pam can't use it any more. – steve Jul 15 '16 at 19:46
If you are using a distribution using `systemd` this is a bad solution IMHO. This will prevent `logind`to a open a session and when/if the user is rebooting the machine, some processes started as the user will not be stopped as expected. – Bigon Dec 19 '16 at 19:30
1

In my case, the reason is the user Hard Open Fils Limit per processor is bigger than the `nr_open`. (`nr_open` is reset when restarting the machine): you can check the `nr_open` by `cat /proc/sys/fs/nr_open`, and hard open files `ulimit -Hn`. If you still want the ssh login user use the Hard Open File Config, you need to increase `nr_open`: `sudo sysctl -w fs.nr_open=NUM_BIGGER_THAN_HARD` – Xin Meng Mar 22 '19 at 14:14
This helped me as well (Debian 8). – AKS Aug 15 '19 at 17:25
See here for a possible reason why `UsePAM yes` works: https://unix.stackexchange.com/questions/673153/sshd-what-are-the-practical-effects-of-setting-usepam-no – zardosht Jun 17 '22 at 16:36

score 5 · Answer 2 · edited Apr 21 '20 at 08:06

5

I had a similar problem, I seemed to only see following strange message:

client_input_channel_req: channel 0 rtype exit-status reply 0.

User I was trying to ssh into didn't have a default shell.

I ran the following:

chsh -s $(which sh) username

And then I was able to ssh.

Note:

Running su username was returning exit-code 1 (was failing) and now it just works.

edited Apr 21 '20 at 08:06

AdminBee

21,637
21
47
71

answered May 24 '16 at 19:14

GabLeRoux

885
9
17

1

Holy crap that's cryptic! This was my problem as well. I tried making a "system" user (no HOME and no SHELL) to use SFTP with and I could authenticate but I couldn't use SFTP or SCP or SSH. This fixed my issue. Thanks! – Dave Feb 03 '17 at 19:14

Santosh Garole · Answer 3 · 2019-10-15T07:19:16.997

I was changed the Open files configuration in kernel parameter file /etc/security/limits.conf to unlimited and lost the connectivity.

After reverting it back to the normal for root user i got the connectivity back.

Wrong Example:
## Example hard limit for max opened files
*        hard   nofile unlimited
root     hard   nofile  unlimited
## Example soft limit for max opened files
*        soft   nofile unlimited
root     soft   nofile unlimited

Correct Ex:
## Example hard limit for max opened files
*        hard   nofile 16000
root     hard   nofile 16000
## Example soft limit for max opened files
*        soft   nofile 16000
root     soft   nofile 16000

score 3 · Answer 4 · answered Oct 22 '20 at 15:09

3

In my case it was the zsh shell that stopped working on the server end. However, I was still able to send commands to verify it: ssh -t username@host 'echo $0'

Requested bash ssh -t username@host 'bash -l' stopped the server from terminating the session.

answered Oct 22 '20 at 15:09

mateuszb

141
3

This worked to stop it from exiting. But still don't know why it exited. Is the fact that this worked insight into the why? – William Neely Mar 21 '21 at 18:44
Might be your .zshrc or profile or another file that is automatically run after logging in that triggers your problem. – mateuszb Mar 22 '21 at 13:31

score 2 · Answer 5 · answered May 18 '16 at 15:33

I have encountered this on Mac OS X where the configuration in ~/.bashrc had an problem which caused ssh to work, but sftp to not work. @stéphane-chazelas seems to have the right idea in the comments above.

On the remote system via SSH, rename ~/.bashrc to ~/.bashrc-MOVED and try it again and see if it works; then restore ~/.bashrc and determine the issue.

On my system the ~/.bashrc contained this:

if [ -z "$PS1" ] ; then
    exit
fi

Which was the likely culprit.

score 1 · Answer 6 · edited Apr 21 '20 at 08:05

1

I had the same issue today. First thing I noticed was /var/log was at 100%.

I fixed that and it didn't solve the issue. I couldn't ssh in nor could I login via the GUI, but I could Ctrl+Alt+F2 to get to CLI and login that way. I typed startx and received an error that /tmp/.X0-lock existed.

I removed that file (technically I removed everything from /tmp) and I was able to login via GUI and also via ssh.

edited Apr 21 '20 at 08:05

AdminBee

21,637
21
47
71

answered Jul 15 '16 at 19:36

Jerry P

11
1

Thanks! In my case it was `/home` filled up to 100% which I have detected by using `df -h` on CentOS 7. – RAM237 Aug 10 '18 at 16:27

Can't ssh, connection terminates immediately with exit status 254

6 Answers6

Linked