58

Am doing some work on a remote CentOS 5.6 machine and my network keeps dropping. Is there a way that I can recover my hung sessions after I reconnect?

EDIT: am doing some updating and installing with yum and am worried this might be a problem if processes keep hanging in the middle of whatever they're doing.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250

7 Answers7

87

There is no way, but to prevent this I like using tmux. I start tmux, start the operation and go on my way. If I return and find the connection has been broken, all I have to do is reconnect and type tmux attach.

Here's an example.

$ tmux
$ make <something big>
......
Connection fails for some reason
Reconect

$ tmux ls
0: 1 windows (created Tue Aug 23 12:39:52 2011) [103x30]

$ tmux attach -t 0
Back in the tmux sesion
Braiam
  • 35,380
  • 25
  • 108
  • 167
  • 1
    dont have tmux, and am not allowed to install things not on my to do list... –  Aug 23 '11 at 09:44
  • 7
    @sergio My heart bleeds :-)) Use `screen`. –  Aug 23 '11 at 09:45
  • 6
    @sergio On a Unix system, virtually any application that does not require root permissions to run may be installed in ones home directory. – Kusalananda Apr 14 '17 at 07:10
38

The recommendations for tmux and screen are both good suggestions. They imply the answer to your question, but don't actually state it. The answer to the question is: there is no way. If you login via ssh, the shell is ended when the connection drops. The work-around is to login and immediately start a virtual terminal of some kind (this can be done with tools like dtach, abduco, tmux, or screen). When the connection drops, the shell you are in is ended, but you can open a new shell and reconnect to the virtual terminal that is running the shell in which you are actually doing your work.

William Pursell
  • 3,497
  • 1
  • 16
  • 19
  • 1
    Assuming the **yum process** still runs (didn't immediately terminate when the shell got SIGHUP), [reptyr](https://github.com/nelhage/reptyr) or similar *might* be sufficient to recover the process or --failing that-- obtain any future output. The shell would generally be terminated on disconnect, though. – Eroen Dec 07 '12 at 22:28
  • @Eroen You mean, even whe using tmux, the OS will terminate the tmux process once it finds that the connection is lost? – Dojo Oct 23 '15 at 12:26
  • @Dojo When the connection is terminated, the tmux instance will be stopped, but the tmux session (and the shells it manages) will stay up. – William Pursell Oct 23 '15 at 15:53
  • If the user lost the connection to the remote ssh server because for example he suddenly accidentally unplugged the RJ-45 cable, I think the ssh server still keeps this session until a predefined timeout (120 seconds for example). So in this case, is there a way to resume this session that is still alive on the ssh server side within 120 seconds after the cable is unplugged? – Gab是好人 Sep 16 '16 at 14:03
10

Someone added reptyr in a comment rather than an answer so I can't upvote it, but it seems a good answer to the edited question and worked great on CentOS.

see: https://github.com/nelhage/reptyr https://www.ostechnix.com/reptyr-move-running-process-new-terminal/

Is there a way that I can recover my hung sessions after I reconnect?

EDIT: am doing some updating and installing with yum and am worried this might be a problem if processes keep hanging in the middle of whatever they're doing.

So same thing, I lost remote connection to a yum update session. So I searched and found this question... then reconnected and used screen this time:

  1. reconnect ssh
  2. screen
  3. install reptyr as per the links above
  4. ps -a | grep yum
  5. reptyr psid

and there I am at the recovered prompt which yum was giving some hours earlier.

Technically the answers are all correct, this is NOT a recovery of a hung session, it is reparenting of an orphaned process to the current session. And very useful it is too...

ingleno
  • 101
  • 1
  • 2
  • This worked to recover my ddrescue. I don't see why everybody is saying it's impossible when this 2011 program can do it just fine. – user Apr 14 '22 at 13:23
  • You saved my day. This answer should be accepted. – Arnie97 Jun 05 '23 at 16:13
9

As William said, the short answer is no, there is no way. To prevent this you could use the screen command before you lose connection

Braiam
  • 35,380
  • 25
  • 108
  • 167
Nicholas Smith
  • 229
  • 1
  • 4
  • from what i understand, i would have to have initiated yum with screen...welll, i didnt. i cant rerun yum it says its still being used and i dont want to force kill it. ..how can i regain control of the running yum? –  Aug 23 '11 at 09:41
  • and byobu is a nice complement to screen , to get it automatically launched in a nice graphical terminal https://launchpad.net/byobu –  Aug 23 '11 at 09:43
  • As far as I know you can't, unless there's some hidden way to redirecting the output from the yum command into your current term session but I can't think of one off the top of my head. – Nicholas Smith Aug 23 '11 at 09:44
  • There are a number of partial solutions to this using debuggers & c. to give a process a new parent shell. [Reptyr](https://github.com/nelhage/reptyr) is one, and [a blog post](http://blog.nelhage.com/2011/01/reptyr-attach-a-running-process-to-a-new-terminal/) describes the issue, the workaround and some other implementations. – Eroen Dec 07 '12 at 22:23
3

No, you can't recover a shell after a disconnection. Instead what you can do is ensure the command you were running continues to run after the disconnect.

To achieve this, use "nohup" and "disown" commands, which are usually builtin commands on most shells, i.e. you don't need to install anything. This only works for non-interactive commands though.

So, the steps would be as follows:

  1. Login to server
  2. Run your command: "nohup sudo yum update &" (this will also log all output to nohup.out file in your current directory)
  3. Run "disown %1"
  4. Feel free to disconnect at this point or wait to be disconnected :)

When you come back to the server, simply "tail nohup.out" to see how the command is doing.

zygis
  • 196
  • 1
  • 3
  • Wouldn't this work pretty poorly in the *general* case? For example, with commands that want input at one time or another... For completely non-interactive commands, it might work well enough and avoid the overhead of (and possibly needing to install) a terminal multiplexer. – user Sep 17 '13 at 14:11
  • Yes, this only works if your commands don't want any user input, I've updated the answer. Overall a terminal multiplexer is the way to go for sure, but it was mentioned that no additional tools were permitted to be installed. – zygis Sep 17 '13 at 15:03
  • Absolutely, it's a valid answer, just slightly limited in its possible use cases. Upvoted with the edit. – user Sep 18 '13 at 07:29
1

As many people have suggested screen and tmux, they both support basic functionality, but have distinct specific features, so one can't say that one is superior to another in all cases. For example, only tmux supports Window-splitting, while only GNU screen can toggle long line wrapping with (Ctrl+a r). See more detailed comparison here.

There also exist tools specifically to fix this problem of ssh:

Autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic. The idea is from rstunnel.

Mosh is a remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes. Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Yaroslav Nikitenko
  • 222
  • 2
  • 12
-1

This can be done using tmux shell. If you get disconnected the fastest way to reattach a session is:

tmux a #0
cmcginty
  • 2,633
  • 4
  • 21
  • 12