rsync keeps disconnecting: broken pipe

Question

I'm using rsync to backup my home directory. This has been working fine for a long time now. Here's the command I'm using:

rsync \
    -pavz \
    --delete \
    --exclude 'mnt/' \
    --exclude '.cache/' \
    --exclude 'Videos/' \
    --exclude 'Music/' \
    --exclude 'Documents/virtualbox' \
    /home/"${USER}" "${server}":"${dir}" 2>> "${errorFile}"

However, I switched the server to which I'm backing up and now rsync starts and runs for a few seconds (up to a few minutes), but then stops with the error message

packet_write_wait: Connection to x.x.x.x: Broken pipe
rsync: [sender] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(820) [sender=3.1.1]

Since it is working on other servers, I suspect that the problem is either the connection or the server itself. The connection seems to be stable. I'm connected via cable and I don't see any interruptions. I also tried pinging the server while doing the backup. The ping has a response rate of 100 % even when the backup is breaking up.

I use kerberos to authenticate on the remote server.

I tried several combinations with ServerAliveInterval, ServerAliveCountMax or ClientAliveInterval in my ~/.ssh/config, but to no avail.

It could be that there is something running on the server that kills the rsync command for some reason, but I don't know how to investigate in that. Any ideas?

Answer 1

Your problem might be (lack of) memory. Back when 1GB was big for a server, rsync would fail on me for large datasets. Perhaps the algorithm has improved of the memory capacities has increased, but I haven't seen that problem in 8 years or so. So really, this is an outside shot, but one worth exploring. Try smaller datasets first. You might also try -- as a form on sanity check -- doing a tar-tar:

tar cf - $HOME | ssh ${server} tar xf -

If that also fails after a few minutes, it's not memory.

Answer 2

I have encountered this with rsync in the past as well. The solution that fixed it for me was running it from within a screen session, which was able to help maintain the connection to the remote server.

screen -LS rsync
[execute your rsync command]
Ctrl-A+D to detach from the session

You can check the status by running screen -x rsync (or whatever you decide to name the session if you do give it a name, which is not required). This will re-attach your current shell to that session. Just remember to detach from it again after you have checked the status so that it keeps running in the background.

You can also execute the command to run via screen in the background in one fail swoop by doing [someone please correct me if I'm wrong] screen -dm 'command'. You may want to man screen before trying that last one.

EDIT:

I am editing my answer because you have confirmed that screen provides no assistance in this scenario, but you replied to my comment suggesting to try scp and see what kind of results you get, to which you replied that oddly enough, it worked just fine.

So my new answer is this: use scp -- or ssh (with tar) -- instead of rsync

Granted, scp doesn't support the vast number of features as rsync, but you'd actually be surprised to discover just how many features that it does support that are almost identical to that of rsync.

Real world scenarios for scp and other alternatives to rsync:

Awhile back, I was tasked with creating a shell script that would pull logs from our production servers and store them locally on a web server so that developers could access them for troubleshooting purposes. After trying unsuccessfully to get the Unix team to install rsync on our servers, I came up with a workaround using scp that worked just as well.

That being said, I recently modified the script so that all it uses is ssh and tar -- GNU tar/gtar, to be exact. GNU tar supports many of the options that you will actually find in rsync, such as --include, --exclude, permission/attribute preservation, compression, etc.

The way I now accomplish this is by ssh-ing to the remote server (via pubkey auth) and using gtar -czf - [other options such as --include='*.log' and --exclude='*core*', etc.] -- this writes all of the info to stdout, which is then piped [locally] to tar -xzf so that no changes are made on the remote production server, and all of the files pulled as-is to the local server. It's a great alternative to rsync in this case. The only thing important thing neither tar nor scp support are incremental backups and the level of block-level error checking that rsync features.

The full command I am referring to when using ssh and tar would be something like this (remote is Solaris 10; local is Debian, for what it's worth):

cd /var/www/remotelogs
ssh -C user@remotehost "cd /path/to/remote/app.directories; gtar -czf - --include='*.log' --exclude='*.pid' --exlude='*core*' *" | tar -xz

In your scenario it would be the opposite -- tar -cf - locally, and pipe to remote server via ssh user@remotehost "tar -xf -" -- there is another answer that references this type of behavior but doesn't go into as much detail.

There are a few other options that I have included to speed things up. I timed everything relentlessly to get the execution time as low as possible. You would think that using compression with tar would be pointless, but it actually speeds things up a bit, as does using the -C flag with ssh to enable ssh compression as well. I may update this post at a later date to include the exact command that I use (which is very similar to what I posted), but I don't feel like getting on VPN at the moment since I'm on vacation this week.

On Solaris 10, I also use -c blowfish, because it is the quickest cipher to authenticate with and also helps speed things up a tad, but our Solaris 11 either don't support it or have this cipher suite disabled.

Additionally, if you choose to go with the ssh/tar option, it would actually be a good idea to implement my original solution of using screen if you are doing a backup that will take awhile. If not, make sure your keepalive/timeout settings in your ssh_config are tweaked just right, or this method will also be very likely to cause a broken pipe.

Even if you go with scp, I always find it to be a best practice to use screen or tmux when doing an operation of this sort, just in case. Many times I don't follow my own advise and fail to do this, but it is indeed a good practice to use one of these tools to ensure that the remote job doesn't screw up because of your active shell session getting disconnected somehow.

I know you want to figure out the root cause of your rsync issue. However, if this is really important, these are two great workarounds that you can experiment with in the meantime.

Answer 3

I was having the same problem on OSX El Capitan and fixed this by upgrading to rsync v3.11. The issue was happening for me on v2.6.9.

Answer 4

Kerberos is only for authentication, that should not cause any problems after you have created a successful connection.

Have you tried using the rsync daemon too?

Are your servers on the same network or do you have a firewall/router between?

You could try setup a netcat session between the servers, that is a simple way to try if you have any connection problems between your servers.

On the first server:

nc -lk <port-number>

And on the client

nc <server> <port-number>

You could leave the connection open, and see if the connection keep it, or if you loose the connection. You can also try write something on the client, see that it ends up on the other side.

Answer 5

You have something on the remote server that writes to stdout. This might be in your .profile or .bash_profile. It could be something less obvious like stty or mesg. If in doubt, copy a transcript into your question of you logging in to the server (redact the hostname by all means).

Answer 6

the only time I've had a problem like this with rsync, I tracked it down to a spare ethernet port on another machine that had the same IP address as my target server. If rsync is flaky, it is almost surely a network reliability or (in my case) configuration problem.

Answer 7

I encountered a similar problem when running rsync or manually (either with cp, scp or in Gnome Nautilus) copying large files from a Linux desktop to a low powered ARM based Linux NAS over a gigabit cabled network (no kerberos in my setup). The NAS drives are shared using samba, and are mounted on the client using cifs. The solution for me was to mount the NAS file system from the client without any caching (see also mount.cifs man pages):

sudo mount -t cifs //server.lan/somedir /mnt/somedir/ -o cache=none

Alternatively, when mounting the NAS drive on the client using gvfs in nautilus this problem would not persist when copying large files (but that does not work in combination with rsync though).

Make Linux write to network filesystem concurrently with local disk reads elaborates further upon why this problem might be occurring.

Answer 8

Just upgrade your rsync versions to ensure they are exactly the same on both the sending and receiving PCs. See my answer here: https://serverfault.com/questions/883487/unable-to-rsync-due-to-broken-pipe/988794#988794.