5

We are using rsync to synchronize data from two NFS servers. One NFS server is on the east coast, the other is on west coast. RTT is about 110ms.

On the east coast NFS server I mount the west coasts NFS server mount point.

<server>:/home/backups on /mnt/backups type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.x,local_lock=none,addr=y.y.y.y).

The data is ALREADY on both servers and just to do a validation of the data (e.g. sync folders and when nothing needs to be changes). The following is how long it takes to validate that east coast server is the same as west cost server of a 7GB folder.

The follow takes about 8 minutes to complete over 7GB of data.

rsync -r -vvvv --info=progress2 --size-only /<local_path>/ /<remote_path>/

The following (which avoids using NFS mount) takes about 15seconds to complete over 7GB of data (same as above).

rsync -r -vvvv --info=progress2 --size-only /<local_path>/ <user>@<west_cost_NFS>:/<remote_path>/

again the above is NOT moving any data as the folders are already synchronized, its just validating the data is the same (based on size of files).

I've tried using -o async on client and in /etc/exports async on the server but the client won't ever show async when I run "mount" on the client. I assume async is default. I've tried changing rsize, wsize as well to larger values, but performance doesn't get much better. Am I just SOL on getting any better performance out of NFS?

Kevin Vasko
  • 327
  • 4
  • 12
  • 1
    I can't speak directly to your question, but this would seem to be an excellent application for a ZFS filesystem. Using ZFS snapshots would give you atomic updates of the slave server with no unnecessary data transfer. – Jim L. Apr 19 '21 at 18:10
  • 1
    @JimL. While that would be an option, both of these NFS server are pretty "solidified" in their current operation. It would be a major hassle to convert them. I appreciate the recommendation though. – Kevin Vasko Apr 19 '21 at 18:13
  • 1
    Since rsync appears to be working perfectly, why try to use nfs at all? While I never did nfs stuff at the level you are doing, I know I largely gave up on it when it came to data syncs and backups because it was missing key file attributes that would make rsyncs fail when done over nfs, so using rsync directly seems to be the solution you've already found works perfectly? – Lizardx Apr 19 '21 at 20:04
  • 1
    @Lizardx well its kind of self interest mainly because this isn't the _only_ operation we do. So I figured speeding up this usecase would speed up other usecases as well. – Kevin Vasko Apr 19 '21 at 20:07
  • My understanding of nfs, which is weak, is that it's not really very robust, it's been limping along since Sun went away, and what I saw using it myself is that key things were not being handled, in particular new ext4 file system attributes, which it didn't support at all last I checked. There's probably some way to speed it up if the factor is 100x via straight rsync, but your macro experience mirrors roughly my micro experience, nfs is slow. And may have too many roots back in the 80s to really be fixed, though you never know, dedicates groups of motivated people can do wonders. – Lizardx Apr 19 '21 at 20:22
  • @Lizardx Well it seems to be the "standard" to provide shared data on the network for Linux machines. I mean the only other option is SMB which I know linux does support, but is kind of more targeted towards Windows. It seems like thats really the only two camps to provide shared storage for users. Granted I guess its more targeted towards local (e.g. <1ms latency) and does work a lot better in that world. – Kevin Vasko Apr 19 '21 at 20:27
  • @roaima USA, ATL to SEA. Granted don't think that's super relevant as its a RTT of ~110ms or so. – Kevin Vasko Apr 19 '21 at 20:31
  • It's worth keeping in mind that the same brilliant engineer who created rysnc also created samba, smb, Andrew Tridgell. I think most of the developer hours/mindshare has gone to smb, not nfs, which is a much weaker method and seems to be closer to just maintenance mode. It has to carry a real legacy, somewhat like Xorg, which comes out of a time before the modern internet existed. I believe that the actual standard today for syncing data would be rysnc, not nfs, particularly when it's a network bigger than a LAN. – Lizardx Apr 19 '21 at 20:38
  • fwiw I asked this not too long ago : https://unix.stackexchange.com/questions/731260/nfs-copy-100-mb-sec-and-scp-at-112-mb-sec?noredirect=1#comment1387445_731260 – ron Feb 15 '23 at 19:03
  • @Lizardx somewhat belatedly, I get at least 20x faster throughput between two virtualised Linux systems with NFSv3 than SMBv3. (I've never spent the time getting NFSv4 to work; v3 was sufficient.) – roaima Feb 15 '23 at 20:10

3 Answers3

8

It seems to me you're trying to use rsync wrong. Rsync's protocol is designed for the exact senario of comparing / synchronising large file systems on two separate servers. It does at much as it can locally on both the local and remote machine before comparing in the middle.

Its protocol is designed such that an rsync agent on one machine talks to an rsync agent on another and the protocol is designed to massively reduce the number of round trips (and total data) required to complete the task.

That is rsync is designed to work:

            [fast]        [slow SSH]        [fast]
File system <----> rsync <----------> rsync <----> File system

Rsync is optimised for network performance between the two agents, but it has no way to control the protocol used to access the disk. So when you mount a remote NFS file system you change the profile of network access:

            [fast]        [fast]        [slow NFS]
File system <----> rsync <------> rsync <---------> File system

Rsync can't do anything about this because it has absolutely no control over the NFS protocol.

One concrete difference here is that over NFS, every file must be individually requested. To explore a file tree containing /foo/bar/baz you have to request / [wait] then request /foo [wait] then request /foo/bar [wait] then finally request /foo/bar/baz. At 110ms latency per request that's 330ms latency and you only got one file.

Rsync's protocol between agents doesn't have this limitation. The agent running on the remote machine eagerly compiles a list of every file and directory in the remote file system being synchronised and sends over everything. There's only one request for the entire file tree!

See how rsync works

Philip Couling
  • 17,591
  • 5
  • 42
  • 82
  • 1
    I mean I get that NFS is the slow weak spot, I was mainly trying to figure out how to speed up NFS for this purpose. Or if I'm missing something in general. – Kevin Vasko Apr 19 '21 at 21:04
  • 1
    @KevinVasko No NFS isn't slow. Rsync is ridiculously fast. You're simply not going to get close to what rsync over ssh can do. – Philip Couling Apr 19 '21 at 21:17
  • So why are you saying ‘between agents’? I’m running but a single tool on a single machine. I look at rsync as a more complex ‘cp’ command where it can do “other” stuff when copying files. Im not running anything on the destination system. – Kevin Vasko Apr 19 '21 at 21:23
  • 1
    @KevinVasko Not true. Running over SSH kicks off another rsync process on the remote machine via SSH, the two proceed to talk to each other over the terminal. You can see it by running `ps -ef | grep rsync` on the remote machine while the sync is active. – Philip Couling Apr 19 '21 at 21:28
  • 1
    The nfs remote mount isn't on the local machine, it's on the remote machine, that's what some of the responses and comments are saying if I'm not misreading it. This answer covers that issue. Or rather, the mount itself is on the local machine, but the files aren't, all you've done is a mount a remote file system on the local machine, after that, all the issues happen, since the file systems are on two different machines, and as the answers note, that's going to be really slow with nfs compared to rsync to rsync, which is totally different than rsync to remote mounted nfs file system.. – Lizardx Apr 19 '21 at 21:29
  • @KevinVasko the other hint that you are running an rsync agent on the remote machine is the requirement in the manual: `Note that rsync must be installed on both the source and destination machines.` See [man rsync](https://linux.die.net/man/1/rsync) – Philip Couling Apr 19 '21 at 21:41
  • @PhilipCouling Gotcha, I was just meaning that *I* as the user wasn’t kicking off anything on the remote. So it is doing it transparently. So I assume that when using NFS rsync “thinks” it’s local and performs the operations as such locally? The latency is an issue like you mentioned with NFS and we see it in other operations as well. I guess there isn’t any getting around that. – Kevin Vasko Apr 19 '21 at 21:43
  • 1
    @PhilipCouling Interesting, didn't know it worked that way. – Kevin Vasko Apr 20 '21 at 13:14
1

Your premise is wrong. When you perform a filesystem compare over NFS you are moving a lot of data - metadata about the files. For a large tree that's a lot of individual requests, each of which has latency.

When you use rsync over an SSH connection you're sending a stream of filenames and metadata for the remote side to validate. It may be the same number of files, and therefore the same amount of metadata, but it's streamed and therefore has very low overall latency.

For a 110ms RTT you could easily end up with 15 seconds vs 8 minutes.

Oh, when you start to use rsync replace the -r flag with -a (ideal) or -rt (sufficient). Unless you include file timestamps rsync will end up considering files on each end of the connection to be out of date with respect to each other.

roaima
  • 107,089
  • 14
  • 139
  • 261
  • So there is there an option for handling this or is my solution over ssh the appropriate solution? I am using `--size-only` flag because the destination machines have the sticky bit set to shift owner of the group/files to a different user. Using `-a` wouldn't work for us, `-t I don't think it copied over the correct timestamp when I initially did it so those are probably are different based on when it was "written". – Kevin Vasko Apr 19 '21 at 20:46
  • You're already using `rsync` so continue to do so. Just make sure you're using the client/server model (i.e. the one with `ssh`). Use `rsync -rtv // //` so you maintain file times. – roaima Apr 22 '21 at 22:44
  • Don't use `--size-only` as you'll get burned by `echo hello >file` (not) being updated with `echo world >file`. – roaima Apr 22 '21 at 22:58
  • Will -rtv update the time without retransferring the entire file? It’s 120TB of data and not really wanting to wait on a resync. – Kevin Vasko Apr 22 '21 at 23:01
  • If you can guarantee no mismatches like the one I've shown then you should find that a one-off run that includes `--size-only` will update the metadata without validating the content. Try first with `--dry-run` and let it run for a few files – roaima Apr 22 '21 at 23:27
0

using RHEL 8.7 currently (Feb 2023) having been using RHEL 7.9 previously, I can tell you that in RHEL 7.9 that I could never get NFS vers=4.2 to work. It would max out at vers=4.1. and only if I did not modify the /etc/nfs/nfs.conf or the /etc/sysconfig/nfs files from their original installed state.

This article mentions Some reviews of NFSv4 and NFSv4.1 suggest that these versions have limited bandwidth and scalability and that NFS slows down during heavy network traffic. The bandwidth and scalability issue is reported to have improved with NFSv4.2. This was last updated in April 2022.

https://www.techtarget.com/searchenterprisedesktop/definition/Network-File-System

I also asked this recently fwiw

Is there any reason to use NFS 3 over version 4.2?

So in my opinion from what I've experienced playing with NFS lately that unless you can get vers=4.2 and proto=rdma to work that maybe using nfs v3 as UDP might give the best performance. And especially specifying async in /etc/exportfs on the nfs server. You are correct you will not see async mentioned on the nfs client side as a mount option; I've tested this and observed a change from 50 MB/sec to 100 MB/sec immediately using rsync -P test_60gb.tar /nfsmount by modifying /etc/exports and then doing exportfs -arv. Having proto=tcp with sync is definitely bad for performance with NFS.

I have not had a chance to try proto=udp yet, in fact have had a hard time trying to make it happen.

ron
  • 5,749
  • 7
  • 48
  • 84