4

I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the environment. Currently I am using rdist. Almost every file I have is changing on a daily basis, sometimes it just changes the date and time inside the file.

I came across few commands like pscp, prsync as well as GNU parallel. I experimented with pscp and rdist on multiple hosts, both are giving similar results.

  1. What is the difference between rdist and prsync in terms of performance? My understanding is that prsync can migrate files on multiple hosts and same is with rdist. My understanding from my tests is that neither prsync nor rdist copy multiple files in parallel on single host; they can only copy file by file in parallel on multiple hosts in parallel. So is there any difference between the two from performance side?

    For rdist, my scripts create a distfile like

    HOSTS( user@server user@server2 user@server3 )
RUN:(/var/inputpath/folder)
-> ${HOSTS}
install (/var/outputpath/folder)

    then I run rdist like following

    rdist-f /dist-file-path -P /path/to/ssh

  2. I tested GNU parallel for local copy using cp and zipping using zip. It is really very fast. This allows copying multiple files in parallel even on local computer.

So my question is, is there a possibility to combine GNU parallel with say pscp or rdist or prsync?

roaima
  • 107,089
  • 14
  • 139
  • 261
user321507
  • 41
  • 1
  • 2

1 Answers1

0

rsync is excellent at syncing 2 dirs if there are few changes. It can even compress the datastream using -z.

Unfortunately rsync is very dependent on latency, so it will often not be able to fill a large connection (e.g. 1 Gbps).

This can be mitigated by running multiple rsyncs in parallel.

From man parallel:

EXAMPLE: Parallelizing rsync

rsync is a great tool, but sometimes it will not fill up the available bandwidth. This is often a problem when copying several big files over high speed connections. The following will start one rsync per big file in src-dir to dest-dir on the server fooserver:

cd src-dir; find . -type f -size +100000 | \
  parallel -v ssh fooserver mkdir -p /dest-dir/{//}\; \
  rsync -s -Havessh {} fooserver:/dest-dir/{}

The dirs created may end up with wrong permissions and smaller files are not being transferred. To fix those run rsync a final time:

rsync -Havessh src-dir/ fooserver:/dest-dir/

If your files are small you can do something like:

find ... |
  parallel -j10 -X rsync -zR -Ha ./{} fooserver:/dest-dir/

Adjust -j10 until you find the optimal number.

The ./{} is needed to make rsync -R take effect and thus copy the directory structure and not just copy all files into a single dir.

Ole Tange
  • 33,591
  • 31
  • 102
  • 198