progress information via pv for directory copy

Question

I need to copy a very large directory (talking in terabytes here) and want to monitor the progress.

I found that pv is a nice utility, but how can I use it for copying directories recursively? (pv src > dst doesn't work because they are directories)

PS: Running on OS X Mountain Lion, pv was installed from Mac Ports

score 21 · Accepted Answer · edited Feb 05 '17 at 16:52

21

Use rsync --progress [SRC] [DST]

Do check the man rsync page because it has a lot of very useful options. -a for archive is a good start, but it depends on your exact requirements.

Copying through a pipe will unnecessarily slow down the copy process, especially if it is files based.

edited Feb 05 '17 at 16:52

Lesmana

26,889
20
81
86

answered Nov 18 '12 at 17:54

jippie

13,756
10
44
64

Why would copying through a pipe slow down the process? The buffering provided by pipes and the multi-tasking associated with pipes is likely to help matters I would say. (and rsync is multi-processes and uses pipes internally IIRC). What's a copy-process that is not file-based? – Stéphane Chazelas Nov 18 '12 at 19:03
Bash is convenient, but not fast. Especially when piping one command to the other. – jippie Nov 18 '12 at 19:25
@StephaneChazelas: Because a pipe's buffer is only 8KB. A program like rsync might want to use a much larger internal buffer. – Zan Lynx Nov 18 '12 at 21:31
1

Note that `--progress` will only show you the per-file progress of each single file, not the overall process of a potentially larger file copy process of several files / folders. – benomatis Jun 16 '20 at 18:09

score 12 · Answer 2 · answered Nov 18 '12 at 17:52

12

You could use tar or pax or cpio:

mkdir -p dst &&
  (cd src && tar cf - .) | pv -trb | (cd dst && tar xpf -)

answered Nov 18 '12 at 17:52

Stéphane Chazelas

522,931
91
1,010
1,501

1

Additionally, you may add a `v` to the `tar` command to see the file names. – peterph Jan 21 '14 at 15:17
So this will zip the source folder before doing the copy, right? My folder is 5 TB in size, so I'm not sure if this is the best way to do it? I really like using pv as it shows a really good progress status, any advice please? – Tak Mar 08 '18 at 13:12
@Tak, no it doesn't zip anything as there's no compression. `tar cf -` outputs a stream containing the files and their metadata, which is fed through a pipe to pv which feeds it untouched to the other `tar` while printing a progress indication, and the other `tar` extracts the data in that stream. All three commands run concurrently and process data as it comes, there is nothing stored on disk, there is nothing stored in memory other than the pipe buffers and the commands internal buffers which just amounts to a few kilobytes. – Stéphane Chazelas Mar 08 '18 at 13:31
@StéphaneChazelas Thanks for the comment. I've tried it and it's working, the only thing is that the progress bar doesn't show the expected percentage it keeps going from left to right. Any idea why? I used pv with this parameters -petra which usually shows the progress bar where the bar kept getting filled until it finishes. I've upvoted your answer as well. – Tak Mar 08 '18 at 14:59
@Tak, `pv` has no way to know how much data there is to transfer. If you have an estimate of that amount, you can pass it to it with the `-s` option (GNU `du -sb src` would give you a good approximation if the files are large enough) – Stéphane Chazelas Mar 08 '18 at 15:06

score 2 · Answer 3 · answered Jan 27 '14 at 17:29

2

Tar.

tar -cf - /var/log/ | pv | tar -C . -x

Example:

# tar -cf - /var/log/ | pv | tar -C . -x
tar: Removing leading `/' from member names
58MB 0:00:05 [ 2.2MB/s] [                   <=>

answered Jan 27 '14 at 17:29

Anomalous Awe

189
1
2
7

score 2 · Answer 4 · edited Apr 13 '17 at 12:36

Here are some commands to copy directories with progress information.

If there are many small files:

cp -av sourcedir targetdir | pv -l -s filecount > logfile

This will report progress based on number of files that are copied.

You can redirect to /dev/null if you don't need logfile.

Use the following command to get filecount:

find sourcedir | wc -l

If there are few huge files:

tar c sourcedir | pv -s size | tar x -C targetdir

This will report progress based on bytes that are copied.

targetdir must exist.

Use the following command to get size:

du -sh sourcedir

If you want to use rsync:

rsync -ai sourcedir/ targetdir/ | pv -l -s filecount > logfile

Get filecount as shown above.

If you are copying on the same system then rsync -a is practically the same as cp -a . The advantages of rsync is when you are copying over the network or if you are updating (or comparing) a previous copy.

See here for more details:

laktak · Answer 5 · 2020-11-17T21:47:18.903

1

I made a progress bar for rsync (in a wrapper):

rsyncy -a FROM/ TO

It looks like this:

rsyncy

More info on GitHub, install with pip3 install --user rsyncy

edited Nov 17 '20 at 21:47

answered Nov 16 '20 at 10:12

laktak

5,616
20
38

score 0 · Answer 6 · edited Dec 29 '14 at 02:59

0

You could do a du -b /directory/ on both source and destination while a normal copy command is in progress, and compare the two. This is just as effective and doesn't slow the copy process by pushing it through a pipe.

edited Dec 29 '14 at 02:59

Hauke Laging

88,146
18
125
174

answered Dec 29 '14 at 02:11

Apostle

1

score -1 · Answer 7 · edited Jan 27 '14 at 01:43

-1

find source-dir -print0 | xargs -0 -I % pv % > dest-dir/%

edited Jan 27 '14 at 01:43

slm

363,520
117
767
871

answered Jan 27 '14 at 01:20

Christopher Hilla

1

3

Please explain tour solutipon, you‘ll want OP to be able to checo it (and learn) – vonbrand Jan 27 '14 at 01:41

progress information via pv for directory copy

7 Answers7

Linked