gzip - redirection or piping?

Question

I will be backing up a large (750GB) disk to an external USB disk using dd.
Should I be using redirection or piping? Which is more efficient? Or is there a difference?

Also, what is the best block size? USB is likely to be the bottleneck here.

dd if=/dev/sda bs=1M | gzip -c > /mnt/sdb1/backups/disk.img.gz

gzip -dc /mnt/sdb1/backups/disk.img.gz | dd of=/dev/sda bs=1M

vs

dd if=/dev/sda bs=1M | gzip -c | dd of=/mnt/sdb1/backups/disk.img.gz

dd if=/mnt/sdb1/backups/disk.img.gz | gzip -dc | dd of=/dev/sda bs=1M

Thanks.

I would try to backup file systems, not raw disk images. And to tune the parameters, you'll have to benchmark. — , Mar 12 '12 at 06:20
`dd` is archaic (any other tool thankfully does away with `bs=` and `count=`, like `ddrescue`), and in your case also pretty much pointless. Just pipe it into `gzip -c disk.img.gz` & `gzip -cd disk.img.gz >/dev/sda`. — , Mar 12 '12 at 11:12
Or do it with rsync. Just construct the command with the parameters, and it will do you everything in the most efficient way it thinks. — vakufo, Mar 12 '12 at 11:53
In the second command of the second listing, what is the `dd if=.../disk.img.gz |` part supposed to do? As far as I understand, `gzip -dc` (=`zcat`) doesn't care for its `stdin` when there's a file argument. — sr_, Mar 12 '12 at 13:57
I don't think Linux can reliably reproduce Windows installations by simply copying the file structure. Hence, the need for bit-wise copying like `dd`. `ddrescue` looks more robust then plain `dd`. Would you use GNU `ddrescue` or `dd_rescue`? What is a good Live CD to download? — mcu, Mar 12 '12 at 20:45

score 11 · Accepted Answer · edited Aug 09 '18 at 11:55

11

You don't need to use dd or piping at all.

</dev/sda gzip >/mnt/sdb1/backups/disk.img.gz
</mnt/sdb1/backups/disk.img.gz gunzip >/dev/sda

I once made a benchmark and found using dd slower than cat for a straight copy between different disks. I would expect the pipe to make any solution involving dd even slower in this case.

edited Aug 09 '18 at 11:55

Stéphane Chazelas

522,931
91
1,010
1,501

answered Mar 13 '12 at 08:38

Gilles 'SO- stop being evil'

807,993
194
1,674
2,175

That *is* an example of using redirection. – psusi Mar 13 '12 at 17:42
@psusi Gah! I meant to write “dd or piping”. Thanks. – Gilles 'SO- stop being evil' Mar 13 '12 at 17:47
Is there a way to combine `pv` in these commands? – Yaron Feb 19 '18 at 21:24
2

@Yaron Sure: replace `gzip` by `pv | gzip` and `gzip -d` by `gzip -d | gv` – Gilles 'SO- stop being evil' Feb 19 '18 at 21:58
So, just to be sure, using redirection on an unmounted device, like ` – mcu Nov 11 '20 at 13:31
1

@mcu Right: `/dev/sda` is the whole disk, and `/dev/sda1` is one whole partition. This just reads the data on the disk/partition without caring if that data may have some meaning as a filesystem. – Gilles 'SO- stop being evil' Nov 11 '20 at 19:02
Would this terminate the copying process somewhat gracefully if the user presses `Ctrl+C`? – mcu Nov 11 '20 at 19:07
1

@mcu It'll interrupt the copy in progress, leaving a partial image file if interrupted during the first command and a half-written (unmountable) disk if interrupted during the second command. A truly graceful interruption that restores the destination disk to its original content if interrupted is not really doable. – Gilles 'SO- stop being evil' Nov 11 '20 at 21:18
How might we adjust the command containing the pipe to add sudo so that it can run as superuser (needed to access the raw disk)? `disk.img` – Steven Lu Dec 21 '22 at 07:34
Update: Seems like `sudo bash -c "disk.img"` is working for me – Steven Lu Dec 21 '22 at 07:37

score 3 · Answer 2 · edited May 23 '17 at 12:39

3

Piping involves one more process and one more user-land copy, so it should be more efficient to use redirection.

But I guess that on nowadays hardware & software caching system, it should not make any real difference. Maybe you can have better results using bs=4k or bs=64k, since it's the pipe's limit under linux. See this question for more detail about different bs parameters.

edited May 23 '17 at 12:39

Community

1

answered Mar 12 '12 at 12:39

Coren

4,970
1
24
43

score 2 · Answer 3 · answered Aug 09 '18 at 11:30

2

Simpler:

gzip -c /dev/sda > disk.img.gz
gzip -dc /mnt/sdb1/backups/disk.img.gz > /dev/sda

answered Aug 09 '18 at 11:30

Mike Redrobe

231
2
5

gzip - redirection or piping?

3 Answers3