31

I need to copy one disk to another. I tried with the command below and it takes nearly a day to copy 1 TB of disk in federo.

dd if=/dev/sda of=/dev/sdb

I have tried the same on a Unix(HP-UX) system with the command below and it completes within a few hours

dd if=/dev/sda of=/dev/rdsk

What is the alternative I could use to copy from disk to disk as faster?

KKD
  • 610
  • 1
  • 6
  • 20
  • 2
    `cp /dev/sda /dev/sdb` or (`pv /dev/sda > /dev/sdb` to get a progress bar) would be a lot quicker. Why would you use `dd` here? `dd` would only be useful with things like `conv=sync,noerror` to handle disks with errors, but even then it would make more sense to use things like `ddrescue` instead (see also `pv`'s `-E` option). – Stéphane Chazelas Nov 10 '16 at 09:27
  • 2
    @StéphaneChazelas [`cat` may be even quicker](http://unix.stackexchange.com/questions/9432/is-there-a-way-to-determine-the-optimal-value-for-the-bs-parameter-to-dd/9492#9492) but the difference isn't that dramatic (maybe bigger for device-to-device than file-to-file as in my experiment). – Gilles 'SO- stop being evil' Nov 10 '16 at 12:44
  • 8
    _"I have tried the same on a Unix system"_ - So on what type of system did you try the first, if not a Unix? Also, what hardware, etc, yaddayadda. – marcelm Nov 10 '16 at 13:13
  • Welcome to [`dd` pitfall #1](https://www.reddit.com/r/linux/comments/b2to3/pitfalls_of_using_cp_instead_of_dd/) – Dmitry Grigoryev Nov 10 '16 at 15:32
  • Used the first in HP-UX (Integrity blade) and earlier used Solaris machine as well. – KKD Nov 11 '16 at 04:15
  • @KDD Sooo, the command is slow on the HP-UX machine, but fast on the Solaris machine? How do the hard disks differ (size, speed grade, etc)? It's really impossible to answer your question with the level of detail you provide. – marcelm Nov 11 '16 at 12:24
  • @marcelm I used command `dd if=/dev/sda of=/dev/rdsk` in both HPUX and solaris it's fast in both as of rdsk. Required point is I need to copy the disk much faster as like I have used HPUX and Solaris. – KKD Nov 15 '16 at 12:00
  • Why dd for big disks? use tar... in a phone or would write a proper answer. – Rui F Ribeiro Nov 17 '16 at 07:09
  • The speed also depends on how the disk is connected. If both are external disks connected on the same USB hub then performance will suffer. If they are internal on distinct disk controllers, it will be much faster. – Florian F Nov 21 '21 at 20:44

4 Answers4

47

dd has many (weird) options, see dd(1).

You should explicitly state the buffer size, so try

dd if=/dev/sda of=/dev/sdb bs=16M

IIRC, the default buffer size is only 512 bytes. The command above sets it to 16 megabytes. You could try something smaller (e.g. bs=1M) but you should use more than the default (especially on recent disk hardware with sectors of 4Kbytes, i.e. Advanced Format). I naively recommend some power of two which is at least a megabyte.

With the default 512 bytes buffer size, I guess (but I could be very wrong) that the hardware requires the kernel to transfer 4K for each 512 bytes block.

Regarding rdsk, the sd(4) man pages say:

At this time, only block devices are provided. Raw devices have not yet been implemented.

Increase of dd's buffer size will give you more performance for read and write operations. Now all disks have hardware read/write buffer. But if you will increase dd's buffer size more than hardware buffer its performance will decrease because dd will read from first disk to buffer when second disk will have written all from its own hardware buffer. You need set bs option of dd command each time different value for different devices.

Khirgiy Mikhail
  • 332
  • 1
  • 5
Basile Starynkevitch
  • 10,411
  • 1
  • 32
  • 52
  • Whether rdsk is available in Linux systems? I have used in Unix systems. – KKD Nov 10 '16 at 06:02
  • 1
    The page cache will probably deal in 4Kb blocks whatever you do, but you can control how many syscalls dd uses to read that 4Kb. I'm sure there's some read size above which the cost of stalling writes is more expensive than the saved syscalls, but have no idea where the sweet spot is. – Useless Nov 10 '16 at 12:17
  • A block size of a few MB is better than the default 512B but [when I benchmarked this](http://unix.stackexchange.com/questions/9432/is-there-a-way-to-determine-the-optimal-value-for-the-bs-parameter-to-dd/9492#9492) I found that `cat` did just as well (for filesystem-to-filesystem transfer, direct block-to-block may have different performance characteristics). However the difference wasn't dramatic in any case. – Gilles 'SO- stop being evil' Nov 10 '16 at 12:43
  • I thought using the wrong BlockSize causes corruption if I try to use the new disk with the same filesystem and a different BlockSize – cat Nov 10 '16 at 13:36
  • @BasileStarynkevitch If we doesn't have RAW device implementation yet. How the Oracle DB RAW I/O operations been performed on Linux? [Raw Device Mapping](https://access.redhat.com/solutions/10163) – KKD Nov 11 '16 at 04:34
  • I guess that it uses the block device, eg. `/dev/sdc2` – Basile Starynkevitch Nov 11 '16 at 06:52
  • I use bs=1 in conjunction with skip= to skip in 1 byte offset. Is there a way to achieve higher write speeds with a byte count skip ? – elig Sep 18 '18 at 01:13
  • 2
    Interestingly, in macOS (a SUS-certified, btw) its [faster to use `/dev/rdiskX`](https://www.raspberrypi.org/forums/viewtopic.php?t=21995) as the target when performing `dd`. – adib Jun 02 '19 at 13:41
  • 4
    in case you wonder whats going on (like I did) add also `status=progress` that will print the whole operation progress. – Aleksander Lech Aug 04 '19 at 14:22
24

Years back in Unix-land dd was the required way to copy a block device. That has carried forward as cargo-cult knowledge even though (on Linux-based systems, at least) cat is almost always faster than dd.

However, even back in history a decent block size helped reduce the number of (slow) system calls, given that each system call triggered an I/O operation. The default block size is 512 bytes (one disk sector). Collecting multiple disk blocks together into a single read was - and is - also acceptable. This example uses a 32MB block size:

dd bs=$((512*2048*32)) if=/dev/source of=/dev/target

As an aside, if you're reading from a stream that's being uncompressed, or one that's coming from a remote network source, you may find that you get short reads less than the desired block size. These should be handled by telling dd to re-read until it gets sufficient input to fill the block, iflag=fullblock; without this you can on occasion end up with a corrupted output stream. That's not the case here, but I've included it for completeness.

Back to the original question, and on current Linux-based systems disks can be most efficiently copied with a simple cat

cat /dev/source >/dev/target

(As noted in the comments on your question pv can be substituted for cat and will give you an indication of progress and throughput.)

roaima
  • 107,089
  • 14
  • 139
  • 261
  • 3
    Specifically, the reason dd had to be used was a bug in GNU cp and a bug in the linux kernel in the early 90s. The reasons for using dd on historical unix systems was very different, and wanting to copy a whole block device was an unusual thing to want to do. – Random832 Nov 10 '16 at 14:50
  • 1
    @Random832 wanting to copy an entire disk would have been unusual, but I do remember needing to copy partitions around (big ones - 150 or even 200MB) – roaima Nov 10 '16 at 14:55
  • 4
    (The specifics of the bugs: the kernel reported disk usage sizes incorrectly [leading cp to conclude that every source file was a sparse file], and cp did not zero out blocks when copying from a sparse file to a device destination. So any zero block in your source would have whatever garbage happened to already be on the disk) – Random832 Nov 10 '16 at 15:06
  • 2
    I love this kind of answer. Thanks for the info. Here's your updoot. – catbadger Dec 10 '19 at 13:26
10

Generally, dd can be avoided in favor of some alternatives. There are several good reasons to use GNU ddrescue instead. In Ubuntu, you can install it with:

sudo apt-get install gddrescue

and just plain ddrescue to use. Note that differently from the package name, the executable does not have the initial g.

Using it is as simple as:

ddrescue inputFile outputFile logFile

The log file (named whatever you choose) lets you pause/stop and restart, without redoing the previous work, which is useful when doing large clones or recovery of disks. By default, it displays progress, current copy speed, average copy speed and number of bad blocks found.

It uses sensible defaults for block size, so copy speed is always as fast as the device can handle, in my experience at least (I've cloned many hundreds of drives with it, all sizes and types).

Often times, drives that are starting to fail have speed issues such as occasional patches of slowness, low average speed, sudden long pauses (bad sectors) or complete resets (severe surface errors). ddrescue can help you identify all the above and restart your clone (provided you specified a log file) even if your drive is resetting itself.

G-Man Says 'Reinstate Monica'
  • 22,130
  • 27
  • 68
  • 117
Roger Heathcote
  • 303
  • 3
  • 9
  • wth is initial g ? –  Jun 30 '20 at 20:54
  • 1
    In the Debian repo's it's called 'gddrescue' but once it's installed you just type 'ddrescue' to run it. It's called that because it's full name is 'GNU ddrescue' and there was some different package called 'ddrescue' already. – Roger Heathcote Jul 01 '20 at 00:14
6

Very nice question. The raw interface is implemented on some unix systems(tru64,hpux,solaris) but not on linux. The raw interface makes transfer faster because unix I/O is skipped. The block interface(/dev/dsk or /dev/disk) is slower because it uses the unix I/O system. To speedup dd (gnu dd can) use bs=30M or bs=20M depending on your hw. The short answer is : NO it is not implemented,at least as far as I know. I am using linux since the old times of kernel version 2.2 and have never seen rdsk used on unix.

Erik
  • 115
  • 1
  • 6
elbarna
  • 12,050
  • 22
  • 92
  • 170