33

Say you want to zero-out a failing hard disk. You want to overwrite as much as possible with zeros. What you don't want is: the process aborts on the first write-error. How to do that?

AFAICS, plain dd only provides an option for ignoring read errors. Thus, something like

dd if=/dev/zero of=/dev/disk/by-id/lousy-vendor-123 bs=128k

is not enough.

ddrescue seems to be better at ignoring errors - but what would be the optimal command line with it?

My try with GNU ddrescue:

ddrescue --verbose --force --no-split /dev/zero /dev/disk/by-id/lousy-vendor-123
maxschlepzig
  • 56,316
  • 50
  • 205
  • 279
  • 1
    `dd conv=noerror` might be a GNU extension, I'm not sure. In any case, it should do the trick. The SATA tell-the-drive-to-erase-itself answer is worth looking into for erasing entire drives, though. – Peter Cordes Sep 13 '15 at 12:42
  • 1
    @PeterCordes, the man page of GNU `dd` documents `noerror` as 'continue after read errors' ... – maxschlepzig Sep 13 '15 at 12:55
  • 1
    `dd conv=noerror` ist POSIX standard but it may be really slow – schily Sep 13 '15 at 13:04
  • 2
    `dd conv=noerror` must be for reading errors (according to the manpage), not writing errors. There is nothing bad in combining it with `conv=notrunc`, which did the trick with ignoring the write errors for me. http://unix.stackexchange.com/a/229379/4319 – imz -- Ivan Zakharyaschev Sep 13 '15 at 13:11
  • A simple write operation to a HDD, such as using `dd if=/dev/zero of=/dev/sdX`, would not perform any disk reads. Why do you think the **dd** command *"is not enough"*? What "errors" are you experiencing? – sawdust Aug 13 '19 at 08:17
  • @sawdust, I don't understand your question. The standard dd is not enough because it aborts on the first write error. – maxschlepzig Aug 14 '19 at 20:39
  • Sorry I read your post too fast. A error during write implies that the controller cannot locate the proper sector to perform the operation. The only workaround for such an error is to not perform the write for that sector. The dd command would not have such an option. You could restart the dd command with the `seek=N` option, where N in this case would be the block number that failed *plus 1* – sawdust Aug 14 '19 at 21:28
  • @sawdust Your two assertions (' ... implies that ...', 'The only workaround ...') are simply false. Take a look at ddrescue (which is mentioned in my question) and check out the 2 top voted answer. – maxschlepzig Aug 14 '19 at 21:37
  • The assertions are not false. Those other programs are doing the same avoidance scheme I described for dd. I've written firmware for disk controllers. There are not many sector-specific sources for errors during a write_sector operation. There is no workaround for an error during write. If a retry also fails, then that's a bad sector that needs to be avoided. – sawdust Aug 14 '19 at 21:59

5 Answers5

20

I prefer badblocks in destructive write mode for this. It writes, it continues doing so when it hits errors, and finally it tells you where those errors were, and this information may help you decide what to do next (Will It Blend?).

# badblocks -v -b 4096 -t random -o badblocks.txt -w /dev/destroyme
Checking for bad blocks in read-write mode
From block 0 to 2097151
Testing with random pattern: done
Reading and comparing: done
Pass completed, 52105 bad blocks found. (0/52105/0 errors)

And the block list:

# head badblocks.txt
2097000
2097001
2097002
2097003
2097004

And what's left on the disk afterwards:

# hexdump -C /dev/destroyme
00000000  be e9 2e a5 87 1d 9e 61  e5 3c 98 7e b6 96 c6 ed  |.......a.<.~....|
00000010  2c fe db 06 bf 10 d0 c3  52 52 b8 a1 55 62 6c 13  |,.......RR..Ubl.|
00000020  4b 9a b8 d3 b7 57 34 9c  93 cc 1a 49 62 e0 36 8e  |K....W4....Ib.6.|

Note it's not really random data - the pattern is repetitive, so if you skipped 1MiB you'd see the same output again.

It will also try to verify by reading the data back in, so if you have a disk that claims to be writing successfully but returns wrong data on readback, it will find those errors too. (Make sure no other processes write to the disk while badblocks is running to avoid false positives.)

Of course with a badly broken disk this may take too long: there is no code that would make it skip over defective areas entirely. The only way you could achieve that with badblocks would be using a much larger blocksize.

I'm not sure if ddrescue does this any better; it's supposed to do that in the other direction (recover as much data as fast as possible). You can do it manually for dd/ddrescue/badblocks by specifying first/last block...

frostschutz
  • 47,228
  • 5
  • 112
  • 159
  • 2
    If I use `-t random` or `-t 0` - does badblocks then just do one write pass? Looking at the man page - it seems that without `-t` it does 4 passes (for '0xaa, 0x55, 0xff, 0x00'). – maxschlepzig Sep 13 '15 at 11:06
  • 2
    It does one pass for each `-t` you provide on the command line. The default is 4 passes as you say. – frostschutz Sep 13 '15 at 11:31
  • Writes per block = patterns (`-t`) x passes (`-p`). `-p` defines the number of passes which is by default 0 i.e. exit after first pass. `-t` is different; it specifies the pattern to write to each _block_; each `-t` adds one more pattern to the list of patterns for each block. When `-w` is given, this starts with 4 patterns: `0xaa`, `0x55`, `0xff` and `0x0`. – legends2k Oct 16 '20 at 17:23
  • 1
    @legends2k sure, badblocks uses the term "pass" in its very own way, but that doesn't change it does a full write read compare pass for each pattern, which by default there are four (in destructive write mode `-w`) unless you specify your own `-t`. – frostschutz Oct 16 '20 at 17:28
  • You're right. Looks like `badblocks` has its own defn. for the term _pass_. Every pattern seems to be like a pass: writes that pattern to all blocks, then reads that pattern from all blocks, proceeds to further patterns sequentially. This feels like a pass (in the general sense), but there's another parameter `-p` which is another loop enclosing the patterns loop. – legends2k Oct 25 '20 at 03:37
  • Using gnome-disk-utility v41.0 (`gnome-disks`) to wipe an SD card failed repeatedly due to a write error. Running `badblocks` in write mode, as suggested in this answer, reported 0 bad blocks. Interestingly, after that run, gnome-disk-utility succeeded to wipe the entire SD card. Not enough data points to prove anything (perhaps just a flaky SD card), but it did help fix my problem. – Joel Purra Nov 23 '21 at 12:01
15

If the disk is not connected by USB, then consider using hdparm (version > 9.31) to carry out an ATA Secure Erase of the disk. This command causes the drive's firmware to wipe the contents of the disk, including bad blocks.

Warning: Use the correct drive letter - I've shown /dev/sdX as an example - don't just copy/paste.

First, check that it understands the ATA commands (most drives manufactured this last decade or more should):

$ sudo hdparm -I /dev/sdX
.
# lots of other info here...
.
Security: 
    Master password revision code = 65534
        supported
    not enabled
    not locked
    not frozen
    not expired: security count
        supported: enhanced erase
    202min for SECURITY ERASE UNIT. 202min for ENHANCED SECURITY ERASE UNIT.

The last two lines of the extract shows that it is supported.

Therefore add a password to the drive (a requirement apparently):

$sudo hdparm --user-master u --security-set-pass p /dev/sdX
security_password="p"

and erase:

$sudo hdparm --user-master u --security-erase p /dev/sdX
security_password="p"

/dev/sdX:
Issuing SECURITY_ERASE command, password="p", user=user

More info on this procedure is available here.

garethTheRed
  • 33,289
  • 4
  • 92
  • 101
  • 1
    This can work over USB if you're lucky, and your USB <-> SATA bridge can pass through non-standard SATA commands (and the Linux driver + hdparm know how to do so on that model). Also, for writing examples, `/dev/sdX` is good, because then if someone misses an occurrence of it when pasting and customizing, there won't be a problem. – Peter Cordes Sep 13 '15 at 12:12
  • @Peter Cordes - Unless you have 24 disks... No, silly me! Thanks, I've changed it to sdX, that'll teach me to rush an answer! – garethTheRed Sep 13 '15 at 12:22
  • 2
    The linked page lists so many hardware/firmware related caveats with this method ... kind of scary – maxschlepzig Sep 13 '15 at 12:58
  • It does indeed show quite a few caveats! All I can say is that it worked for me without any trouble. – garethTheRed Sep 13 '15 at 13:17
3

I see four workable answers here:

  1. The hdparm method posted by garethTheRed is probably best if you are connected directly to your computer. Apparently, though, if you try it connected via USB, you can brick your drive. If you are doing this for a drive you are about to dispose of, then that may be a good thing. However, you probably want to secure erase before discarding.

  2. The technique reported by imz -- Ivan Zakharyaschev will work, but may be very slow. I would suggest if you do not want the data to be recoverable, use /dev/urandom instead of /dev/zero; e.g.,

    dd iflag=fullblock oflag=direct conv=noerror,notrunc if=/dev/urandom of=/dev/sdX

  3. I would advice against the following. For something faster that does the same thing, use the technique reported by maxschlepzig (in the question):

    ddrescue --verbose --force --nosplit /dev/urandom /dev/sdX

    This will be faster than the dd command, but not as fast as the hdparm command. See below why I don't recommend this...

  4. The badblocks command will also work, but you can't randomize the data that way, and again it will be very slow.

Finally, I would be remiss if I did not point out the number one reason people want to completely erase a disk is they are about to dispose of it. In that case, if you haven't already, you might want to try and recover the disk first. If you read a block and it returns the I/O error, then next time you write to the same block the disk will try to reallocate a different block from a reserve list. Once the reserve list is full then you will get I/O errors on writes. That is when you really should discard the drive.

So you can do something simple like:

dd if=/dev/sdX of=/dev/null conv=noerror

And, then, to rewrite the bad blocks, just something like:

dd if=/dev/zero of=/dev/sdX bs=128k

If this command works, if you are brave, you can reformat your disk and use it again.

Alternatively, you can run the badblocks command on the disk twice. The second time it should report no bad blocks...

badblocks -v -s -w -t random /dev/sdX
badblocks -v -s -w -t random /dev/sdX

This will take longer, but is more reliable.

It is also worth noting that none of the techniques really do a secure erase, except the hdparm command. Remember all those bad blocks? Those still have some of your original data mostly intact. A data recovery expert could access these to see a small amount of what was previously on your hard drive.

In regards to ddrescue and why I advice against it, I have the following antidote:

The problem is ddrescure will be TOO good at ignoring errors. I had a hard drive that consistently with dd dropped write speed at about the 102 GB mark and started producing a write errors at the 238 GB mark. I was quite impressed that ddrescue continued to churn through the disk at a constant speed, even reporting no errors. 17 hours later, when it was at the 1300 GB in when I happened to notice the drive light itself stopped flashing. A quick check revealed the whole USB enclosure had gone offline. I pulled the drive out of the cradle. I noticed ddrescue just happily reported it was still copying with no errors, even with the disk in my hands. I plugged the disk into another machine and found it was now a brick.

I don't blame ddrescue for making the drive a brick. The drive was failing and would become a brick. I just find disturbing ddrescue doesn't even give an error count of how many write errors it is ignoring. In this usage, ddrescue leaves you think it has been completely successful, regardless of all write failures. The fact is it should not have been able to continue at full speed in the section with the slow down. The reason that section was slow, is many blocks had been relocated by the drive, causing lots of seeking when accessing that section. So that is probably the point when ddrescue's output became fictional.

agc
  • 7,045
  • 3
  • 23
  • 53
user6856
  • 171
  • 3
  • 1
    'To securely erase you need to overwrite the same block multiple times with random data.' - can you substantiate your claim with a reference (i.e. peer reviewed article that shows recovery of zeroed data on modern hard disks)? Also, the harddisk firmware is a black box. How do you know that the ATA secure delete command isn't just implemented as write-zero-pattern-to-each-sector? – maxschlepzig Sep 20 '15 at 06:07
  • 1
    Just the reverse. It looks like in 2006 there were a number of studies that showed old school multiple writes are no longer required with modern drives, as modern drives store the data so dense that a single write is just as effective. – user6856 Sep 20 '15 at 11:15
1
dd conv=notrunc

probably did the trick for me.

The mentioned

dd conv=noerror

must be for reading errors (according to the manpage). There is nothing bad in combining the two.

My complete command for zeroing a disk looked like:

dd iflag=fullblock oflag=direct conv=noerror,notrunc if=/dev/zero of=/dev/TARGET_DISK

Adding a custom bs= may also be wanted for some case.

d33tah
  • 1,361
  • 14
  • 28
imz -- Ivan Zakharyaschev
  • 15,113
  • 15
  • 61
  • 123
1

A fast and mature method is to use sdd.

If you just like to destroy all content, call:

sdd -inull bs=1m of=/dev/rdsk/cXdXtXp0 -noerror

Always use the "raw" disk driver interface.

If you like to repair a disk and keep as much of the old content as possible, call:

sdd if=/dev/rdsk/cXdXtXp0 of=/dev/rdsk/cXdXtXp0 bs=1m -noerror

This will replace all unreadable blocks with zeroes at 512 byte level. You may like to modify the retry count via try=#, the default is 2.

Note that sdd is faster than dd in case of errors as it first tries to read with the supplied blocksize and in case of errors, it reads with 512 bytes. If there are read errors, sdd makes random seeks and dummy reads to calm down the firmware of the drive.

The enhanced error recovery features have been developed in the 1980s while I was working for the second biggest Sun-Microsystems OEM.

Sdd sourcecode is included in the schily tools:

http://sourceforge.net/projects/schilytools/files/

schily
  • 18,806
  • 5
  • 38
  • 60