4

I need to scrub a couple of very large HDDs. However, I can't do that from a desktop. I need to do it on the move, from a laptop. A single pass of scrub(1) on a HDD would take more than a day, but there is no way I can leave my laptop stationary for that long.

scrub(1) itself doesn't support any kind of offset command line parameter.

Is there a way to do what scrub(1) does (writing random bytes), but in a way that can be resumed? Basically the command would need to print out the offset when I interrupt it, and it needs to accept an offset parameter for resuming.

Kal
  • 743
  • 1
  • 7
  • 16
  • 2
    Is SATA secure erase not an optrion? – Jasen Oct 10 '20 at 08:30
  • @Jasen Secure erase would probably take as long, and the drive would still need power during that time. And some people still prefer their cargo cult solutions. – Michael Hampton Oct 11 '20 at 01:05
  • @MichaelHampton: Yes, but with an external HDD with its own power supply, it might keep erasing itself if you unplug the USB cable from the laptop, leaving the HDD just plugged into the wall. Or into a powered USB hub if you have a 2.5" bus-powered external HDD. Although apparently not all USB<->SATA adapters support the commands properly: [Repair an SSD that's been unplugged during Secure Erase](https://superuser.com/q/1172879). (Maybe including ones built-in to external HDDs) – Peter Cordes Oct 11 '20 at 01:11
  • secure erase can complete in as little as 20 seconds depending on the drive, which was why I suggested it. – Jasen Oct 11 '20 at 03:33
  • @Jasen I *believe* the 20 second thing only works for SSD drives where the data does not actually need to be overwritten. The cells are simply marked as empty and there is supposed to be no way for anybody to bypass the controller to retrieve the data. – Kal Oct 11 '20 at 04:46
  • 2
    no it only work on drives where the data is stored encrypted, and the key can be rapidly discarded, this includes some spinning types, you can check using `hdparm -I`. eg HITACHI HTS722020K9SA00 supports "2min for ENHANCED SECURITY ERASE UNIT" the unit of measure is 2 minutes, so a reported "2 minutes" usually means instant. OTOH SAMSUNG MZ7KM480HMHQ-00005 only does a slow secure erase (and it's an SSD) – Jasen Oct 11 '20 at 05:06

2 Answers2

8

dd can be coerced into producing a progress report (by signalling it with SIGUSR1) and can be told to start writing part way through (using seek)

You then just need a source of random bytes, such as /dev/urandom

Jasen
  • 3,715
  • 13
  • 14
  • 2
    You can also use `status=progress` on GNU – Oskar Skog Oct 10 '20 at 08:39
  • `/dev/urandom` will run out of entropy pretty quickly. – d3jones Oct 10 '20 at 16:45
  • The entropy calculation of the values provided by `/dev/urandom` will drop to 0, @d3jones but the content will still be "random" (based on previous initialization) and, being urandom, it won't block either. Thus it should be fine. – Ángel Oct 11 '20 at 00:09
  • @Ángel: You might want a *faster* CSPRNG than `/dev/urandom`, though. It tries to be very high quality so it's quite slow to read. Although still good enough to keep up hard drives. My 3.9GHz Skylake (i7-6700k) can `dd if=/dev/urandom of=/dev/null bs=64k` at ~235MB/s. (Arch Linux, kernel 5.7.6-arch1-1). Probably getting some entropy from `rdrand` so that helps. – Peter Cordes Oct 11 '20 at 01:10
  • depends on kernel version. urandom used to limit around ~10M/s and is the cause for all the "urandom is slow" comments, but the algorithm changed so you get >200M/s now on the same hardware. it's now fast enough for wiping entire hard drives. it's the same with my cryptsetup approach really - if you have AESNI it should be fast enough. without hardware accelerated encryption it will be kind of slow (faster than old urandom, probably slower than new urandom, I have not measured it) – frostschutz Oct 11 '20 at 10:21
  • 1
    for fast PRNG (single pass scrub, shred, ...) I guess you can just partition the disk (GPT allows 128 partitions by default) and do one partition at a time – frostschutz Oct 11 '20 at 10:23
  • @frostschutz: all of a ~4GHz CPU to achieve just 235MB/s is very slow by CPU standards. Yes, on that fast CPU it's enough to keep up with even a fast rotational HDD; on a laptop with a slower CPU that can't turbo forever it could fall behind for fast areas of the drive. When I said faster, I meant something closer to memset speed (tens of GB/s), and doesn't even break a sweat (or heat up your CPU, or slow down gaming) to run at 200 to 500 GB/s. e.g. https://espadrine.github.io/blog/posts/shishua-the-fastest-prng-in-the-world.html#benchmark has some benchmarks. ChaCha8 is a CSPRNG. – Peter Cordes Oct 11 '20 at 13:41
  • @frostschutz: `openssl speed aes` shows I'm getting ~280 MB/s from aes-128 cbc (3.9GHz Skylake, Arch Linux). Apparently not using AES-NI, according to `perf record` / `perf report` and looking at the asm instructions using CPU time; no wonder it's so slow. Good idea to encrypt zeros, although you could probably do that via `openssl ... < /dev/zero | dd if=-` with a non-crappy build of `openssl`. If we had a `seek` command to lseek stdin, you could do `(seek position; openssl < /dev/zero) > /dev/sdXX` to avoid any extra copying of data through a pipe into another process. – Peter Cordes Oct 11 '20 at 14:01
  • 1
    @PeterCordes, something like `(pos=1234 perl -e 'sysseek STDOUT, $ENV{pos}, 0 or die' && openssl ...) > /dev/sdx` maybe – ilkkachu Oct 11 '20 at 14:32
  • (update on openssl performance: `openssl aes-128-cbc` (not "speed") *is* using AES-NI, and getting more like 790MB/s, reading from /dev/zero and piping into `pv > /dev/null`. IDK why `openssl speed` benchmarks the fallback scalar code.) – Peter Cordes Oct 11 '20 at 14:56
  • @ilkkachu: Yup, that would do it. And to find out the current position without `dd`'s SIGUSR1 output, `cat /proc//fdinfo/0` to get the file position on modern Linux. – Peter Cordes Oct 11 '20 at 14:57
5

You could use ddrescue and cryptsetup:

truncate -s $(blockdev --getsize64 /dev/sdx) sdx-zero
cryptsetup open --type plain --cipher aes-xts-plain64 sdx-zero sdx-random
ddrescue /dev/mapper/sdx-random /dev/sdx sdx-scrub.map

To resume, preserve sdx-scrub.map and then just repeat the same commands again.

If you use the same passphrase every time, this method also allows verification:

cmp /dev/mapper/sdx-random /dev/sdx && echo OK || echo FAIL

However, to make this resumable, you'd have to use cmp -i SKIP -n LIMIT.

With verification, the process will take twice as long. Without verification, you have a Schroedinger's scrub.

In the above example, /dev/sdx is the drive to be scrubbed.

sdx-zero is a sparse file containing only zeroes, same size as /dev/sdx. It must be backed by a filesystem that supports sparse files properly, ext4/xfs/btrfs works, tmpfs/fat/ntfs does not.

cryptsetup encrypts zeroes to random data, so /dev/mapper/sdx-random is a seekable block device full of random data (unlike /dev/urandom which is not seekable).

ddrescue reads random data from sdx-random and writes it to /dev/sdx, thereby scrubbing it while tracking progress in sdx-scrub.map. It will also show you a progress bar as well as errors if any.

If you prefer not typing a passphrase every time, you can also create a keyfile instead:

printf "%s" $(uuidgen) > sdx-scrub.key

Generate the keyfile only once and preserve it between calls, then just add --key-file sdx-scrub.key to the cryptsetup command.

See also https://unix.stackexchange.com/a/352378/30851

frostschutz
  • 47,228
  • 5
  • 112
  • 159
  • This is an amazingly cool answer; I never knew cryptosetup could use a sparse file. I suppose if you try to *write* to `/dev/mapper/sdx-random` it'll start filling up actual space on disk in `sdx-zero`? Thank you! –  Oct 14 '20 at 02:13