4

Linux has shred, from the GNU coreutils package, to securely overwrite data in-place when removing files. What is the equivalent on BSD systems (and specifically on macOS)?

Kusalananda
  • 320,670
  • 36
  • 633
  • 936
CarriMegrabyan
  • 43
  • 4
  • Do you have access to Homebrew and can install GNU coreutils through `brew`? (... or via any similar package-manager on macOS) – Kusalananda Jul 15 '22 at 15:10
  • 2
    shred is a as much snake oil on Linux as it is on BSD. – Marcus Müller Jul 15 '22 at 15:57
  • 1
    @MarcusMüller Still, it _is_ the equivalent on all platforms. – Kusalananda Jul 15 '22 at 16:01
  • @Kusalananda true, let me put as much in an answer. – Marcus Müller Jul 15 '22 at 16:02
  • If you want to be sure, try booting the shredOS distro. **Downside** it will remove all your files, not just the one you want to be gone. This is the equivalent of nuking from orbit. – Criggie Jul 16 '22 at 01:01
  • @MarcusMüller Calling it snake oil seems like a big stretch. It's just that its utility is limited to HDDs using non-CoW filesystems like ext4. HDDs are still common, and I believe ext4 is still the most common FS for linux installations. – JoL Jul 16 '22 at 06:15
  • @JoL nah, it is a tool without an underlying threat model for which security problem it actually solves. The "overwrite with multiple runs of random data" [is actually without technical merit on modern disks](https://security.stackexchange.com/questions/10464/why-is-writing-zeros-or-random-data-over-a-hard-drive-multiple-times-better-th/10474#10474), so compared to a simple dd from /dev/zero, shred's value proposition is really snake oil! And:I don't think ext4 without any volume mapper below is that common for new installations of user-friendly distros any more, but might be mistaken. – Marcus Müller Jul 16 '22 at 07:46
  • 1
    @MarcusMüller, though `shred` can be used for just wiping with zeroes, and IIRC it can print stats on the progress, which is useful for wiping full disks, and makes it strictly better (more featureful) than a plain `cat /dev/zero > ...`. Having a volume mapper like LVM underneath shouldn't matter that much, since (I expect) it's very common that there are minimal changes to the mappings during the lifetime of your regular desktop install. Or at least that one seldom _moves_ existing mappings or resizes fs's to be smaller. [...] – ilkkachu Jul 16 '22 at 08:58
  • 1
    [...] Merely extending an fs shouldn't leave copies of data around, AFAIU. Btrfs and ZFS would be different, though. And then there's the thing that even if repeated overwrites aren't _technically_ necessary, they might be _politically_ required. :) – ilkkachu Jul 16 '22 at 09:00
  • @ilkkachu but... you're making my points for me :) yeah, it might be *politcally required*, because as you know, snake oil cures the common cold in 3 days (or more, sometimes less)! stat printing: undubitatbly useful, but then, we're in the realm of hard drives, so we can rely on SATA encryption and just wiping the encryption key would be more effective and fast enough that you don't need a progress bar ;) But yeah, I see where that is coming from. – Marcus Müller Jul 16 '22 at 10:18
  • @MarcusMüller, how well I could trust drives themselves doing encryption is another thing... And when you have that drive that _wasn't_ encrypted, you're back to wiping the data again, though there, using the drive's security erase function might be better. In any case, "political" requirements can be very real, too,`shred` doesn't actually cost anything, and it's at least as good as wiping with zeroes. Hardly snake oil. – ilkkachu Jul 16 '22 at 10:32
  • @ilkkachu I think we'll have to agree to disagree here! Political, what I'd call feel-good-requirements of specific tool usage, are pretty much my definition of snake oil. I fully understand how you can be in corporate of even regulatory bonds that force you to apply a tool, doesn't make it any less snake-oily :) But as said, I still agree, a nice tool to get information why zeroing out stuff. – Marcus Müller Jul 16 '22 at 10:47
  • 1
    @MarcusMüller, yeah, I think I associate _cost_ as a somewhat necessary part of snake oil, as in someone selling useless crap for high prices based on fraudulent claims (while pretty much knowing it's a fraud). I don't see that in `shred`, since it's not sold for money, but is free to use, and it's not the authors of it promoting it as an do-all wonder tool (I think?), but more like the dozens of web sites which might just be dated or ignorant. Not as useful as some might claim, yeah; cargo cult, maybe; snake oil, not _that_ much in the way I understand it. :) – ilkkachu Jul 16 '22 at 11:03
  • 1
    ah I see! yeah, that definition makes sense. I have more of a "distracts from the actual solution to the problem" aspect in mind, but yeah, you're right, the origin of that word is probably more travelling salesmen **selling*** wonder cures! – Marcus Müller Jul 16 '22 at 11:11
  • didnt know ext4 had journaling. @ilkkachu, thanks for point out that `cat /dev/zero >> filename` should work. I have to also unlink the file with `rm` after that right? – CarriMegrabyan Jul 18 '22 at 07:00
  • @CarriMegrabyan, actually, catting from `/dev/zero` isn't that good as a real tool. It won't know how much to write, and with `> filename` it'd first truncate the file, releasing the allocated blocks. It might not reserve the same blocks again, and if rewriting like that is going to work at all, it depends on rewriting the same blocks. `>> filename` would append, not overwrite... Something like `head -c $size_of_file /dev/zero 1<> $filename` might work on mac. – ilkkachu Jul 18 '22 at 07:26
  • your command did erase the file. what exactly is `1<>` doing here? – CarriMegrabyan Jul 29 '22 at 05:13
  • never mind, found it out. while the file contents of the file appear to be zeroed out, the file itself still is present. The file still needs to be removed via `rm`. is there a way to verify that there are no copies of the file on disk? – CarriMegrabyan Jul 29 '22 at 05:26

3 Answers3

7

The answer that qorg11 gave is the technically accurate answer to your question (so go and upvote it!).

The reality is that shred is useless. All BSDs, as well as Linux, have modern file systems, where "overwrite a file to actually overwrite the bytes on the storage medium" simply does not happen because it's not how the structure of a file system looks like and how the storage device works.

What shred does is take a file and overwrite its contents with zeros. Great, but many modern filesystems are "copy on write" (i.e., when you change something in a file, you don't modify the same blocks on the storage device, you make a modified copy of the affected block). Apple's APFS is definitely one of these file systems.

Some file systems have "sparsity detection": you write all-zeros, so the file system doesn't write these zeros to disk but just denotes a hole in the file.

All SSDs have write wear balancing: you're really not protecting against an attacker that can access the raw data as stored on the memory chips because all writes to the system are essentially copy-on-writes to evenly distribute the load of writing data across the medium (writing the same cell often leads to ageing and increased error probability; since an SSD has uniform access latency, wildly distributing logical blocks across the physical medium has no downside). On the flip side, modern file system drivers support the TRIM operation (often called "discard"), which signals to the SSD that the data can be zeroed, and the SSD will then quite likely not even read the zeros if being asked for that logical block.

So, as qorg11 says, shred is the equivalent of shred, even on non-GNU systems, if you install GNU coreutils, but an rm works just as well, in reality.

Kusalananda
  • 320,670
  • 36
  • 633
  • 936
Marcus Müller
  • 21,602
  • 2
  • 39
  • 54
  • 3
    Umm, shred does not simply override a file’s content with zeros. It overwrites with random data, and then optionally may overwrite with zeros in a final pass so that you look clean. This relies on overwrites with random data being performed in place and not being optimized away. Of course, that’s not necessarily true - a filesystem might take advantage of any overwrites to optimize block placement - but it’s true on many file systems. Similarly, the file system might squash all but the last block write, even if flushed or synced. – Krazy Glew Jul 16 '22 at 03:30
  • Of course shred may not handle block remapping in flash, or even on regular discos as a result of errors or funky disco vendor optimizations. // ultimately, there must be primitives that cross all of the levels of abstraction to do safe erase. And at the moment that is easiest if completely erasing a hardware device. – Krazy Glew Jul 16 '22 at 03:33
  • 7
    `All BSDs, as well as Linux, have modern file systems, where "overwrite a file to actually overwrite the bytes on the storage medium" simply does not happen` that's not true at all. The most common Linux filesystem nowadays is still ext4 which is journaled. Btrfs, ZFS and F2FS still didn't quite get mainstreamed – phuclv Jul 16 '22 at 03:50
  • Fedora installations have defaulted to btrfs for quite a while, haven't seen an Android phone with an ext4 data partition in a while. But you're right, ext4 is not COW; and there was even a proposed safe delete patch (which essentially overwrites after delete) for ext4 that never made it into the kernel - [because filesystem maintainers deemed it snake oil to have a journal and then claim you made deletion securer by overwriting the data extents](https://lwn.net/Articles/462437/). So in the case of ext4, it's clearly the journaling that drives shred ad absurdum, not COW. – Marcus Müller Jul 16 '22 at 07:33
  • you are saying `rm` is equivalent to `shred`, and its impossible to completely delete a file on modern file systems like ext4 and APFS? without zeroing out the entire file system? – CarriMegrabyan Jul 18 '22 at 07:08
  • 1
    Ext4 is not a copy-in-write system, only a journaling file system, so that the changes made to a file appear in a log, which might be rolled back; that's not as bad as having a copy, because journals get purged regularly automatically. However, multiple rounds of overwrite are useless. On apfs, yes, that's copy-on-write and you never overwrite the original data when you change a file. No, you don't have to zero your whole drive, you could also use the TRIM operation to tell the drive to discard the data from unused blocks, if an ssd. – Marcus Müller Jul 18 '22 at 07:13
6

You can install the GNU coreutils with brew install coreutils in macOS (using Homebrew), pkg install coreutils in FreeBSD, or pkg_add coreutils in OpenBSD and then run gshred, which will behave exactly like shred in Linux.

Kusalananda
  • 320,670
  • 36
  • 633
  • 936
montepinar
  • 86
  • 4
3

rm itself has the -P flag.

OS X man page documents it as:

-P   Overwrite regular files before deleting them.  Files are
     overwritten three times, first with the byte pattern 0xff,
     then 0x00, and then 0xff again, before they are deleted.

OpenBSD man page:

-P    Attempt to overwrite regular writable files before deleting them. Files 
      are overwritten once with a random pattern. Files with multiple links will 
      be unlinked but not overwritten.`

On FreeBSD however it does nothing, but the flag was kept for compatability.

The BUGS section of the OpenBSD man page warns about its limitations: http://man.openbsd.org/rm#BUGS

The -P option assumes that both the underlying file system and storage medium write in place. This is true for the FFS and MS-DOS file systems and magnetic hard disks, but not true for most flash storage.

These also applies to shred.

carbin
  • 246
  • 2
  • 5