Can losetup be made efficient with sparse files?

Question

So my setup is like this.

$ truncate -s 1T volume
$ losetup -f --show volume
/dev/loop0
$ mkfs.ext4 /dev/loop0
$ ls -sh volume
1.1G volume
$ mount /dev/loop0 /mnt/loop

Now I have a 1.1TB volume, as expected. The overhead of ext4 expanded the sparse file to 1.1G, but that's fine. Now to add a file.

$ dd if=/dev/urandom of=/mnt/loop/file bs=1M count=10240
$ ls -sh volume
12G volume

Cool, now I don't want the file.

$ rm /mnt/loop/file
$ ls -sh volume
12G volume

The free space is still taking up space, as expected, and $ fallocate -d volume frees up 1gb.

My question is, how can I zero out the free space here without expanding the volume to the full size? $ dd if=/dev/zero will expand it to full size, and with conv=sparse makes it create a useless sparse file inside the volume.

TL;DR: Is there a way to make losetup ignore writes of null blocks to null sectors, while allowing everything else?

Hmm... when you remove the file, the random data is still present on your disk. This makes you possible to recover the deleted data when you dig into the a disk. If you _secure erase_ the file by zeroing out all the bits, you can use `fallocate -a` to free up the spaces. If you did not _secure erase_ the file while it was still here, it would be painful to _secure erase_ the entire disk afterwards, either by `fstrim` or by `dd sparse` the entire (loop) disk. — midnite, Mar 09 '22 at 19:42

sourcejedi · Accepted Answer · 2018-08-23T08:06:42.263

To automatically discard data blocks when they are no longer used, use mount -o discard .... Or you can manually run fstrim.

This feature was apparently added to the loop device in Linux 3.2. https://outflux.net/blog/archives/2012/02/15/discard-hole-punching-and-trim/

In the general case, mount -o discard is not guaranteed to be effective, because some types of device are allowed to ignore discard requests when they are busy. This would not be a concern for the size of your sparse file though.

In this general case - e.g. if you also wanted to send discard requests to an underlying physical device - the most robust method is to run fstrim at regular intervals.

score 0 · Answer 2 · answered Aug 23 '18 at 04:50

Not a solution using built in tools, but I whipped up a python script to do what I want. Here it is if it helps anyone. Still looking for a built in solution, if there is one.

#!/usr/bin/python3.6
import os
import time
import sys

if len(sys.argv) != 2:
    print('''Usage: {} <file>

Fills a file/device with zeros, with efficiencies for sparse files.'''.format(sys.argv[0]))
    exit(1)

f = open(sys.argv[1], 'rb+', 0)

bs = 1024*1024 # block size
zb = b'\0'*bs  # a full block of zeros

lasttime = round(time.time())
block = f.read(bs)
while len(block) == bs:

    # IFF a block contains non-zero characters, zero it
    if block != zb:
        f.seek(-bs, os.SEEK_CUR)
        f.write(zb)
    block = f.read(bs)

    # Print the processed bytes every second
    # Could add a MB/s or % completed, but too lazy
    if round(time.time()) != lasttime:
        print(f.tell())
        lasttime = round(time.time())

# Hit EOF, might have missed bytes at the end
# Zero last block unconditionally, too lazy to check if its required
f.seek(-bs, os.SEEK_CUR)
f.write(zb)
f.close()

Can losetup be made efficient with sparse files?

2 Answers2