How to atomically defragment ext4 directories

Question

Fragmentation seems to create a lot of unnecessary seeks when traversing a directory tree on a HDD:

# stat -c %F 00 01 02
directory
directory
directory

# filefrag -v 00 01 02
Filesystem type is: ef53
File size of 00 is 12288 (3 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       0:  428351942.. 428351942:      1:            
   1:        1..       2:  428352760.. 428352761:      2:  428351943: last,eof
00: 2 extents found
File size of 01 is 12288 (3 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       0:  428351771.. 428351771:      1:            
   1:        1..       2:  428891667.. 428891668:      2:  428351772: last,eof
01: 2 extents found
File size of 02 is 12288 (3 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       0:  428351795.. 428351795:      1:            
   1:        1..       2:  428352705.. 428352706:      2:  428351796: last,eof
02: 2 extents found

e4defrag isn't able to defrag them

# e4defrag -v 00
ext4 defragmentation for directory(00)
[1/116] "00"
    File is not regular file        [ NG ]

So how do I defragment a directory? Not its contents, but the directory itself. The directories are in use, so it should be done atomically, just like defragmenting regular files does not interfere with their use.

@roaima The title is my question: [How to] atomically defragment ext4 directory files? — the8472, Mar 12 '17 at 05:21
@roaima 1. I said "directory file", referring to the directory object itself ([everything is a file](https://en.wikipedia.org/wiki/Everything_is_a_file), just like block devices are files!) 2. atomically is a word. but "online defragmentation" might work too in this context. 3. I already edited my question. — the8472, Mar 12 '17 at 09:19
"Your example is of a directory." - I *intentionally* highlighted that with the `stat` command. So I am quite aware of that. — the8472, Mar 12 '17 at 09:19
You don't, at least not online. Offline, `e2fsck` has an option to defrag directories, and then there is the old `e2defrag`. Of course, having only two fragments isn't going to affect performance in any meaningful way. — psusi, Mar 17 '17 at 13:25
@psusi running `fsck -D` asked to optimize some extent trees (*Inode 41947219 extent tree (at level 1) could be shorter. Fix? yes*), but it didn't rewrite the directories in question to a single contiguous range. — the8472, Mar 19 '17 at 21:35

score 2 · Accepted Answer · answered May 07 '17 at 20:47

Since there does not seem to be any online defragmentation tool for directory indices and even the offline defragmenters don't seem to help I had to resort to rebuilding the directory tree recursively.

I've written a small tool (defrag-dirs) for that purpose. Alas, that approach requires the application using the directory tree to be taken down during defragmentation, which can take a considerable amount of time when dealing with millions of files.

score 0 · Answer 2 · answered May 05 '23 at 22:10

If you have the free space, you could create a parallel tree of new directories on the same device and hard link the files to them, then swap names of the head directory, and later after a reboot or other app shutdown, drop the original directory tree. We used to make such clone trees with minimal file space needs to test changes, or to install sw releases beside running production.

You might build each directory directories first (speeding tree traversal), and then oldest files first, as the youngest files are most likely to be ephemeral.

I am not sure how this works on file types other than flat files and directories. I expect symbolic links can themselves be hard linked, and probably named pipes, but devices? And of course I am pretty sure the fake trees in Linux like /prod and /dev, reflecting kernel tables, are off limits.

How to atomically defragment ext4 directories

2 Answers2