6

How do I delete this large directory?

stat session/
  File: ‘session/’
  Size: 321540096       Blocks: 628040     IO Block: 4096   directory
Device: 903h/2307d      Inode: 11149319    Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-09-29 14:34:40.910894275 +0200
Modify: 2022-09-29 14:35:09.598400050 +0200
Change: 2022-09-29 14:35:09.598400050 +0200
 Birth: -

Note that the size of directory (not the content, but the directory entry itself) is over 300MB. Number of inodes is over 11 million.

The directory has no subdirectories, only large number of files.

None of the usual commands work. I have tried these:

  • rsync -a --delete empty_dir/ session/
  • rm -rf session
  • find . -type f --delete

If I run ls -f1 inside, it hangs.

If I run mv -- * ../.tmp_to_delete inside, it hangs.

If I run du inside, it hangs.

At the moment the rsync --delete is running since two days, reading at the rate of up to 7MB/s, and I see no change in the stat output for the directory.

I assume the large size of the directory is the problem.

Chris Cooper
  • 103
  • 1
Bojan Hrnkas
  • 201
  • 1
  • 6
  • 2
    Is unmounting the FS an option? If not there's probably little improvement you can do over `rm -rf`. – Stéphane Chazelas Sep 29 '22 at 13:09
  • 1
    Dup of [Efficiently delete large directory containing thousands of files](//unix.stackexchange.com/q/37329) but I don't see any answer there that would work faster than `rm -rf session`. – Stéphane Chazelas Sep 29 '22 at 13:15
  • How would unmounting help? Please elaborate. I've seen the other post, but nothing I found there helps. – Bojan Hrnkas Sep 30 '22 at 18:51
  • 2
    Problem is that removing an entry in that directory means looking up the file in that huge directory, then deallocate the space it uses, set is nlinks to 0 and edit the directory to remove the entry all of which is costly. Since the directory is to be gone in the end, there's a lot of that process that doesn't need to be done. You can just deallocate the directory and the files in it without having to modify it each time. Not something you can do with the normal FS API, but you could modify the FS some other way when not mounted like using the `kill_file` of `debugfs` (if ext4 FS) – Stéphane Chazelas Sep 30 '22 at 19:24
  • 2
    In my quick test,`debugfs` was quite slow, so while in theory, you can make it faster, I don't know of a practical way you could achieve it. Maybr `eatmydata debugfs` could work. Or just use `rm -rf` and wait the few days it's going to take. – Stéphane Chazelas Sep 30 '22 at 19:26
  • **xargs rm * &** might work. – Nils Oct 01 '22 at 19:47
  • 1
    @Nils no, that won't work. At all – roaima Oct 01 '22 at 23:04
  • The problem is solved (see last edit) - thank you for your answers. – Bojan Hrnkas Oct 03 '22 at 10:00

3 Answers3

3

Solved: 4 days later the rsync finished the job - all files have been deleted - but it needed at least 2 days to read directory info, before deleting a single file.

This is information for anyone who might have a similar problem: use screen and be patient.

Bojan Hrnkas
  • 201
  • 1
  • 6
  • 1
    Generally speaking `rm -rf` would have been simpler and therefore more correct. However, I doubt any of us here could have predicted the exceedingly long wait before any deletions started – roaima Oct 03 '22 at 19:41
1

Put the main directory onto a filesystem of its own. Instead of deleting all files, unmount it, recreate the filesystem and mount the empty filesystem.

Nils
  • 18,202
  • 11
  • 46
  • 82
1

Deleting the files will take longer when applications are accessing the directory. Moving the directory itself to another location (on the same volume!) and creating a new directory helps. But if the application is still running, it could have a handle on that big directory and will therefore not use the new directory.

Therefore:

  • stop the application
  • move directory
  • create new directory
  • start application
  • delete files old directory including files
Ramon
  • 31
  • 3