Disk full but cannot find were is the used space (Ubuntu)

Question

I have a small Intel NUC with a 30GB drive. My problem is that this drive is full but cannot find the cause of it.

df reporting the following

Filesystem     1K-blocks      Used Available Use% Mounted on
udev              899412         0    899412   0% /dev
tmpfs             189284      2676    186608   2% /run
/dev/sda2       28414508  27751116         0 100% /
tmpfs             946404         0    946404   0% /dev/shm
tmpfs               5120         4      5116   1% /run/lock
tmpfs             946404         0    946404   0% /sys/fs/cgroup
/dev/loop0           128       128         0 100% /snap/bare/5
/dev/loop1         56832     56832         0 100% /snap/core18/2128
/dev/loop2         56832     56832         0 100% /snap/core18/2246
tmpfs             946404         0    946404   0% /tmp
/dev/loop3        314880    314880         0 100% /snap/makemkv/381
/dev/loop4         66688     66688         0 100% /snap/gtk-common-themes/1515
/dev/loop5         63360     63360         0 100% /snap/core20/1169
/dev/loop6         63360     63360         0 100% /snap/core20/1081
/dev/loop7         33280     33280         0 100% /snap/snapd/13270
/dev/loop8        317184    317184         0 100% /snap/makemkv/385
/dev/loop9         33280     33280         0 100% /snap/snapd/13640
/dev/loop10        66816     66816         0 100% /snap/gtk-common-themes/1519
/dev/sda1         306584      5356    301228   2% /boot/efi
tmpfs             189280         4    189276   1% /run/user/1000

Computing this it gives somewhere around ~14GB of used disk space.

Running sudo lsof | grep REG | grep -v "stat: No such file or directory" | grep -v DEL | awk '{if ($NF=="(deleted)") {x=3;y=1} else {x=2;y=0}; {print $(NF-x) " " $(NF-y) } }' | sort -n -u | numfmt --field=1 --to=iec | tail -10

gives me a list with few processes that matters:

5,5M  /usr/lib/php/20190902/fileinfo.so
6,8M  /usr/lib/jellyfin/bin/libcoreclr.so
8,0M  /var/log/journal/6296b00d07874d0a9533eed0efb81840/user-1000.journal
8,2M  /usr/lib/jellyfin/bin/System.Private.Xml.dll
8,3M  /usr/lib/locale/locale-archive
8,9M  /usr/lib/jellyfin/bin/System.Private.CoreLib.dll
10M  /usr/lib/udev/hwdb.bin
24M  /snap/snapd/13640/usr/lib/snapd/snapd
27M  /usr/lib/x86_64-linux-gnu/libicudata.so.66.1
64M  /memfd:pulseaudio

Running sudo du -sh / --exclude=disks --total gives me a total of 13GB.

So, basically I am out of ideas on how to find out were are the missing ~16GB that the system reports as being somewhere filling out my drive.

An the report is actually behave as such, running

cd ~/ && touch example && echo "FooBar" > example
-bash: echo: write error: No space left on device

Thank you in advance, and, any ideea is a good ideea, basically a have a device that at the current moment is unusable and my options are running low (basically a clean reinstall / buy a larger ssd for a device that should not use more than 20gb)

Answer 1

Some possibilities to try to find the thing filling your "/" partition:

• lsof -nP +L1 # should list all files that were deleted (unlinked) but are still opened by a process and therefor still occupy the dist
• See also that answer: https://unix.stackexchange.com/a/68532/27616 which gives a few additionnal informations and things to try
• Another possibility: check (with df -ih / ) if you don't have "millions" of small files in that / filesystem : each file takes at least a "small" amount of disk (usually because it occupies at least 1 inode, which size varies depending on the file size and the filesystem). This can add up... if the minimum disk space occupied is 512bytes, having 1 million files of 1 byte each will still occypy 512 million bytes instead of 1 million bytes. df will show the disk space occupied (full inode space coounted), whereas du will show the added file sizes (ie, just the content of those files, not the space that the inode(s) containing this content occupies)
• Another possibility: there could be some big file(s) hidden by a mounted filesystem. Ie, some files could lie "underneath" a mounted filesystem (maybe you put a lof of big files in the /tmp directory (the one that is in the / filesystem, and which is then used as a mount point to mount the /tmp filesystem) ? This could happen if you put things in there while the /tmp filesystem was not mounted. To check this, you can, in linux, remount / (using a free loop device) as read-only somewhere (say: mount it underneath a /mnt/readonlyroot/ mount point), and browse that using du -hs /mnt/readonlyroot, and compare to du -hxs / # -x prevents du to descend to another filesystem mounted underneath /, such as the /tmp filesystem for exemple).
  • command to mount (a 2nd time) / as readonly underneath some mount point: you can (from the top of my memory... I can't check that on linux right now) :
    • mkdir -p /mnt/rootreadonly/ to create the directory mount point (which will be, ironically, inside the "/" filesystem...)
    • mount -o loop -o ro /dev/sda2 /mnt/rootreadonly (to make that "/" filesystem appear there as read-only. I specify sda2 here as you show your "/" filesystem is in "/dev/sda2" in your question. Some other person reading this answer should first check the output of mount to see where their / filesystem come from...)