Why do I have no disk space despite the fact my root directory only takes up a small amount?

Question

I'm using Ubuntu Linux 16. I'm running out of disk space ...

[rails@server ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/simfs       30G   29G  1.5G  96% /

but what I don't understand is when I get the total size of the "/" directory, it is showing way less than 29G ...

[rails@server ~]$ sudo du -sh /
du: cannot access ‘/proc/883/task/883/fd/4’: No such file or directory
du: cannot access ‘/proc/883/task/883/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/883/fd/4’: No such file or directory
du: cannot access ‘/proc/883/fdinfo/4’: No such file or directory
4.5G    /

What's going on here? More importantly, how do I figure out where all this mystery disk space is getting used?

Edit:

Output as requested by GracefulRestart ...

[rails@server ~]$ sudo lsof -nP 2>&1 | grep 'deleted'
postmaste  443      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  443      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  469      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  469      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  470      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  470      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  471      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  471      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
postmaste  472      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste  472      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)
httpd     1130          root  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1130          root  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131        apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131        apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1133   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1133   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1134   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1134   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1135   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1135   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1136   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1136   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1137   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1137   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1138   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1138   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1139   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1139   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1140   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1140   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1141   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1141   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1142   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1142   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1143   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1143   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1144   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1144   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1145   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1145   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1146   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1146   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1147   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1147   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1148   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1148   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1131 1149   apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1131 1149   apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1132        apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1132        apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
httpd     1158        apache  mem       REG              144,4           147485837 (deleted)/dev/zero (stat: No such file or directory)
httpd     1158        apache  mem       REG              144,4               65537 (deleted)/VE20428-SYSV0176748a (stat: No such file or directory)
postmaste 1270      postgres  mem       REG              144,4           147159237 (deleted)/dev/zero (stat: No such file or directory)
postmaste 1270      postgres  mem       REG              144,4                   0 (deleted)/VE20428-SYSV0052e2c1 (stat: No such file or directory)

Edit 2: Here's the output from the second command ...

[rails@server ~]$ sudo du -h --max-depth=1 /; ls -plah / | grep -v /
214M    /opt
4.0K    /media
12K /boot
6.7M    /run
4.0K    /lost+found
du: cannot access ‘/proc/3478/task/3478/fd/4’: No such file or directory
du: cannot access ‘/proc/3478/task/3478/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/3478/fd/4’: No such file or directory
du: cannot access ‘/proc/3478/fdinfo/4’: No such file or directory
0   /proc
4.0K    /dev
4.0K    /.cpt_hardlink_dir_a920e4ddc233afddc9fb53d26c392319
114M    /tmp
0   /sys
14M /etc
3.6M    /root
396M    /home
12K /.rbenv
4.0K    /srv
1.9G    /usr
1.9G    /var
8.0K    /mnt
4.5G    /
total 64K
-rw-r--r--  1 root root    0 May  5  2018 .autorelabel
-rw-r--r--  1 root root    0 Nov  8 17:14 reboot
-rw-r--r--  1 root root    0 Nov  8 17:15 .vzfifo

try with a bind mount https://unix.stackexchange.com/a/358331/30851 or check from live cd — frostschutz, Nov 08 '19 at 21:53
That's a useful utilty, but sadly it only confirmed what I already new -- the top usage directory within "/" is "/var" at 1.9 GB. So still uncertain what the issue is. — Dave, Nov 08 '19 at 22:09
It is possible for deleted files to hold disk space on a disk. See if you get any output for `lsof -nP 2>&1 | grep 'deleted'`. The column before the path should be a file size. — GracefulRestart, Nov 08 '19 at 22:13
@GracefulRestart, that produced a whole bunch of stuff, which I included as an edit to my question. However, I'm a little clueless as to what it means and what to do next. For instance, is "(deleted)/dev/zero" a file path that I can delete? — Dave, Nov 08 '19 at 22:34
According to that output, `postmaster` (PostgreSQL) and `httpd` (Apache) are the processes holding those deleted files open. Restarting those processes should release those holds. Unfortunately, it does not look like that would equal the ~25GB of missing space... What does the output of `du -h --max-depth=1 /; ls -plah / | grep -v /` show? Any large files at the root (/)? — GracefulRestart, Nov 08 '19 at 23:03
@GracefulRestart, I included the output from the max-depth=1 command as a second edit. I also did a "sudo reboot now." Although nothing has changed with the disk usage, the Apache processes have dropped off the list and only the postmaster/Postgres ones remained. — Dave, Nov 09 '19 at 00:04
Not sure if I understand your question correctly. Do bear in mind that `/` includes all directories which are not mapped to other partitions or disks. So, if you check `/`, it will very probably include `/lib`, `/tmp`, `/var` and also `/usr/lib` and others which are rarely mapped. There used to be a great utility called `xdiskusage` which graphically shows where things are. — jcoppens, Nov 09 '19 at 04:30
@jcoppens, My question is how do I get the usage reported by "df" to go down? Utilities like "du" and "ncdu" report that "/" and its sub-directories are only using fractions of what "df" is reported being used on "/". — Dave, Nov 09 '19 at 18:20
try running `du -xh / | sort -h > /tmp/ssu` to get a sorted list of the directories by space used. then examine say the tenth from the end entry (the last entry will be for "/", and probably the one before that will be "/var") — icarus, Nov 10 '19 at 00:07

Why do I have no disk space despite the fact my root directory only takes up a small amount?

0 Answers0