18

The /dev.. is full:

SERVER:/dev # df -mP /dev
Filesystem         1048576-blocks      Used Available Capacity Mounted on
udev                     12042     12042         0     100% /dev

There is no files that consume space!

SERVER:/dev # find . -ls | sort -r | head -2
2790517    0 -rw-r--r--   1 root     root            0 Dec 16 10:04 ./devnull
1490005831    0 -rw-------   1 root     root            0 Dec 16 07:54 ./nul
120387    0 lrwxrwxrwx   1 root     root           12 Dec 03 05:42 ./disk/by-uuid/xx..foo..xx -> ../../dm-13
SERVER:/dev # du -sm * 2>/dev/null | sort -nr | head -4
1   shm
0   zero
0   xconsole
0   watchdog

swap is used heavily: 

SERVER:/dev # free -m
         total       used       free     shared    buffers     cached
Mem:         24083      23959        124          0        327      21175
-/+ buffers/cache:       2455      21627
Swap:        10245      10245          0

deleted but still used files (?): 

SERVER:/dev # lsof /dev | grep deleted
su         4510   bar   14u   REG    0,14 6269616128 2689827477 /dev/shm/kdfoo.a4o (deleted)
grep       4512       root    1u   REG    0,14 6269616128 2689827477 /dev/shm/kdfoo.a4o (deleted)
bash       4517   bar   14u   REG    0,14 6269616128 2689827477 /dev/shm/kdfoo.a4o (deleted)
sh         4606   bar   14u   REG    0,14 6269616128 2689827477 /dev/shm/kdfoo.a4o (deleted)
ksh       24134       root    1u   REG    0,14 6329864192 2685851781 /dev/shm/foo5.44m (deleted)
ksh       29209       root    1u   REG    0,14 6269616128 2689827477 /dev/shm/kdfoo.a4o (deleted)
su        29571   bar   14u   REG    0,14 6329864192 2685851781 /dev/shm/foo5.44m (deleted)
grep      29573       root    1u   REG    0,14 6329864192 2685851781 /dev/shm/foo5.44m (deleted)
bash      29578   bar   14u   REG    0,14 6329864192 2685851781 /dev/shm/foo5.44m (deleted)
sh        29694   bar   14u   REG    0,14 6329864192 2685851781 /dev/shm/foo5.44m (deleted)
SERVER:/dev #

My question: what is using up all the 12 GByte space of "udev on /dev type tmpfs (rw)"?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
newuser999
  • 503
  • 2
  • 4
  • 13
  • 1
    Can you paste in the output of `df -h` and `ps u --sort=-pmem`? – Atle Dec 16 '13 at 21:47
  • Weird. What other mount points do you have? Any containers or virtualization going on? Also, you ran `du *`: any chance of a dot file? What does `du -x /dev` say? – Gilles 'SO- stop being evil' Dec 16 '13 at 22:47
  • 1
    Weird indeed. Also, the output of `mount | grep /dev` would help. – Alexios Dec 17 '13 at 22:31
  • 2
    My take: /tmp is mounted/linked to /dev/shm and various tools and programs use it. But since it is shared with udev, udev is also full (all tmpfses are probably full). Recommendation: Check the other tmpfs and /tmp, mount /tmp to its own tmpfs with a size limit, mount option "size=XXX" – vasquez Dec 19 '13 at 08:28
  • OP appears (from 'find . -ls | sort -r | head -2' listing) to have files names /dev/devnull and /dev/nul (as opposed to the usual '/dev/null') Are these perhaps 'sparse' files? – D McKeon Dec 20 '13 at 03:15

1 Answers1

8

Shared memory is using the 12GB.

On your Linux release /dev/shm part of the /dev filesystem (on some releases, it has its own a dedicated file system mounted there).

As shown by lsof, the sum is 12 GB:

/dev/shm/foo5.44m is 6269616128 bytes   
/dev/shm/kdfoo.a4o is 6269616128 bytes

Neither find nor ls can display theses files because they are unlinked (= their names have been deleted).

derobert
  • 107,579
  • 20
  • 231
  • 279
Emmanuel
  • 4,167
  • 2
  • 23
  • 30