Unreferenced inodes on EC2 linux instance

Question

I have an Amazon EC2 instance which I am using as a NFS file server. It's using a 5x1TB volume RAID0 array. The system is very I/O intensive and files are being written/copied/deleted through the NFS share all the time. Quite often, I notice that there's a BIG discrepancy between the disk space used and the available free space. (I am checking while the system is idle and no files are being written to the file share/system). My only "Fix" for this is to shutdown the instance and restart it (reboot doesn't work and just hangs the machine). When it restarts, it runs fsck and I can see in the system log (many) "unreferenced " Inodes being cleaned up (this is not the entire log):

   25.110924] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 122291727
[   25.114687] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 122291723
[   25.118610] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 122291703
[   25.135184] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 122291722
[   25.140005] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 122291725
[   25.144013] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 122291705
[   25.148008] EXT4-fs (dm-1): 735 orphan inodes deleted
[   25.150286] EXT4-fs (dm-1): recovery complete
[   26.126887] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[  OK  ]

I can't find any solution for this anywhere online. Does anyone know what's causing this or how to prevent it? Or maybe fix it without unmounting the drive ?

Some more info:

Version information:

Linux version 3.10.42-52.145.amzn1.x86_64 (mockbuild@gobi-build-64003) (gcc version 4.8.2 20131212 (Red Hat 4.8.2-7) (GCC) ) #1 SMP Tue Jun 10 23:46:43 UTC 2014

RAID0 array mount on /etc/fstab as follows:

/dev/vg0/data /data ext4 defaults,auto,noatime,noexec 0 0

/etc/mdadm.conf:

DEVICE /dev/xvdk /dev/xvdj /dev/xvdi /dev/xvdh /dev/xvdg
ARRAY /dev/md0 metadata=1.2 name=ip-172-31-10-215:0 UUID=4c4fb472:e0540788:69a83d01:a75a8a3e

/etc/exports:

/data *(rw,sync)

Clients mount the NFS share as follows:

x.x.x.x:/data  /mnt/fileserver nfs defaults 0  0

Answer 1

The behavior you describe can be caused by applications holding files open, even after they've been deleted. If an application has a file open (e.g. tail) and another application comes along and deletes the file (e.g. rm), the first application will continue to hold a reference to the file until the application closes the file. At that point, the filesystem will recognize the file is deleted and not open by any applications, and it will clean up references.

Here is an overly simplistic explanation of how files and inodes are related. A file is essentially a record in a filesystem that assigns a name (or names) to a specific inode. Open files are actually referenced by inode. When you delete a file, you're actually removing the link between the name and the inode, but an open file also maintains a link between the open file descriptor and the inode as well. Closing the file deletes the link between the open file descriptor and the inode. The inode will not be reclaimed by the filesystem until all of the links have been deleted.

When you look at the free space reported by the filesystem, it tells you the space associated with all of the inodes currently marked as used. When you look through all of the directories and sum up the file space used by each file/directory, it can be less if files have been deleted but are still open. Your directory scan won't see the space used by files that have had their name links removed.

When you hard-shutdown the system, you don't give a chance for applications to close their files. Without that chance, the filesystem does not get a chance to reclaim the inodes used by the open file descriptors of deleted files. When the system boots, the filesystem sees these inodes with nothing pointing them. These are called "orphaned inodes", and the filesystem lets you know that it's deleting the file reference.

One tool you can use to find processes with open file descriptors is lsof. If you run this on a process, it will show all open file descriptors of that process. Deleted files will usually show up as (deleted), depending on the version.