Removing a vagrant file I see .nfs0000000000b869e300000001

Question

I removed a file and now I see:

$ ls
total 64
-rw-rw-r-- 1 502 17229 Sep 17 16:42 page_object_methods.rb
drwxrwxr-x 7 502   238 Sep 18 18:41 ../
-rw-rw-r-- 1 502 18437 Sep 18 18:41 new_page_object_methods.rb
-rw-r--r-- 1 502 16384 Sep 18 18:42 .nfs0000000000b869e300000001
drwxrwxr-x 5 502   170 Sep 21 13:48 ./
13:48:11 *vagrant* ubuntu-14 selenium_rspec_conversion

and if I try to remove it:

$ rm .nfs0000000000b869e300000001
rm: cannot remove ‘.nfs0000000000b869e300000001’: Device or resource busy

What does this indicate? What should I do

Answer 1

A file can be deleted while it's open by a process. When this happens, the directory entry is deleted, but the file itself (the inode and the content) remain behind; the file is only really deleted when it has no more links and it is not open by any process.

NFS is a stateless protocol: operations can be performed independently of previous operations. It's even possible for the server to reboot, and once it comes back online, the clients will continue accessing the files as before. In order for this to work, files have to be designated by their names, not by a handle obtained by opening the file (which the server would forget when it reboots).

Put the two together: what happens when a file is opened by a client, and deleted? The file needs to keep having name, so that the client that has it open can still access it. But when a file is deleted, it is expected that no more file by that name exists afterwards. So NFS servers turn the deletion of an open file into a renaming: the file is renamed to .nfs… (.nfs followed by a string of letters and digits).

You can't delete these files (if you try, all that happens is that a new .nfs… appears with a different suffix). They will eventually go away when the client that has the file open closes it. (If the client disappears before closing the file, it may take a while until the server notices.)

Answer 2

User @mtak on another question suggests:

You could try runningfuser /path/to/.nfsto check which process is using the .nfs file. – mtak May 2 '14 at 9:13

^^^^^ That works ^^^^^ And kill off the offending process to cause it to release the file handle.

e.g.

$ rm -rf ~/Downloads
rm: cannot remove ‘/nfshome/x/Downloads’: Directory not empty
$ ls -alstr ~/Downloads
total 38864
  972 -rw-r--r--   1 x users   988438 Dec 20  2016 .nfs00000000018d307a00000369
31812 -rw-r--r--   1 x users 32503812 Dec 20  2016 .nfs00000000018d307f0000036b
  636 drwx--x--x 134 x y   647168 Aug 28 10:37 ..
  240 drwxr-xr-x   2 x y   241664 Aug 28 10:43 .
$ rm -rf ~/Downloads
rm: cannot remove ‘/na-homes/x/Downloads/.nfs00000000018d307a00000369’: Device or resource busy
rm: cannot remove ‘/na-homes/x/Downloads/.nfs00000000018d307f0000036b’: Device or resource busy

$ fuser /nfshome/x/Downloads/.nfs00000000018d307400000367
/nfshome/x/Downloads/.nfs00000000018d307400000367:  8231m
$ ps -elf |grep 8231
0 S x     1493 15153  0  80   0 - 28177 pipe_w 10:55 pts/39   00:00:00 grep --color=auto 8231
0 S x     8231  7660  0  99   - - 481464 poll_s Jul19 ?       00:06:01 /usr/libexec/tracker-extract
$ kill 8231
$ kill 8231 # kill twice to check first kill worked, . . 
            # escalate to kill -9 8231 if first kill didn't work, . . 
            # use sudo or root or other user to kill if ownership prevents kill working.
-bash: kill: (8231) - No such process
$ rm -rf ~/Downloads

$ ls -alstr ~/Downloads/
ls: cannot access /nfshome/x/Downloads/: No such file or directory

YAY! Success.

YMMV of course. It might be a different process sitting with the file open.

The tracker-extract process was restarted automatically after I killed it.

What is this tracker-extract thing ? (I see this on centos/redhat)

https://stackoverflow.com/questions/26737900/tracker-extract-and-tracker-store-processes-consuming-huge-amount-of-ram

extra/tracker 1.2.3-1 (gnome)
    All-in-one indexer, search tool and metadata database

Answer 3

As NFS is "stateless", there need to be a way to emulate the UNIX method of opening a file and then remove it keeping an open filehandle.

Any NFS file operation causes the chain:

open(); seek-last-off(); doit(); close();

to be run and this is the reason why NFS survives a server reboot.

Once the process on the client that opened the old file terminates, the file will disappear.

Correctly implemented fileservers will run a script each night that removes all such files that are older than a week. The reason is that in case the client reboots while holding such a file, the file would stay forever.

Answer 4

Some other process is likely still using the file (that is, has an open filehandle to it). Either ignore the file, or use lsof or the like to try to find what process has that file open (or reboot everything!).

Answer 5

I faced a similar situation, but in my case I am not able to delete a file created by my own program. I was sure of this because it was present in a directory created by my program. I did not know where and when I ran that program. Solution: I simply exited from all my terminals. I logged in again and simply deleted the file.

P.S. My answer is only valid for the scenerio I have specified.