Is it safe to move a file that's being appended to?

Question

I have a node.js process that uses fs.appendFile to add lines to file.log. Only complete lines of about 40 chars per lines are appended, e.g. calls are like fs.appendFile("start-end"), not 2 calls like fs.appendFile("start-") and fs.appendFile("end"). If I move this file to file2.log can I be sure that no lines are lost or copied partially?

Answer 1

As long as you don't move the file across file-system borders, the operation should be safe. This is due to the mechanism, how »moving« actually is done.

If you mv a file on the same file-system, the file isn't actually touched, but only the file-system entry is changed.

$ mv foo bar

actually does something like

$ ln foo bar
$ rm foo

This would create a hard link (a second directory entry) for the file (actually the inode pointed by file-system entry) foo named bar and remove the foo entry. Since now when removing foo, there is a second file-system entry pointing to foo's inode, removing the old entry foo doesn't actually remove any blocks belonging to the inode.

Your program would happily append to the file anyways, since its open file-handle points to the inode of the file, not the file-system entry.

Note: If your program closes and reopens the file between writes, you would end up having a new file created with the old file-system entry!

Cross file-system moves:

If you move the file across file-system borders, things get ugly. In this case you couldn't guarantee to have your file keeping consistent, since mv would actually

• create a new file on the target file-system
• copy the contents of the old file to the new file
• remove the old file

or

$ cp /path/to/foo /path/to/bar
$ rm /path/to/foo

resp.

$ touch /path/to/bar
$ cat < /path/to/foo > /path/to/bar
$ rm /path/to/foo

Depending on whether the copying reaches end-of-file during a write of your application, it could happen that you have only half of a line in the new file.

Additionally, if your application does not close and reopen the old file, it would continue writing to the old file, even if it seems to be deleted: the kernel knows which files are open and although it would delete the file-system entry, it won't delete old file's inode and associated blocks until your application closes its open file-handle.

Answer 2

Since you say you're using node.js, I assume you'd be using fs.rename() (or fs.renameSync()) to rename the files. This node.js method is documented to use the rename(2) system call, which does not touch the file itself in any way, but merely changes the name under which it is listed in the file system:

"rename() renames a file, moving it between directories if required. Any other hard links to the file (as created using link(2)) are unaffected. Open file descriptors for oldpath are also unaffected."

In particular, note the last sentence quoted above, which says that any open file descriptors (such as your program would be using to write to the file) will continue to point to it even after it has been renamed. Thus, there will be no data loss or corruption even if the file is renamed while it's simultaneously being written to.

---

As Andreas Weise notes in his answer, the rename(2) system call (and thus fs.rename() in node.js) will not work across filesystem boundaries. Thus, attempting to move a file to a different filesystem in this way will simply fail.

The Unix mv command tries to hide this limitation by detecting the error and, instead, moving the file by copying its content to a new file and deleting the original. Unfortunately, moving files like this does risk data loss if the file is moved while it is being written to. Thus, if you want to safely rename files that may be simultaneously written to, you should not use mv (or, at least, you should make absolutely sure that the new and old path are on the same filesystem).