7

I have a process that receives a video file (RAW) and transcodes it with FFMPEG generating three (different resolutions) resultant files. I'm using a distributed task queue system (Celery) to process every process from FFMPEG in a different asynchronous task.

The three tasks run, according to the flow

  • Convert video
  • Upload result to a bucket in cloud
  • Delete result

And a last task upload the RAW video (used for transcoding) to bucket, and delete it.

If I start the three tasks asynchronously, and delete the RAW file just after, will the tasks (that are using the RAW file) be interrupted by deleting the file?

PS: I assuming that, the RAW file is loaded in memory, and opened three times, while the transcoding task were started.

G-Man Says 'Reinstate Monica'
  • 22,130
  • 27
  • 68
  • 117
Mauro Baraldi
  • 213
  • 2
  • 3
  • 6

2 Answers2

10

The assumption that the complete RAW file is in memory is not true. Normally, when a file is opened the process gets a file descriptor which can be used to read/write the file.

When a file is opened by a process and then is deleted while the file is still open does not actually delete the file instantly. The file is actually deleted when there are no processes anymore with handles (file descriptors) to that file. You can use lsof to see if the file still has handles and when you delete such file it is often listed with the (deleted) text appended to the line.

Disk space is also not reclaimed when an open file is deleted so it is safe to still use the file as long as it is open. When the deleted file does not have active file descriptors anymore the filesystem will reclaim the consumed disk space.

Lambert
  • 12,495
  • 2
  • 26
  • 35
2

If all three ffmpeg processes have opened the input file before you delete it, then it is quite possible that this will work.  But, if ffmpeg closes the file and reopens it, then it will fail.  If you can determine that ffmpeg doesn't do that, then you're probably safe.

G-Man Says 'Reinstate Monica'
  • 22,130
  • 27
  • 68
  • 117
  • All the tasks of file opening are called before the task of deleting RAW video. Also the processes open only once the file and then write to a resultant file. – Mauro Baraldi Oct 22 '15 at 05:36
  • 1
    Perhaps I wasn't clear.  My point was that, unless you have read the source code of ffmpeg (or tested it *very* ⁠thoroughly), you can't really be sure whether it does anything weird like closing and reopening the file.  There's no good reason why it should do something weird like that, but the thing about weirdness is that it's often difficult to anticipate. – G-Man Says 'Reinstate Monica' Oct 22 '15 at 05:42