14

I have a bunch of videos which I want to check if they are complete or not. Some of them may be downloaded partially, but they are not faulty. How can I efficiently check if these video are completely downloaded?

If I had the links, I would have checked the size of them, but I don't.

I tried to use ffprobe and mediainfo. ffprobe reports minor problems on partially downloaded files, but it also reports similar problems with some of completely downloaded files. Should I use ffmpeg to read the whole files and compare the length of the videos to check if they are downloaded? Is there a better solution?

Ho1
  • 2,552
  • 3
  • 20
  • 25
  • Are you talking about files that are currently being downloaded, i.e. the program that's downloading them is still writing to the file as it receives more data? Or are you talking about files whose download was aborted at some point in the past? – Gilles 'SO- stop being evil' Mar 12 '16 at 12:27
  • @Gilles The are downloaded in the past, and now some of these files are incomplete. I want to find incomplete video files, so that I can manually find the links and resume the download. – Ho1 Mar 12 '16 at 12:32
  • @Ho1 did my answer satisfy your question? – J363 May 05 '16 at 00:40
  • @J363 Thanks for your answer. Could you please respond my comment? – Ho1 May 07 '16 at 08:03

3 Answers3

14

ffmpeg is an OS agnostic tool that is capable of determining if a video file has been completely downloaded. The command below instructs ffmpeg to read the input video and encode the video to nothing. During the encoding process, any errors such as missing frames are output to test.log.

ffmpeg -v error -i FILENAME.mp4 -f null - 2>test.log

If a video file is not totally downloaded, there will be many lines in the test.log file. For example, .1 MB missing from a video file produced 71 lines of errors. If the video is fully downloaded and hasn’t been corrupted, no errors are found, and no lines are printed to test.log.

Edit

In the example I gave above, I tested the entire file because the test video I downloaded was a torrent, which can have missing chunks throughout the file.

Adding -sseof -60 to the list of arguments will check the last 60 seconds of the file, which is considerably faster. 

ffmpeg -v error -sseof -60 -i FILENAME.mp4 -f null - 2>test.log

You'll need a newer version of ffmpeg, 2.8 was missing the sseof flag, so I used 3.0.

Community
  • 1
J363
  • 500
  • 3
  • 22
  • 1
    This works somehow, but it requires processing all the video. As an alternate method, I found that opening the video in VLC, clicking somewhere near the end takes less time and is more practical. Is it possible to define similar thing in `ffmpeg`? I mean seeking to 99% without processing all the video, and checking if it works fine without error. – Ho1 May 07 '16 at 08:02
  • @Ho1 see the updated command. – J363 May 08 '16 at 12:41
  • Thanks for your reply. Are you sure this works? At least it doens't work for the videos with less than 60 seconds, but I could not also use it with any other values, for example 10 seconds. It seems that it does not have any effect. The first method works fine and generates several errors for a test file, but the second does not. I have the `-sseof` option, but It does not output anything. – Ho1 May 10 '16 at 08:44
  • 1
    I am positive the command works because I tested it several times with and without the -sseof flag. The original command checks the whole file, which is the suggested behavior. The amended command you requested asserts the assumption that data is missing from the end of the file into the command, which may not be the case. My test file is 99.9% of a 137MB mp4 downloaded with a torrent client. When I use the -sseof flag and progressively increase the amount of time it looks backwards, I eventually get the exact same output as if I ran the command searching the entire file. – J363 May 11 '16 at 07:40
  • It seems to work on many files, *except few.* **I am accepting your answer. Thanks for your help.** And a side note: downloaded torrent can be checked with the torrent client, with the download hash available. It checks every single chunk of the file according to the hashes and it is far more reiiable. I don't have the links here. – Ho1 May 11 '16 at 07:45
  • 1
    With `-ss` and `-sseof`, ffmpeg does read and process the whole file. So, anyway, opening the video in VLC and clicking near the end of it would be faster. – Ho1 May 12 '16 at 14:54
  • I am aware that a torrent can be hashed in a torrent client, which is why I used the torrent client to stop the download at 99%. I was able to use the torrent client's hash function to verify that a small amount of the file was missing. If chunks can be missing from any part of the file, why is it best to only check the end of the file? Where is the conclusion that the end of the file being missing derived from? – J363 May 29 '16 at 21:59
  • 1
    You're right, but I am downloading over http. – Ho1 May 30 '16 at 07:33
  • This doesn't work for me. This command exits silently on a particular video, but VLC shows the total length as "0:00" and when I try to seek I get a plethora of errors. ("TS Discontinuity (received 3, expected 1) for PID 256") – szxk Jun 24 '16 at 02:00
  • A perfectly valid MP4 file creates thousands of lines of output. – Ken Sharp Jun 28 '20 at 20:29
  • How about checking only first and last second instead of the last 60 seconds? – alper Aug 15 '22 at 11:50
9

MediaInfo displays a line:

Truncated: Yes

if a file is not complete as expected by the format specifications.

As there is technically no difference between a file wrongly (not meeting the specs about file boundaries) muxed and partially downloaded files, it is technically impossible to do the difference between a buggy file and a partially downloaded file.

Another (advanced) test could be done e.g. reading the index of an .mp4 file and checking that the file offset + frame size of the last frame is within the file size of the file you get, but it is not exactly what your are looking for (if there is metadata e.g. posters in the source file, at the end of the file, and the file is truncated just before this poster, the partial download is still not detected in every case). It is not implemented in MediaInfo but you can add a MediaInfo feature request.

In all cases, it is very difficult to detect all partial downloads, because the total file size in not indicated in most video file formats so you can be sure a file is truncated, but you can not be sure a file is not truncated. The only way to be sure you downloaded the complete file is to get the file size from somewhere else (and better: have its hash e.g. MD5).

PS: this question is not specific to any operating system.

Jérôme Martinez
  • 207
  • 1
  • 3
  • Thanks for your answer. I don't want an exact answer, an approximate answer would be enough for me. Do you think this question should be moved to superuser.com? – Ho1 Mar 14 '16 at 13:03
  • I couldn't get such field even by using mediainfo version v0.7.83 using `-f` (full information display). Could you please help on getting `Truncated` field? – Ho1 Mar 14 '16 at 13:17
  • Just remove 1 byte in your file and you should have this "truncated" field displayed (no need of -f, default output is enough). FYI it is implemented for the most used formats today: MP4/MOV, Matroska, MXF. are you testing with another format? – Jérôme Martinez Mar 14 '16 at 16:45
  • 2
    Thanks a lot for the answer. To check many files at once, I used the following: `mediainfo * | grep -E "^(IsTruncated|Complete name)"` where the filename of the corresponding file is being displayed directly above the "IsTruncated: Yes" line. – 0x01 May 13 '18 at 11:47
  • `MediaInfo` has 1 start rating in macOS, is it safe to use it? – alper Jan 19 '21 at 19:48
  • 1
    Haters gonna hate (last 1 star I got is because MediaInfo doesn't play video... It was never said that it is a video player) so sometimes you see 1 star, but right now [MediaInfo AppStore version](https://apps.apple.com/us/app/mediainfo/id510620098) has 4.7 stars in average. Jérôme, developer of MediaInfo. – Jérôme Martinez Jan 20 '21 at 07:19
7

I had a file where ffmpeg (v3.1.1) with -sseof -60 did not show any warnings and just exited with no indication that something was wrong. Mediainfo did not show the file was truncated either. Only leaving out the -sseof -60 would report any error with ffmpeg. So I went back to trying ffprobe. It seemed slightly faster than ffmpeg.

ffprobe -v error -count_frames -i filename.mp4

produced this output

[h264 @ 00000000004e6a60] Invalid NAL unit size.
[h264 @ 00000000004e6a60] Error splitting the input into NAL units.
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000004e5280] stream 1, offset 0x1350135: partial file

I did not yet encounter any false positives (as reported by the OP).

Edit: ffprobe was on my testfiles about 10% faster than ffmpeg, but only if you instruct it to use all cores with the option '-threads 0'. Otherwise it uses only one core/thread and is slower.

Jan Ehrhardt
  • 171
  • 1
  • 4
  • This does not provide an answer to the question. Once you have sufficient [reputation](http://unix.stackexchange.com/help/whats-reputation) you will be able to [comment on any post](http://unix.stackexchange.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](http://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/low-quality-posts/152354) – Braiam Jul 30 '16 at 17:36
  • 1
    @Braiam It does answer the question, because a compelety downloaded file will produce no output and a incomplete file will. – Jan Ehrhardt Jul 31 '16 at 19:09