How to efficiently import videos

Question

I need to do frame-by-frame analysis of videos. I can import the video files I need, but each file is too large to import at once. I can use the Import[fn, "Frames",{n}] command to import images one-by-one, but this takes a long time once n gets large. For example, the first frame it takes 0.14 sec, but the 200th frame takes 2.2 sec. Mathematica seems to be reading from the start of the file each time I do this. Obviously, this will not work. Is there some way to speed this up. E.g. using streams? Or tricking Mathematica to use ImageCapture to get images from a video file instead of a webcam? Thanks

Answer 1

About:

This is my second answer. After playing with Mathematica's video import I have noticed that it makes import mistakes (duplicated frames + artifacts) and thus developed ffmpeg package for Mathematica which is based on ssch answer.

https://github.com/kmisiunas/ffmpeg-mathematica

The speed of this package is as follows on moderate computer:

• Importing one frame is from 2 to 100 times slower, because current method for seeking frame is slow.
• Importing many frames (1000+) is from 2 to 10 times faster.

I am switching to ffmpeg importer because for bulk analysis it proved to be faster and more accurate than default method.

Usage:

Mimicking Mathematica's Import function:

FFImport[ "file.avi", {"Frames", 1}] 

gives first frame (slow way).

FFImport[ "file.avi", {"Frames", Range[1,1000]}] 

gives first 1000 frames (fast)

To test status of ffmpeg library you can run FFmpeg[].

More information about ffmpeg installation in the link above.

Answer 2

If you only want to read it linearly you can tell ffmpeg to dump the video to stdout and then read widthheightbytes-per-pixel bytes at a time to get the video frame-by-frame:

openVideo[fname_, w_, h_] := 
 Module[{video}, 
  video["stream"] = 
   OpenRead[
    "!ffmpeg -i " ~~ fname ~~ " -loglevel quiet -f rawvideo -pix_fmt rgb24 -", 
    BinaryFormat -> True];
  video["SkipFrame", n_Integer: 1] := Skip[video["stream"], Byte, n*3*w*h];
  video["NextFrame"] := Partition[Partition[
     BinaryReadList[video["stream"], "Byte", 3*w*h]
   , 3], w];
  video["NextFrame", n_Integer] := Table[video["NextFrame"], {n}];
  video["NextImage"] := Image[video["NextFrame"], "Byte"];
  video["NextImage", n_Integer] := Table[video["NextImage"], {n}];
  video]

Here's an example:

(* Create a test movie *)
file = $TemporaryPrefix <> "testvid.avi";
Export[file,Table[Rasterize[i, ImageSize -> {352, 200}], {i, 1, 50}]];

video = openVideo[file,352,200]
video["NextImage",2]
video["SkipFrame",10]
video["NextImage"]

For some reason Mathematica seems to read through the entire video when doing Close[video["stream"]] and it might be worth to kill -15 the ffmpeg process manually to speed it up for large files.

For a 720x404 h264 mp4 video on a modest laptop it takes 13s to skip 200 frames and 0.07s to read a frame. The skipping could possibly be sped up by starting a new ffmpeg process that begins at desired frame. I would compare speeds to Import but Mathematica can't read it (for the test video Import takes .2s for first frame compared to .02s ). So as a bonus with this way you can work with many many more video formats :)

Answer 3

The performance can be improved by loading more than 100 frames at a time. To test this we can run:

LoadNFrames[n_] := (Import[video , {"Frames" , Range[1000, 1000 + n]} ]; n);

times = Table[ AbsoluteTiming[ LoadNFrames[n] ], 
         {n, {1, 2, 5, 10, 20, 30, 50, 100, 150, 200, 300, 400, 500, 600, 700, 800}}]

One can see that in Mathematica there is a penalty for loading only few frames. We can improve the performance by preloading the video and storing it in a buffer. For instance:

buffer = Import[video, "ImageList"]; 

GetFrame[n_] := buffer[[n]]

The method uses a lot of memory, therefore, for large videos you will need to load parts of the video. I also typically cut the region of interest before putting it o the buffer.

Answer 4

If using Windows you can try my MathMF package (see here) which loads video frames sequentially using the Media Foundation platform via LibraryLink.

The speed is pretty good. Exact timings will vary according to CPU speed and the performance of the decoder, but as an example my PC here can read a 1280x720 WMV file at 25 frames per second and a 760x554 AVI file at 73 frames per second.

Answer 5

If you need to perform a frame by frame operation, you could use something like VideoMapList:

VideoMapList[myOperation[#Image]&, video]

This will apply myOperation to each frame, and return all the results in a list. The time to fetch the frames this way is much faster than the Import strategy.

Another advantage is that only one frame will be in memory at any one time.