Extract images from large AVI movies (50-100 GB)

Question

I will get large RGB AVI files (35 Hz, 1600$\times$1200 pixels) of up to 100 GB size. All three channels contain the same grayscale image. I want to extract each AVI file into one channel png images. To do that I wanted to use the following code snippet:

nImages = Length@Import[aviFileName, "Frames"];

ParallelDo
[
  image = ColorConvert[Import[aviFileName, {"Frames", i}], "Grayscale"]; 
  imageName = ToString@PaddedForm[i, 6, NumberPadding -> {"0", ""}];
  Image`ImportExportDump`ImageWritePNG[StringJoin[imageName, ".png"], image];
  , {i, 1, nImages}
];

Questions:

Can such large files be read this way?
How can the performance be improved?

This is not a very backed up answer, but from my experience it can be tough to work with files that big in Mathematica, because Mathematica usually tries to load everything into memory when Importing. Because of that i try to avoid loading files bigger than ˜1GB into Mathematica. For your specific problem i would try a specialised tool like VirtualDub or Adobe Premiere or FinalCutPro/Compressor if you're on a mac. But i would very much like if someone with more Mathematica knowledge proves me wrong :) — Thies Heidecke, Mar 11 '17 at 15:53
@Thies Heidecke: I also tried VirtualDub but it exports rgb avi files to three channel png images, although I selected the filter "Grayscale", the same problem is with Quicktime. After that I used XnConvert to extract only one channel. — mrz, Mar 11 '17 at 16:01
Have you tried this on your avi file? I'm sure that importing all frames of 100GB avi files, which you do in your first line, won't work because it will load them into your RAM. If you have more than 100GB RAM indeed and this works, then one way to try would be to turn it into a list {{1, frame1},{2, frame2},...} and process this in parallel. Still, it is not sure that this will work. — halirutan, Mar 11 '17 at 16:04
@mrz: I think if you can afford the extra space and processing needed, you could use VirtualDub to convert the avi to a list of three channel .pngs and then use Mathematica to do the grayscale conversion in a second pass. This way Mathematica only needs to hold one image in memory at a time. — Thies Heidecke, Mar 11 '17 at 16:04
@mrz: ImageMagick also seems worth a look. It seems it could do both steps you want (avi->png and rgb->grayscale). — Thies Heidecke, Mar 11 '17 at 16:07
@halirutan: The maximum what I tried with the upper code was 1GB. VirtualDub and Quicktime do work with large files, — mrz, Mar 11 '17 at 16:18
@Thies Heidecke: I would like to have a complete solution with mathematica. Concerning ImageMagick check: Dimension@ImageData@image, to see how many channels a png has. — mrz, Mar 11 '17 at 16:22
If you are on Windows you can use MathMF from here. It reads frames one at a time so there should be no problem working with a very large file. — Simon Woods, Mar 11 '17 at 17:31

mrz · Accepted Answer · 2017-03-17T13:56:21.620

Here is an update where I compared the image extraction (8bit, 1 channel) from two AVI files:

My computer (Mathematica 11.0.1, Win 10 Pro, Intel i7 7490-MX, 3.1GHz, 4 cores) has 16 GB RAM and it never used in total more than 5 GB RAM, all running processes included.

The code can be used for multiple AVIs in different subdirectories in a certain base directory:

(* Read in folder which is base dir to search for avi files in all subdirs *)

ChoiceDialog[{FileNameSetter[Dynamic[baseDir], "Directory"], Dynamic[baseDir]}];

SetDirectory[baseDir];

aviNames = FileNames["*.avi", baseDir, Infinity];

numberAVIFiles = Length[aviNames];

aviDirs = DirectoryName[#] & /@ aviNames;

pngBaseDirs = FileBaseName[#] & /@ aviNames;

pngFullDirs = Transpose@{aviDirs, pngBaseDirs} /. {a_, p_} :> StringJoin[a, p];


(* Create a new directory for pngs if it does not exist otherwise delete it and create a new one *)
(If[DirectoryQ[#], DeleteDirectory[#, DeleteContents -> True]; CreateDirectory[#], CreateDirectory[#]]) & /@ 
  pngFullDirs;

Do[
   numberImages = Length@Import[aviNames[[j]], "Frames"];

   ParallelDo[

    image = ColorConvert[Import[StringJoin[aviNames[[j]]], {"Frames", i}], "Grayscale"];

    fileNameCounter = ToString@PaddedForm[i, 6, NumberPadding -> {"0", ""}];

    Image`ImportExportDump`ImageWritePNG[
     StringJoin[pngFullDirs[[j]], "\\", pngBaseDirs[[j]], "_", fileNameCounter, ".png"], image];

    fileName = StringJoin[pngBaseDirs[[j]], "_", fileNameCounter, ".png"];


    , {i, 1, numberImages}
    ];

   , {j, 1, numberAVIFiles}
   ];

Extract images from large AVI movies (50-100 GB)

1 Answers1