Managing memory for video frames

Question

I am trying to import a gif animation, which is about 80MB. The goal is to convert the image to grayscale, and then do some calculation on ImageData for each frame. The operations are in the sample code:

data = ColorConvert[Import["MOV_5705.gif"],"Grayscale"];
datag = RemoveAlphaChannel /@ data;
datag = ImageData /@ datag;
datag = Flatten /@ datag;

The issue is that Mathematica will use up all 32 GB of memory available to me by the end. I cannot even print out data from datag list. The gif file has 554 frames and each frame holds 2073600 values, which gives approximately 9 GB of double precision data. Is there any way to make processing the frames manageable. (the gif file can be downloaded here)

Answer 1

I know halirutan has already provided an answer, but here's a nice way to deal with GIF import in a frame-wise manner (or really in chunks of frames).

Mathematica does a very dumb thing when it reads GIFs. It calls the function System`Convert`CommonGraphicsDump`IEImageRead which, if intelligent, would merely process the requested frames. Instead it always takes every single frame, which is returned as an Image3D calls Image3DSlices.

Why it does this is truly beyond me, but you can check it like so:

Block[{Image3DSlices = Echo[Head[#]] &},
 Import[gif, {"ImageList", 5}]
 ]

Image3D

Import::fmterr: Cannot import data as GIF format.

$Failed

Instead of using that we'll use the built in "GIFTools`" library which is intelligent about this. The core functions we'll want are these:

{
 GIFTools`Private`$ReadOneFrame,
 GIFTools`Private`$ReadPalettes,
 GIFTools`Private`$ReadAllFrames,
 GIFTools`Private`$ReadRasterBits,
 GIFTools`Private`$ReadFileMetadata, 
 GIFTools`Private`$ReadFrameMetadata,
 GIFTools`Private`$ReadGlobalPalette
 }

The most useful parts here are $ReadFileMetadata and $ReadOneFrame, but all of these can be used to dramatically outperform the default Import implementation.

Now we can effectively map a function over blocks of frames extracted using Range.

Here's the full code. Most of it is just making the interface nice, implementing protections and validations, etc.

gifBlockMapLoadLib[] :=
  (
   Replace[
     If[DownValues@
        System`Convert`CommonGraphicsDump`ImageReadMetadataGIF == {}, 
      Quiet@Import[
         "https://i.imgur.com/7Wkns4d.gif", {"GIF", "Elements"}];
      ];
     DownValues@System`Convert`CommonGraphicsDump`ImageReadMetadataGIF,
     {} :>
      Throw[
       Failure["BadGIFInit", <|
         "MessageTemplate" -> "GIF tools not loaded"|>], "GIFBlockMap"]
     ];
   );
gifBlockMapGetFile[gif_] :=

  With[{base = ExpandFileName@Replace[gif, File[f_] :> f]},
   If[! FileExistsQ@base,
    Throw[
     Failure["BadGIF", 
      <|"MessageTemplate" -> "GIF file `` does not exist", 
       "MessageParameters" -> {gif}|>
      ],
     "GIFBlockMap"
     ],
    base
    ]
   ];
gifBlockMapGetImageCount[gif_] :=
  Replace[
   Quiet@
    System`Convert`CommonGraphicsDump`ImageReadMetadataGIF[gif, 
     "ImageCount"],
   {
    {i_Integer} :> i,
    _ :>
     Throw[
      Failure["BadGIF", 
       <|"MessageTemplate" -> "`` is not a valid GIF file", 
        "MessageParameters" -> {gif}|>
       ],
      "GIFBlockMap"
      ]
    }
   ];
gifBlockMapGetBlockSpec[frameCount_, block_] :=
  Module[
   {
    start, stop, step
    },
   If[! IntegerQ@frameCount,
    Throw[
     Failure["BadGIF", 
      <|"MessageTemplate" -> "GIF frame count `` is not an integer", 
       "MessageParameters" -> {frameCount}|>
      ],
     "GIFBlockMap"
     ]
    ];
   If[block[[3]] <= 0,
    Throw[
     Failure["BadBlock", 
      <|"MessageTemplate" -> "Block step `` must be greater than 0", 
       "MessageParameters" -> {block[[3]]}|>
      ],
     "GIFBlockMap"
     ]
    ];
   {start, stop, step} =
    Replace[List @@ block, 
     {
      All -> frameCount,
      i_Integer?Negative :>
       frameCount - i,
      i_Integer?(# > frameCount &) :>
       Throw[
        Failure["BadBlock", 
         <|
          "MessageTemplate" -> 
           "Block parameter `` is larger than frame count ``", 
          "MessageParameters" -> {i, frameCount}|>
         ],
        "GIFBlockMap"
        ],
      Scaled[s_] :>
       Max@{Ceiling[s*frameCount], 1}
      },
     1
     ];
   If[start >= stop, 
    Throw[
     Failure["BadBlock", 
      <|"MessageTemplate" -> 
        "Block start `` must be smaller than block end ``", 
       "MessageParameters" -> {start, stop}|>
      ],
     "GIFBlockMap"
     ]
    ];
   {start, stop, step}
   ];
gifBlockMapRead["GIF", frame_, ___][gif_, ___] :=
  Replace[
   Switch[
    frame,
    _Integer,
    {GIFTools`Private`$ReadOneFrame[gif, frame]},
    {__Integer},
    GIFTools`Private`$ReadOneFrame[gif, #] & /@ frame,
    All,
    GIFTools`Private`$ReadAllFrames[gif],
    _,
    $Failed
    ],
   Except[{__Image}] :>
    Throw[
     Failure["BadFrames", 
      <|
       "MessageTemplate" ->
        "Can't read frames `` from ``", 
       "MessageParameters" -> {frame, gif}
       |>
      ],
     "GIFBlockMap"
     ]
   ];
gifBlockMap[
   func_,
   gifFile : (_String | _File), 
   mapFunc : Map | Scan : Map,
   blockSpec :
    Span[
     _Integer | Scaled[_?(0 <= # < 1 &)],
     _Integer | Scaled[_?(0 < # <= 1 &)] | All,
     _Integer?Positive | Scaled[_?(0 <= # <= 1 &)] | All
     ],
   sampling : _Integer?Positive : 1
   ] :=
  Function[Null, Catch[#, "GIFBlockMap"], HoldFirst ]@
   Block[
    {
     System`Convert`CommonGraphicsDump`IEImageRead = gifBlockMapRead
     },
    Module[
     {
      gif,
      frameCount,
      start,
      end,
      step,
      curFrame,
      oldFrame,
      frameRange
      },
     gifBlockMapLoadLib[];
     gif = gifBlockMapGetFile[gifFile];
     frameCount = gifBlockMapGetImageCount[gif ];
     {start, end, step} = 
      gifBlockMapGetBlockSpec[frameCount, blockSpec];
     curFrame = start;
     Switch[mapFunc, Map, Table, Scan, Do][
      oldFrame = curFrame;
      curFrame = 1 + Min@{curFrame + step, end};
      frameRange = Range[oldFrame, curFrame - 1, sampling];
      If[Length@frameRange > 0,
       func@Import[gif, {"GIF", "ImageList", frameRange}],
       Nothing
       ],
      Ceiling[frameCount/step]
      ]
     ]
    ];
gifBlockMap[
  func_,
  gifFile : (_String | _File), 
  mapFunc : Map | Scan : Map,
  blockSpec : _Integer?Positive | Scaled[_?(0 < # < 1 &)] | All : 10,
  sampling : _Integer?Positive : 1
  ] :=
 gifBlockMap[
  func, gifFile, mapFunc,
   1 ;; All ;; blockSpec, sampling
  ]

Then we can see this in action:

gif = URLDownload["https://i.imgur.com/7Wkns4d.gif"];

gifBlockMap[Blend, gif, Map, 10] // GraphicsRow[#, ImageSize -> 800] &

It performs okay (at minimum it cuts down our memory consumption):

Blend /@ Partition[Import[gif, "ImageList"], 10] // 
  MaxMemoryUsed // AbsoluteTiming

{0.253601, 41266472}

gifBlockMap[Blend, gif, 10] // MaxMemoryUsed // AbsoluteTiming

{0.807644, 11808856}

We're just under four times slower and use bit under four times less memory.

Amusingly, this outperforms the direct "ImageList import, since we're being less general and don't use Image3D and things:

Import[gif, "ImageList"] // MaxMemoryUsed // AbsoluteTiming

{0.137272, 41266400}

gifBlockMap[Identity, gif, Map, All][[1]] // 
  MaxMemoryUsed // AbsoluteTiming

{0.118931, 20667552}

Even better though is to just directly call $ReadAllFrames:

Image3DSlices@GIFTools`Private`$ReadAllFrames[First@gif] // 
  MaxMemoryUsed // AbsoluteTiming

{0.034736, 41164192}

Or if you can work with the Image3D object instead:

GIFTools`Private`$ReadAllFrames[First@gif] // 
  MaxMemoryUsed // AbsoluteTiming

{0.017178, 20520712}

On the other hand for a big GIF this method makes things more tractable:

bigGIF =
  URLDownload["https://www.wolframcloud.com/objects/b3m2a1/chem/gradient_descent_walkers.gif"];

tm1 =
 gifBlockMap[Head, bigGIF, 35] // MaxMemoryUsed // AbsoluteTiming

{8.12303, 42231864}

tm2 =
 Head /@ Partition[Import[bigGIF, "ImageList"], UpTo[35]] // 
   MaxMemoryUsed // AbsoluteTiming

{1.83082, 470784832}

MapThread[
  ((# - #2)/Max@{##}) &,
  {tm1, tm2}
  ] // N

{0.774613, -0.910295}

It takes 77% longer but use 91% less memory

Answer 2

The bad news is that your code spiked at about 60 GB RAM on my machine before the result is returned. The problem is clearly the Import routine. I would like to offer an alternative if you have more than 22 GB of RAM available.

data = Import[file, {"GIF", "RawData"}];

This gives you the list of frames. Each frame is a matrix of color-indices. So to reconstruct the first frame, you need to extract the color-map.

colorMap = With[{values = Import[file, {"GIF", "GlobalColorMap"}]}, 
   Thread[Range[Length[values]] -> values]];

This needs another 6 GB of RAM, so if you are unsure, then first extract this map and store it as mx locally. The map itself is tiny but again, Import seems to like eating RAM.

Then you could do

data[[1]] /. rules // Image[#, "Byte"] &

to get the first frame as an image. But since you are not interested in the frames, but you only need a flattened list, you can directly work on data.

I tested it with Mathematica 11.3 on OS X.