0

I know there have been a few questions about efficient memory storage on mathematica.stackexchange, but I could not find a sufficient answer for my problem (except that it might not work at all).

So what I do have is a large Array, which has Length 4 and is constructed via the following function (with input data from some 8bit per channel image):

(*definition*)
computeDistancesTest = Compile[{{imageData, _Real, 3}, {param, _Real}},
   Map[1/(2.*Pi*param^2)*
      Exp[-Total[(Transpose[imageData, {2, 3, 1}] - #)^2]/(2*
          param^2)] &, imageData, {2}]
   , CompilationTarget -> "C", 
   CompilationOptions -> {"InlineExternalDefinitions" -> True}, 
   RuntimeOptions -> "Speed"];

(*creation*)
n=100;
testImageData = RandomReal[{0, 1}, {n, n, 3}];
distances = computeDistancesTest[testImageData, 1/10.];

The computeDistancesTest function has essentially computes some kind of distance value for each possible pair of pixels in the image. The created array has n^4 entries so by increasing n (corresponding to the image resolution) this array gets considerably more entries. As the function creating the array is compiled the output has MachinePrecision and is also a packed array due to containing all real numbers with the same Precision.

Here are the values I get with Mathematica 10.1 for n=100 (as set above):

Developer`PackedArrayQ[distances]

True

 ByteCount@distances

800000168

So here are my questions:

  • Why is the ByteCount so high (8 bytes per entry seems way to much for me, especially considering that the array is a PackedArray)?
  • How can I reduce the memory consumed by the produced array considering that I want to handle images with resolutions that would make my 16 GB of RAM explode with the array created above?
  • I do not need high Precision, so I could sacrifice MachinePrecision, is there an option to reduce memory usage that way?
Wizard
  • 2,720
  • 17
  • 28
  • Machine reals do occupy 8 bytes per entry. – ilian Jun 05 '15 at 18:30
  • @ilian: That makes sense, but I do not need that kind of precision and also the Packing of the array should reduce the size of the array. Right now it just seems to occupy 8bytes per entry, which is uncompressed. – Wizard Jun 05 '15 at 18:54
  • Packed arrays do not involve any compression. It's the same amount of space that a C array would take. – ilian Jun 05 '15 at 19:01
  • @ilian: Strange, why does an unpacked array have a bigger ByteCount then? – Wizard Jun 05 '15 at 20:44
  • 2
    An unpacked array can contain arbitrary expressions of different types, so the internal representation is more complicated. – ilian Jun 05 '15 at 21:12
  • @ilian I guess it might be a duplicate. In general I would also be interested in alternatives to reducing Precision. It's a shame that mathematica does not seem to be able to use float16 etc. instead of float64. Python has no problems with that and it obviously reduces the amount of memory needed drastically. And yes there are lots of situations where float16 gives enough precision. – Wizard Jun 06 '15 at 23:13

0 Answers0