0

I have a program for counting pixel elements in RGB with a precision of two decimal places. Each image consists of 280 x 250 pixels, and the images are stored in a folder containing 1200 images. At the end of my computations, I get a matrix of around 70 million elements, each element consisting of three components. So I get around 210 million elements. It is not only time consuming, but I run out of memory. I have 16GB memory and I added a swap space of around 56GB. All my RAM was consumed and around 29GB of my swap space was used.

I have four questions.

  1. Is there any way to speed up and memory-optimize my code?

  2. While I use SetPrecision with 2, I still see elements with more than 2 digits of precision. Why?

  3. Can I do a parallel sort to ascending/descending output?

  4. How about parallel counting?

Here is the code:

Export["/home/rjo/FINAL-DATA-TESTING.csv", SetPrecision[Flatten[ParallelTable[Flatten[ImageData[Import["/home/rjo/Documents/Wolfram Mathematica/delta/"<>"delta"<>ToString[x]<>".bmp"]], 1],{x, FileNames["*","/home/rjo/Documents/Wolfram Mathematica/delta/",Infinity]//Length}], 1], 2], "CSV"];
m_goldberg
  • 107,779
  • 16
  • 103
  • 257
user6527
  • 79
  • 2
  • 4
    Doing things in parallel will only make the memory situation worse. Do you really need to have all the files in memory at the same time? Can you not process then one/some at a time? – Marius Ladegård Meyer May 09 '17 at 07:34
  • 1
    SetPrecision may force arbitrary precision arithmetic, which will slow things down and cause higher memory consumption. If you use machine-precision numbers (avoid SetPrecision) and ensure that you always work with packed arrays, then 210 elements will take 8*210 = 1680 MB = 1.7 GB. If the array ever gets unpacked, it will typically inflate to at least 3 times this size. – Szabolcs May 09 '17 at 08:35
  • 1
    Use $HistoryLength=0 (or some other small number). Pack the data with ToPackedArray[]. Don't use Parallel or SetPrecision, do this with machine arithmetic. – Kelly Lowder May 09 '17 at 15:34
  • I understand you've given code, and that's great, but if we can't evaluate the code then it's hard to help you. Specifically here, we don't have the bitmap files you have. If you could reproduce your problem using a bitmap uploaded to a hosting sit like imgur that would make it easier to help. – Jason B. May 10 '17 at 00:45
  • I am not saying this is the best solution, it's just a possible solution. If you want to get it done as fast as you could. You could use Memory Optimized EC2. The most expensive model is X1.32xlarge, memory is 1952G, price is $13.338 per Hour. Of course, you have to set up your EC2 instance and migrate your Mathematica to the EC2 instance. – webcpu May 10 '17 at 08:16

1 Answers1

6

The following should hopefully reinforce why it is preferable to use machine numbers instead of extended precision numbers (even with very low precision):

data=RandomReal[1,10^6];
Developer`PackedArrayQ[data]
ByteCount[data]
ByteCount[SetPrecision[data,2]]

True

8000144

72000080

Carl Woll
  • 130,679
  • 6
  • 243
  • 355