8

I have some legitimate data sets that overload the memory available to TeX. Of course the recommended step is to cut the number of data points instead of changing the "main memory size". (In my case the overflow occurs a the axis level since it is mainly the number of coexisting curves what produces the overflow.) The question is, does anybody found a rule of thumb of what is the size of the dataset that takes the whole default memory available?

Is the number of points what can tell if the memory will be full? Is the number of characters enclosed in a coordinates environment? Is the number of points multiplied by the number of curves to draw a given point in the data?

To be specific, what is the maximum number of points that can be plotted in a no markers style?

The idea is that one can (e.g. programmaticaly) tell in advance if a certain set can be plotted and depending on that resample the data.

alfC
  • 14,350
  • 1
    Not an answer to your question, but I have found that simply using pdflualatex instead of pdflatex allows me to plot datasets that would otherwise lead to memory exhaustion. – Jake Mar 17 '11 at 11:08
  • 1
    Is there a particular reason why you do not want to increase the main memory size? I mean, resampling is a lossy operation, and any automated tool might fail (unless it uses nonlinear compression techniques with error estimators). When you use pgfplots intensively, you will easily break the memory limits - which, after all, have not been intented to plot anything. – Christian Feuersänger Apr 23 '11 at 20:53
  • @jake I think that in TeX Live 2010 pdflualatex is just lualatex. Don't know about MikTeX 2.9. – Sharpie Apr 23 '11 at 22:41
  • @Christian: the question is not how to manage to do a particular plot, but whether I would need to lossely compress the data before trying to plot, given, for example, the default memory limits. – alfC Apr 24 '11 at 01:21

1 Answers1

11

PGFPlots allocates storage inside of its axis environment as follows:

For every plot, it maintains meta data which does not vary much from plot to plot (unless the number of supplied options differs extremely). The meta data also contains cycle list information.

Aside from this constant overhead, the current versions of pgfplots store the coordinates for each plot. This includes the x, y, z (if any), and meta data (color data, if any) coordinates by default. Each coordinate is stored as floating point number, and any digit is stored (which scales linearly in the number of digits you provided).

I think a good estimate is to take the number of points, the number of coordinates to store for each point (2 for a 2d plot without colordata, one more for a 3d plot, and one more if color data is needed), and the number of digits for each coordinate. That what indentifies the mem usage for a plot with no markers style from the perspective of pgfplots.

However, pgf also allocates memory: it allocates the low level pdf stream data needed to typeset the individual lines. For a line plot (straight lines, nothing smoothed) without markers and without individually colored segments, this will boil down to two fixed points numbers plus a constant number of characters (parenthesis, commas, one char for the line-to operator). The fixed point numbers will have between 3 and 11 characters.

Note that this system-level-buffer of pgf also stores any drawing commands for the axis. This can be considered to be "constant overhead" in most cases - it won't influence the suggested resampling control, I guess.

I hope this answers the question deep enough. I think pgfplots mainly consumes the "main memory size", but if you need more information, one might need to profile the other memory types of TeX as well.

But as already mentioned in my comment above, I would rather suggest to increase the main memory size (the pgfplots manual contains detailed instructions how to do it) than to attempt semi-automatic down-sampling just to make TeX happy (especially considering that the system default of the main memory size varies between TeX distributions and will probably change in the future anyway).

Christian Feuersänger
  • 55,840
  • no doubt a very in-depth answer. From what you wrote: let say one is trying to have a line plot with no markers and that each floating point takes 8 bytes(?). From this I get that for a default memory of 300000 bytes, I can do plots of about 15000 2d-points, for the extra data, lets say 5 bytes per point the number can go down to ~10000 points. – alfC Apr 24 '11 at 01:28
  • 1
    @alfC I encourage you to verify your hypothesis by means of an experiment: try a sequence of line plots, each with the same arithmetic expression, but with a different number of samples. Then, collect a table 'numpoints' versus 'memused' (see the .log file for this information). Plotting this file will allow you to verify the statement. If there is a huge gap between expected and actual, you may contact me again and I can investigate. – Christian Feuersänger Apr 25 '11 at 18:29
  • 1
    I did the systematic tests and a bare bones single line plot (no markers) in an empty page, and for the default memory limit (3000000) allowed up to 12175 points. (The exact number I guess will depend also in the number of "ticks" of the plot.) (The resulting PDF is .) The number can decrease slightly if there are other elements (eg other plots) in the page. But for example two independent plots of ~11000 points or two line plots of ~10000 points in the same axis where allowed. My conclusion is that data must be decimated to a maximum of ~10000 if the default memory is to be assumed – alfC Dec 03 '11 at 08:22