Warning for TeX main capacity exceeded when loading files with \pgfplotstableread

Question

I am loading quite a few data files via \pgfplotstableread, but do not further use them via pgfplots. I just kept them in the preamble and switched to plotting them in a standalone document. But now I've run my main document again, I've got:

PGFPlots: reading {xxxxxxxxx/xxxxxxxxx/xxxxxxxxx.csv}
PGFPlots: reading {xxxxxxxxx/xxxxxxxxx/xxxxxxxxx.csv}

Runaway definition?
->
! TeX capacity exceeded, sorry [main memory size=5000000].
\pgfplotsapplistXpushback@smallbufoverfl ...@toka 
                                                  \the \t@pgfplots@tokb }\ex...
l.87 ...sv}{\xxxxxxxxxWordWordWordxxxxxxxx}
                                                  ^^M

If you really absolutely need more capacity,
you can ask a wizard to enlarge me.

Kind of looks like I forgot to close some bracket somewhere. However on first check that does not seem to be the case. Is the loading of the data files already too much and should I outsource that?

edit (2016-07-13)

In hindsight, I could have edited this into the post way earlier but unfortunately, as I am not that active anymore, I just caught wind of this question's popularity. So ultimately, I plotted the graphs and figures, respectively, one-by-one in an external file (with the help of the standalone class). For huge data, I used LuaLaTeX. To get a proper start for this, these might help:

• standalone - Can I specify the font size in the document?

• Best practice - how can I get my whole document (including math) to only use font X with LuaLaTeX and fontspec?

Answer 1

Combining our chat and the comment to give some ideas though the real solution is mostly art instead of technique.

pgfplots doesn't create a miracle within TeX. It simply uses PGF for harvesting all the points(or data points) and then tells TikZ there you go these are the points these are the colors and linetypes, deal with it. When harvesting it collects it within TeX macros. That's what is defined by the TeX capacity. Now you can say why not using my gazillionabyte RAM memory as matplotlib, matlab etc. does. But it is what it is. TeX is old.

First tool is to enlarge the main memory used by TeX. Related questions are How to expand TeX's "main memory size"? (pgfplots memory overload) and especially Increase TeX capacity, as non-root

Couple of indirect tricks can still be pulled off though. For example,

1. you can read the data table, use it and then let the data macro to \relax. For example,

   % \documentclass{....}
   %.....
   %\begin{document}
   %.....
   \pgfplotstableread{
   %Huge file
   }\mytable
   % Used in the drawing or typeset
   %....
   \addplot+[] table[...] {\mytable}
   %....
   %After the figure or table we overwrite the macro that holds the table
   \let\mytable\relax
   %....
   %\end{document}
   

   this still assumes that the table is small enough to be handled at one shot but two or more of them are not. Otherwise we are back to where we started.

2. You can read fewer variables by skipping every other 4-5 points if you have too much detail.

3. You can draw it in external programs and include the resulting plot by introducing to pgfplots. Section 4.3.7 in v1.10 manual explains this.