increasing hard coded memory limits of latex

Question

I am drawing a 3d-surface plot with pgfplots, with input data from some simulation results.

As the shader-option faceted interp results in huge pdf-files, I wanted to switch to faceted and increase the number of data points... But at some point, I had problems with compiling with pdflatex due to the limited main memory size, so I switched to lualatex. Even with lualatex, I got some error like Tex capacitiy exceeded, save size = 100000. So I increased the save size, but there seems to be a hard coded limit in Tex at 250.000! (The compilation uses only around 570MiB of RAM at the maximum, so at this point I still have 30 GiB of free RAM...)

My code looks like this:

\documentclass[class=scrbook, 11pt,convert]{standalone}
\usepackage{pgfplots}
\usepackage{pgfplotstable}
\pgfplotsset{compat=newest}
\pgfplotsset{colormap={ownbluered}{[1cm] rgb255(0cm)=(0,0,255); rgb255(30cm)=(0,255,255);  rgb255(60cm)=(0,255,128); rgb255(80cm)=(0,255,0); rgb255(90cm)=(192,255,0); 
rgb255(115cm)=(255,255,0); rgb255(140cm)=(255,128,0); rgb255(155cm)=(255,0,0); rgb255(160cm)=(230,0,0)}
}

\pgfplotsset{
    nonlinear colormap trafo/.code 2 args={
        \pgfkeysalso{
            y coord inv trafo/.code={%
                \pgfmathparse{##1 < 30 ? (##1-30)*(-#2)/30 : (##1-30)*#1/30}%
            },
        }%
    },
    nonlinear colormap around 30/.code 2 args={
        \pgfkeysalso{
            colorbar style={
                nonlinear colormap trafo={#1}{#2},
            },
        }%
    },
}

\begin{document}
\def\infile{linear_directivity_1deg.txt}
\def\filenamedB{\infile_dB.txt}
\def\directivity{6.3}% in dB!
\def\directivitymin{-26}% in dB!
\edef\threedB{45}%
\pgfmathparse{\directivitymin+30}
\edef\minimum{\pgfmathresult}%
\pgfmathparse{\directivitymin/2+30}
\edef\halfminimum{\pgfmathresult}%

\begin{tikzpicture}
\begin{axis}[height=10cm, colorbar, colormap name=ownbluered, z buffer=sort, view={135}{25}, mesh/ordering=y varies, axis equal image,  nonlinear colormap around 30={\directivity}{\directivitymin},
             colorbar style={
                             ytick={\minimum,\halfminimum,30,\threedB,60},
                             %yticklabel={\pgfmathparse{int(\tick)}\pgfmathresult\,dBi},
                             %yticklabel={\SI{\pgfmathprintnumber[precision=1]{\tick}}{\dBi}},
                             yticklabel={\pgfmathprintnumber[precision=1]{\tick}\,dBi},
                             %y unit = dBi,
                             yticklabel style={align=right, text width = 4em,},    
                             yshift=-1cm,    
                             height=5cm,                     
                             },
             axis lines=none, enlargelimits=false,]
            \addplot3[mesh/rows=181, surf, point meta=\thisrow{dir},line width = 0.025pt] table [x = x,y = y,z = z,]   {\filenamedB};
\end{axis}
\end{tikzpicture}
\end{document}

The read .txt-file can be found here.

An example plot with not so many points (only 5 degree steps instead of 1 degree):

Is there any possibility to improve my code, so that less save size is needed? Or could I compile parts of the software and change the built-in limits for the limits in tex.mf?

Answer 1

This answer addresses your sub-question "Is there any possibility to improve my code, so that less save size is needed?"

Fortunately, the answer is yes (provided you have at least pgfplots 1.12 and you continue to use lualatex).

I expected the code to make use of the new lua backend, a partial reimplementation of some expensive functions in Lua. If it would have done that, it should have avoided the TeX memory limitations. However, the .log file claims that

Package pgfplots info on input line 50: Using 'lua backend=false' for plot 0 (ty
pe 'mesh'): point meta choice does not support LUA.

which lead me to work around the current point meta choice. I remember that the lua backend is disabled if point meta contains a TeX macro (as these cannot easily be migrated to the Lua script base). As soon as I used point meta=explicit combined with \addplot .... table[..., meta=dir] instead of point meta=\thisrow{dir}, the memory problems were solved and I could compile your example with lualatex. Note that both formulations of the point meta are equivalent for your case, but your formulation is easier to read and to remember.

Here is the complete example with result:

\documentclass[class=scrbook, 11pt,convert]{standalone}
\usepackage{pgfplots}
\usepackage{pgfplotstable}
\pgfplotsset{compat=newest}
\pgfplotsset{colormap={ownbluered}{[1cm] rgb255(0cm)=(0,0,255); rgb255(30cm)=(0,255,255);  rgb255(60cm)=(0,255,128); rgb255(80cm)=(0,255,0); rgb255(90cm)=(192,255,0); 
rgb255(115cm)=(255,255,0); rgb255(140cm)=(255,128,0); rgb255(155cm)=(255,0,0); rgb255(160cm)=(230,0,0)}
}

\pgfplotsset{
    nonlinear colormap trafo/.code 2 args={
        \pgfkeysalso{
            y coord inv trafo/.code={%
                \pgfmathparse{##1 < 30 ? (##1-30)*(-#2)/30 : (##1-30)*#1/30}%
            },
        }%
    },
    nonlinear colormap around 30/.code 2 args={
        \pgfkeysalso{
            colorbar style={
                nonlinear colormap trafo={#1}{#2},
            },
        }%
    },
}

\begin{document}
\def\infile{linear_directivity_1deg.txt}
\def\filenamedB{\infile_dB.txt}
\def\directivity{6.3}% in dB!
\def\directivitymin{-26}% in dB!
\edef\threedB{45}%
\pgfmathparse{\directivitymin+30}
\edef\minimum{\pgfmathresult}%
\pgfmathparse{\directivitymin/2+30}
\edef\halfminimum{\pgfmathresult}%

\begin{tikzpicture}
\begin{axis}[height=10cm, colorbar, colormap name=ownbluered, z buffer=sort, view={135}{25}, mesh/ordering=y varies, axis equal image,  nonlinear colormap around 30={\directivity}{\directivitymin},
             colorbar style={
                             ytick={\minimum,\halfminimum,30,\threedB,60},
                             %yticklabel={\pgfmathparse{int(\tick)}\pgfmathresult\,dBi},
                             %yticklabel={\SI{\pgfmathprintnumber[precision=1]{\tick}}{\dBi}},
                             yticklabel={\pgfmathprintnumber[precision=1]{\tick}\,dBi},
                             %y unit = dBi,
                             yticklabel style={align=right, text width = 4em,},    
                             yshift=-1cm,    
                             height=5cm,                     
                             },
             axis lines=none, enlargelimits=false,]
            \addplot3[mesh/rows=181, surf, shader=interp, point meta=explicit,line width = 0.025pt] table [x = x,y = y,z = z,meta=dir]   {\filenamedB};
\end{axis}
\end{tikzpicture}
\end{document}

Note that the resulting .pdf file is 2.5 MB big due to the many patches in your example, and it takes a very long while to render.

If you want to reduce its size, you could make use of shader=interp; the resulting file size reduces to 1.1 MB and the rendering time is considerably faster. In this case, the figure looks as follows:

Some remarks of sorts "outlook": Perhaps the lua backend will become smart enough to understand your initial code. And perhaps it will also become smart enough to read input files without requiring the expensive TeX routines. Both is planned and would reduce the insanely long compilation time by orders of magnitudes...

EDIT

the case with shader=faceted resulted in the following memory usage on my system

Here is how much of LuaTeX's memory you used:
 25089 strings out of 494748
 100000,109195293 words of node,token memory allocated
 3090 words of node memory still in use:
   82 hlist, 6 vlist, 1 rule, 10 math, 244 glue, 5 kern, 5 penalty, 32 glyph, 72
 glue_spec, 1 write, 5 local_par, 204 pdf_literal, 2 pdf_refxform, 10 pdf_colors
tack nodes
   avail lists: 2:46,3:211,4:287,5:5,6:47,7:1,9:93,10:3
 27874 multiletter control sequences out of 65536+600000
 26 fonts using 1033763 bytes
 79i,10n,103p,718b,2731s stack positions out of 5000i,500n,10000p,27176575b,100000s
<</home/ludewich/tl2014/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb><</

I believe the 2731 is the save size (limit 100000)