Compiling a 870 mb latex file

Question

I have a tikz file that is 870 mb and I want to compile it using some latex. How can I achieve this? I have tried increasing the amount of main memory to latex to the maximum and it still doesn't render. Currently I get the following output while compiling. I computer generated the file using this tool https://gephi.org using the tikz output export.

http://pastebin.com/4bD9r1Ld

I performed the fix in the answer below and now I get this error

http://pastebin.com/KMZDW8qZ

Answer 1

The error message says:

! TeX capacity exceeded, sorry [pool size=3141349].

The pool size can be raised, for testing it can be done on the command line using an environment variable:

$ pool_size=10000000 latex openordconceptnet4.tex

This can also be configured in texmf.cnf, see entry for pool_size.

However, it cannot be raised to infinity and there are lots of other memory limits. Some memories can be increase the same way as pool size. For other values the format must be regenerated to have an effect.

The real problems start when the memory limits cannot be increased further. Then it can be tried to divide the large file in smaller files and concat the output files later. Or the TeX code must be tried to make more efficient with the memory limits.

Answer 2

Based on the code you posted for a smaller example, the tool you are using generate redundant LaTeX code

\usepackage{tikz, tkz-graph}
\usepackage[active,tightpage]{preview}
\PreviewEnvironment{tikzpicture}
\setlength\PreviewBorder{5pt}

It is not clear to me what is the use of the last 3 lines. preview is useful to generate stand alone output (for example, to be used in the AucTeX preview mode of emacs).

The main code seems to be divided in two parts: one part where \node are defined, and the second part where \Edges are defined. Several optimizations are possible.

\begin{tikzpicture}
\node at (-2.6682776,7.9326984) [circle, line width=1, fill=COLOR0,  inner sep=0pt, minimum size = 14.3428574pt, label={[label distance=0] 315:Myriel}] (1) {};
\node at (-4.1808344,9.4046475) [circle, line width=1, fill=COLOR0,  inner sep=0pt, minimum size = 2pt, label={[label distance=0] 315:Napoleon}] (2) {};
...
\node at (2.3879364,1.7951599) [circle, line width=1, fill=COLOR3,  inner sep=0pt, minimum size = 10.2285728pt, label={[label distance=0] 315:Brujon}] (76) {};
\node at (7.1218353,4.9839259) [circle, line width=1, fill=COLOR8,  inner sep=0pt, minimum size = 10.2285728pt, label={[label distance=0] 315:MmeHucheloup}] (77) {};

In the example it seems that a lot of parameters of the nodes are exactly the same circle, line width=1, inner sep=0pt. Every there is a duplicate some words of memory are consumed without any reason. A more efficient is to collect the same values and declare them only one time. This can be done using \tikzset{<parameters>}

\begin{tikzpicture}
\tikzset{circle, line width=1, inner sep-0pt, label distance=0pt}
\node at (-2.6682776,7.9326984) [fill=COLOR0, minimum size = 14.3428574pt, label={315:Myriel}] (1) {};
\node at (-4.1808344,9.4046475) [fill=COLOR0, minimum size = 2pt, label={315:Napoleon}] (2) {};
...
\node at (2.3879364,1.7951599) [fill=COLOR3, minimum size = 10.2285728pt, label={315:Brujon}] (76) {};
\node at (7.1218353,4.9839259) [fill=COLOR8, minimum size = 10.2285728pt, label={315:MmeHucheloup}] (77) {};

For the \Edges parameters are unnecessarily swapped.

\tikzset{EdgeStyle/.style = {-, shorten >=1pt, >=stealth, bend right=10, line width=0.5, color=COLOR0}}
\Edge (2)(1)
\Edge (5)(1)
\Edge (6)(1)
\Edge (7)(1)
\Edge (8)(1)
\tikzset{EdgeStyle/.style = {-, shorten >=1pt, >=stealth, bend right=10, line width=1, color=COLOR0}}
\Edge (9)(1)
\tikzset{EdgeStyle/.style = {-, shorten >=1pt, >=stealth, bend right=10, line width=0.5, color=COLOR0}}
\Edge (10)(1)

First of all, there are many unnecessary duplication. Some parameters can get a value one once instead of repeating the same values over and over and wasting TeX memory words. Thus the cade can use \tikzset{EdgeStyle/.style = {<parameters>}} for the fixed parameters, and \tikzset{EdgeStyle/.append style} to modify the parameters that need to be modified for the following edges

\tikzset{EdgeStyle/.style = {-, shorten >=1pt, >=stealth, bend right=10, line width=0.5, color=COLOR0}}
\Edge (2)(1)
\Edge (5)(1)
\Edge (6)(1)
\Edge (7)(1)
\Edge (8)(1)
\tikzset{EdgeStyle/.append style = {line width=1, color=COLOR0}}
\Edge (9)(1)
\tikzset{EdgeStyle/.append style = {line width=0.5, color=COLOR0}}
\Edge (10)(1)

From the sample it seems that the only parameters that change are line widht and color. In the example there are countless cases where both parameters are used while only one of them needs to be changed, thus a further optimisation of the code above is

\Edge (8)(1)
\tikzset{EdgeStyle/.append style = {line width=1, color=COLOR0}}
\Edge (9)(1)
\tikzset{EdgeStyle/.append style = {line width=0.5}}
\Edge (10)(1)

Some of the above optimisations can be done by a simple search and replace mechanism (in the sample I have been able to save about 10% of memory words with them). The last optimisation can be done by some clever scripts. Also another optimisation would be to group together as many changes as possible (e.g., all \Edges with the same line width or color)

Hope it helps