6

Introduction

Hello, I am facing very long compile times of more than 14 minutes for a ~130 page document, including around 40 PGF-Plots/TikZ-Images. Due to LaTeX's memory limits and (in fact) for speed up reasons, I decided to use the external-package.

Then I noticed, that each image takes up to 25 seconds to render (looking into my pdflatex.log from Miktex 2.9. Edit: When compiling the same TikZ-images using TikzEdt, it takes only a split second from beginning to the end of compilation and the PDF being there.

Today, I created a MWE for demonstrating this effect, which seems to increase with the number of other packages loaded:

MWE (find it also here: Github)

\documentclass[]{article}

\usepackage{pgfplots}
\usepackage{graphicx} 

% Comment out the next three lines for faster compilation time
\usepgfplotslibrary{external}
\tikzexternalize
\tikzsetexternalprefix{figtemp/}

\begin{document}

\section{First}
    \begin{tikzpicture}
        \tikz \draw[thick,rounded corners=8pt]
        (0,0) -- (0,2) -- (1,3.25) -- (2,2) -- (2,0) -- (0,2) -- (2,2) -- (0,0) -- (2,0);
    \end{tikzpicture}

\section{Second}
    \begin{tikzpicture}
        \draw (-1.5,0) -- (1.5,0);
        \draw (0,-1.5) -- (0,1.5);
    \end{tikzpicture}

\section{Third}
    \begin{tikzpicture}
        \draw (0,0) arc (0:315:1.75cm and 1cm);
    \end{tikzpicture}

\end{document}

MWE Output

This example gives me the two respective pdflatex.log-Files, depending on whether externalization is enabled or disabled:

Externalization enabled

Total time: ~14.0 s

2019-03-05 13:21:31,240+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:21:31,246+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:31,246+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:31,246+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:33,607+0100 INFO  pdflatex - executing write18 shell command: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname "figtemp/TeX-TikZ-slow-MWE-figure0" "\def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}"
2019-03-05 13:21:33,885+0100 INFO  pdflatex - starting with command line: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname figtemp/TeX-TikZ-slow-MWE-figure0 \def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}
2019-03-05 13:21:33,891+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:33,891+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:33,891+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:36,177+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:36,326+0100 INFO  pdflatex - executing write18 shell command: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname "figtemp/TeX-TikZ-slow-MWE-figure1" "\def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}"
2019-03-05 13:21:36,614+0100 INFO  pdflatex - starting with command line: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname figtemp/TeX-TikZ-slow-MWE-figure1 \def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}
2019-03-05 13:21:36,619+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:36,620+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:36,620+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:39,167+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:39,311+0100 INFO  pdflatex - executing write18 shell command: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname "figtemp/TeX-TikZ-slow-MWE-figure2" "\def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}"
2019-03-05 13:21:39,583+0100 INFO  pdflatex - starting with command line: pdflatex -enable-write18 -halt-on-error -interaction=batchmode -jobname figtemp/TeX-TikZ-slow-MWE-figure2 \def\tikzexternalrealjob{TeX-TikZ-slow-MWE}\input{TeX-TikZ-slow-MWE}
2019-03-05 13:21:39,592+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:39,592+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:39,592+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:41,817+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:42,029+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:21:42,485+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:21:42,489+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:21:42,489+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:21:42,489+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:21:45,223+0100 INFO  pdflatex - finishing with exit code 0

second run, using the externalized images

Total time: ~2.4 s

2019-03-05 14:15:03,628+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 14:15:03,629+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 14:15:03,629+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 14:15:03,629+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 14:15:06,080+0100 INFO  pdflatex - finishing with exit code 0

Externalization disabled

Total time: ~5.1 s

2019-03-05 13:23:19,734+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:23:19,742+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:23:19,742+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:23:19,742+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:23:22,045+0100 INFO  pdflatex - finishing with exit code 0
2019-03-05 13:23:22,476+0100 INFO  pdflatex - starting with command line: pdflatex.exe -synctex=1 -interaction=nonstopmode -enable-write18 TeX-TikZ-slow-MWE.tex
2019-03-05 13:23:22,485+0100 INFO  pdflatex - allowing known shell commands
2019-03-05 13:23:22,485+0100 INFO  pdflatex - enabling input (output) from (to) processes
2019-03-05 13:23:22,485+0100 INFO  pdflatex - allowing all shell commands
2019-03-05 13:23:24,842+0100 INFO  pdflatex - finishing with exit code 0

Conclusion

Externalizing my three (very simple) TikZ-Images blows up compilation time almost by a factor of 3 in the first run.

When using a tool like TikzEdt, all my TikZ-images compile in a split second.

What can I do, to speed this up?

Edit:

I found out that TikzEdt uses a precompiled .fmt file, what makes it so fast. As I've got no experience in precompiling, I could use some help for this and how to pass it to the tikzexternalize calls of pdflatex. In this question someone is tryin the same, but I can't seem to organize my preamble to work as desired and meanwhile speed up the externalized image compilation.

DPF
  • 171
  • It actually makes sense. You are doing much more processing when externalizing the pictures, so there is some overhead. However your example leads to a wrong conclusion because the pictures are far too simple, so the overhead of externalizing compensates for the time it takes to actually draw the picture. Try with something larger and your conclusion will change. – Phelype Oleinik Mar 05 '19 at 12:58
  • When using a tool like TikzEdt, all my TikZ-Images compile in a split second - which allows for a very fast feedback on changes I made in the code.

    I'd rather suspect some unnecessary initalization for the externalized pdflatex call.

     – DPF Mar 05 '19 at 13:04
  • Isn't the point of externalization to be faster in later runs,when the pictures did not change? If the first run (or after changes) it has to do strictly more work than without externalization – sheß Mar 05 '19 at 13:05
  • Actually, this was my intention, yes - but when I had to recompile everything, I found the total time inacceptable and started my research on this. Now I need help from some TeX-Pros, who know of the internals of TikZ, pdflatex and so on. – DPF Mar 05 '19 at 13:07
  • 1
    I might be off, but you call \tikzexternalize \usepgfplotslibrary{external} in the wrong order. On my PC this actually gives me an Undefined Control Sequence. Also, again. tikzexternalize is not expected to give any speed improvements the when it actually re-draws the tikz pictures. So it would be good if you could tell us how long it takes if you compile it a second time without changing the pictures. If this is the same, this would suggest that tikzexternalize fails to recognize that the image has not changed – sheß Mar 05 '19 at 13:11
  • You're right. This happened when I was tidying up the code for this question. I also added the .log for the second run. It is faster, but still it doesn't explain the huge amount of time. – DPF Mar 05 '19 at 13:20
  • BTW compiling your file takes me 3.0 seconds the first time and 1.0 seconds on later runs, and commenting out those lines also gives me similar 1.0 seconds for running time. It may just be my laptop being faster, but the fact that you're able to get fast compile times with TikzEdt suggests otherwise. So your real question may be what's different between the respective invocations... – ShreevatsaR Mar 05 '19 at 13:30
  • I tried to analyze the behaviour of the called pdflatex instance using Process Monitor. I had quite some file activity to all kinds of LaTeX-fonts and styles - seemingly consuming a lot of the time. – DPF Mar 05 '19 at 13:36
  • Hmm my previous numbers were with TeX Live; I can kind of reproduce the behaviour you see with MikTeX (8 seconds, 2 seconds and 2 seconds respectively). – ShreevatsaR Mar 05 '19 at 13:45
  • What you describe is still consistent with overhead in the externalization consuming a lot of time. That's just a shot in the dark, but it might help to precompile the https://tex.stackexchange.com/questions/16734/precompiled-preamble-with-tikz-externalize?lq=1 – sheß Mar 05 '19 at 14:06
  • Just out of curiosity. Why do you care? In principle you will have to compile each figure only once ever. So this delay will not matter at all for any further compilations of the document – sheß Mar 05 '19 at 14:07
  • Tikzexternalize will never be faster than standalone and \includegraphics. I supposed it depends on whether you editor can handle more than one LaTeX file at a time. – John Kormylo Mar 05 '19 at 15:57
  • 1
    Can you look into what TeX distribution TikzEdt (which is so much faster) is using? Does it leave any traces in any log files? Does it also use the same MikTeX distribution? Can you update your question with exactly which version of which operating system, and which version of MikTeX you're using? – ShreevatsaR Mar 05 '19 at 16:35
  • TikzEdt uses my system-wide MikTeX installation. I just found out, that it uses a precompiled preamble. This seems to speed up the rest immensely. Regarding the question @sheß linked to, I can't seem to find out a good way of passing a suitable precompiled praeamble to tikzexternalize... – DPF Mar 05 '19 at 18:52
  • 1
    I think not many people would notice that your current question is how to create a preamble. Those who've seen this question already may not open it again, and even for someone who opens it, the actual question is at the very end. Your best bet may to be create a new question, focusing solely on how to create a precompiled preamble. It would have to be a very different question from the current one, going into a lot more detail on what exactly you've tried and what problems you've encountered with it. (BTW, sorry to hear about your experience; a document taking 14 minutes sounds horrible!) – ShreevatsaR Mar 06 '19 at 22:59

1 Answers1

1

\tikzexternalize will always take longer than compiling without externalization on the first run. It needs to create the figures just like it has to do w/o tikzexternalize, and on top of the usual work, it has to export them to files. This includes overhead that is added by calling new instances of pdflatex and interacting with the OS separately for each picture.

The performance gain is best illustrated with a slightly more intensive TikZ example, as suggested by @Phelype Oleinik:

\documentclass{article}
\usepackage{pgfplots}
\usepackage{graphicx} 
\usepgfplotslibrary{external}
\tikzexternalize
\tikzsetexternalprefix{figtemp/}
\begin{document}
\section{First}
        \begin{tikzpicture}
                \foreach \i in {1,...,10000} {
                    \draw[thin,rounded corners=8pt] ({\i*0.05},0) -- ({\i*0.05},2) -- ({1+\i*0.05},3.25) -- ({2+\i*0.05},2) -- ({2+\i*0.05},0) -- ({\i*0.05},2) -- ({2+\i*0.05},2) -- ({\i*0.05},0) -- ({2+\i*0.05},0);
                }
        \end{tikzpicture}
\end{document}

The following relationships for run times should be expected:

[First run with \tikzexternalize] > [Run without \tikzexternalize] >>> [Later runs with \tikzexternalize].

Edit: To get a feeling for the additional work the first tikzexternalize run has to do, compared to the plain run w/o tikzexternalized, you can study the log files in the figtemp folder.

As a very crude way of getting a benchmark for how long it should take, you can see how fast your computer is in creating an empty tikz image in a single pdflatex run:

\documentclass{article}
%your preable as in the main document
\begin{document}
\section{First}
        \begin{tikzpicture}
            \coordinate (aa) at (0,0)
        \end{tikzpicture}
\end{document}

As this is (very) roughly equivalent to the extra work tikzexternalize has to do for each picture during the first run. So the first run should overall take:

t0 + n t1 seconds, where t0 is the time it takes to compile the document without tikzexternalize and t1 is the time the TikZ picture above takes to compile.

sheß
  • 3,622
  • 1
    In general, I agree with your opinion about the actual use-case for tikzexternalize. However, I suspect that somethin else is hindering each shell escaped pdflatex instance unnecessarily - as the manual, separate compilation of my actual images is up x20 faster than while being externalized automatically. – DPF Mar 05 '19 at 13:29
  • Mh, it is hard to infer this from what you post. And I do not understand against what you are comparing this. To get a feeling for what's going on you could start looking at the log files in the figtemp/ folder. – sheß Mar 05 '19 at 13:35
  • The x20 speedup when using TikzEdt is compared with the "per image" time in my actual document (~25 s) – DPF Mar 05 '19 at 13:38
  • Do you mean "per image" as is "compiling each image in a separate document" or as in "without tikzexternalize"? Because the latter would be the wrong comparison. The first run should be expected to take roughly t0 + n t1, where t0 is the time it takes to compile it without tikzexternalize, n is the number of images and t1 is the time it takes to compare the empty TikZ picture from my edit above – sheß Mar 05 '19 at 13:43
  • I mean the separate document, as TikzEdt does for every recomilation. BTW: your empty image takes on my laptop ~ 1.9 s. – DPF Mar 05 '19 at 13:44
  • I don't know TikzEdt, but 1.9s*3+5.1s = 10.8s, which is not so far from your 14s. As I said, this is just a very rough proxy, and it is likely that each image in you document requires more packages to be loaded, etc, than my empty picture above. So I'd still say there's nothing surprising about the behavior you describe. You might squeeze time a little by reducing the number of packages you load etc. but overall I think the pattern will persist – sheß Mar 05 '19 at 14:03
  • 1
    Sorry I couldn't help you. Good luck – sheß Mar 05 '19 at 14:03