Ultrafast PDFLaTeX with precompiling

Question

I try to improve the time pdflatex needs to compile my book.

Really working example

book.tex

%&preamble
\begin{document}
Hi
\end{document}

and

preamble.tex

\documentclass{article}

I run the following commands on bash console:

$ pdflatex -ini -jobname="preamble" "&pdflatex preamble.tex\dump"
$ latexmk -pdf -pvc -e '$latex=q/latex %O -shell-escape %S/' book.tex

A window opens with a (nearly) live preview that will be updated after each change in the book.tex file. The PDF Viewer evince reloads automatic when the .pdf changed.

Open Questions

I had no success to compile with the preamble of my book in the preamble.tex file.

• How can I find out what I can precompile?
• Can I precompile \newcommand's, \usepackage's?
• Can I precompile a pure chapter without header?

Other approaches for speed up

(I do not understand how to combine these ideas for best result)

• pdflatex knows a -draftmode I measured 20% faster compilation with time pdflatex -draftmode 50pagetest.tex

Answer 1

"precompile" is probably a slightly confusing phrase to use as TeX is not a compiler but (mostly) a macro expansion language, but anyway...

In general you can dump most macro definitions and register assignments into a format. What you can't do is ship out pages. So in practice you can dump most LaTeX preambles.

Rather than having to edit the file so that it only works with the preloaded preamble format it is possible to leave the preamble as normal, but define the dumped format to skip the commands that were previously executed in the dump.

My truly ancient mylatex files on ctan do this or there is a newer version of that with additional features and better maintained: mylatexformat

The main thing you have to beware of (and which I suspect you are falling over) is if any of commands that you dump use \jobname (for example to open auxiliary files) then you have to ensure that the jobname when you dump the format is the same as the jobname when you produce the document. Also If the macros are assuming that files opened by commands in the preamble are still open when the document is processed then you will need to re-open them when you use your preloaded format loading the format file will re-set TeX's internal state with respect to its internal memory but it will not re-assign the file handles to the filesystem.

The first hit on searching for mylatex on this site shows an example discussing this in the context of tikz externalize.

TikZ's externalization and mylatex

---

I wrote the above back in 2012, If writing it now I would stress two other classes of things that can not be dumped.

• OpenType fonts (so affecting any xelatex or lualatex preamble that loads system fonts), and
• Lua state, so affecting any package that uses \directlua if used with luatex. For any particular Lua code you can usually \dump a macro definition that executes the code in \everyjob or \AtBeginDocument however this often requires substantial re-arrangement of the package code, so using mylatex with an unchanged latex document is often not possible when using lualatex.

Answer 2

For the benefit of people using a search engine arriving here looking for how to convert a never-changing but slow-loading preamble into a super-quick "precompiled" format.

The instructions here take you through a simple process, as follows. You need the mylatexformat package to make this work.

1. Split your preamble into "static" and "might change" stuff, with the static stuff first.

2. At the end of the static stuff add the command \endofdump

3. Run the following from the command line/terminal, replacing both instances of foo with the name of your document:

   pdftex -ini -jobname="foo" "&pdflatex" mylatexformat.ltx foo.tex
   

4. Insert the code %&foo at the top of your .tex file, replacing foo with the name of your file.

Your compiles will now happen much faster.

Note: if you have any "might change" preamble, you need to keep the \endofdump command, otherwise compilation will start at \begin{document} skipping all your preamble.

Answer 3

In this answer I will explain how does "precompiled preamble" works, how to use it in PDFLaTeX/XeLaTeX/LuaLaTeX, as well as some pitfalls.

Normally just follow the instruction suffices. However if something goes wrong, knowing what is going on under the hood helps you to diagnose the issue.

How to use mylatexformat? [basic instruction]

Say, you have a normal LaTeX document a.tex:

\documentclass{article}
\usepackage{amsmath}
\begin{document}
\begin{align*}
  1+2 &= 3
\end{align*}
\end{document}

Assume a.tex does not contain space, or special character such as %#{}~.

First you execute pdftex -ini -jobname=a "&pdflatex" mylatexformat.ltx a.tex. This will generate a.fmt. _{(the output .fmt file name is specified by -jobname, if not specified default to mylatexformat.fmt, nevertheless using -jobname the same as file name is less likely to cause errors)}

Then you execute pdftex "&a" a.tex. This will generate a.pdf as usual.

Remark on reading mylatexformat package documentation

• -ini and -initialize (as suggested in mylatexformat documentation) is the same thing. But -initialize doesn't work in TeXLive, so use -ini is safer.
• Its instruction on writing """abc.tex""" on the command-line is for Windows only, on Linux/Bash you need "\"abc.tex\"". (basically pass the file name wrapped in one layer of double quote to TeX)
• &a specifies that the format in file a.fmt should be used.
• If the first line of a.tex is %&a, then the 2nd command you don't need to explicitly specify "&a" on the command-line, anything such as pdftex a.tex or pdflatex a.tex will suffice. Nevertheless this method does not work with LuaTeX.

How to use mylatexformat? [advanced instruction]

Use XeLaTeX/LuaLaTeX instead of PDFLaTeX

Replace pdftex with xetex or luahbtex respectively. Furthermore you need to replace the preloaded format name &pdflatex with &xelatex or &lualatex.

You can tell that xelatex and lualatex executable are just xetex and luahbtex with xelatex.fmt and lualatex.fmt formats preloaded by seeing what it prints out at the beginning.

Furthermore note that first-line %&a (as written in package documentation) will not work in LuaTeX.

For example:

% a.tex
\documentclass{article}
\usepackage{amsmath}
\csname endofdump\endcsname
\usepackage{unicode-math}  % incompatible with precompiled preamble, see below
\begin{document}
\begin{align*}
  1+2 &= 3
\end{align*}
\end{document}

Then in the shell:

luahbtex -ini -jobname=a "&lualatex" mylatexformat.ltx a.tex
luahbtex "&a" a.tex

This only speed up by a small amount, in my experience for an almost-empty document it is 0.6s → 0.4s.

Do not precompile the whole preamble

If some part of the preamble either:

• changes frequently (so you don't want to include into the format), or
• relies on \jobname (for example \usepackage{currfile}, or pythontex), or
• is incompatible with TeX format, such as \usepackage{unicode-math} or \usepackage{fontspec}, _{(see "two other classes of things that can not be dumped." in David Carlisle's answer)}
  • For some reason for LuaLaTeX even tikz is incompatible.
  • I don't see any easy way to easily tell what package is compatible and what is not, you will just have to try including/excluding it.

then you must exclude it from the preamble.

To do that use \csname endofdump\endcsname (the spelling must be exactly like that) -- see mylatexformat documentation for more details. Using this instead of \endofdump makes sure the file is compilable (the line is simply ignored) if mylatexformat/precompiled preamble is not used (i.e. the file is compiled the normal way).

What is precompiled preamble/format?

Basically, it is just using TeX's INITeX feature to speed up processing of a fixed preamble part.

TeX supports generating formats that can be loaded later. Refer to TeX by Topic chapter 33 "TeX and the Outside World" for some details.

_{(Reading TeXbook might be a bit confusing, just keep in mind that typing anything on the command-line is equivalent to typing it on the ** first line in the TeX prompt.)}

In short:

A format file (usually with extension .fmt) is a compact dump of TEX’s internal structures. Loading a format file takes a considerably shorter time than would be needed for loading the font information and the macros that constitute the format.

So you can "define some commands, dump the internal state to a file", then when you "load" that file it's "as if" the commands have been defined similar to at the time how the format was dumped.

For an example:

% preamble.tex
\documentclass{article}
\usepackage{amsmath}
\dump
% a.tex
\begin{document}
\begin{align}
  1+2 &= 3
\end{align}
\end{document}

In the shell execute pdftex -ini "&pdflatex" preamble.tex, this will

• start with the existing pdflatex.fmt format _{(might be at /var/lib/texmf/web2c/pdftex/pdflatex.fmt depends on operating system etc.)}
• execute from file preamble.tex
• when \dump is executed generates file preamble.fmt.

The -ini is needed to enable the \dump command.

After that if pdftex "&preamble" a.tex is executed (this means "execute a.tex with preloaded format preamble.fmt") it will generate a.pdf with the correct content. _{(For an analogue, pdftex "&pdflatex" a.tex is equivalent to pdflatex a.tex)}

What does mylatexformat package do?

As you can see above, it's rather inconvenient to have to split the file into separate preamble.tex. So mylatexformat allow you to put them in the same file and it will automatically detect the split between the preamble and the main file.

How does mylatexformat work (regarding the command-line)?

There's 2 steps -- generating the format, and using the format.

In the first step, apart from -jobname, the command-line is:

pdftex -ini "&pdflatex" mylatexformat.ltx a.tex

Because of historical reasons, it's made such that (almost) anything put on the command-line is equivalent to typing it to the first line of ** prompt. So you can also run pdftex -ini then on ** prompt type in &pdflatex mylatexformat.ltx a.tex, both are equivalent.

Most of this is familiar. Recall that pdftex -ini "&pdflatex" mylatexformat.ltx will start with preloaded format pdflatex.fmt and execute content of file mylatexformat.ltx. However, the additional a.tex will be put on the input stream.

At the very end of mylatexformat.ltx, after redefining \begin etc. there's

%% Trick lookahead to allow mylatex.ltx and the document filename to be
%% given on the same command line. (initex &latex mylatex.ltx {abc.tex})
\expandafter\input\endinput%
\endinput

so effectively _{(some TeX programming knowledge is needed)} \input a.tex is executed. See also wipet's answer.

That's why it's useful to pass another pair of quotes in the command-line.

For other details of the implementation (how it separates the preamble from the document) read the documentation.

More speedup by unzipping the format file

Refer to https://tex.stackexchange.com/a/708544/250119.

This can save 0.1 to 0.2 seconds each compilation.