latexmk with precompiled preamble

Question

Problem Description

I am trying to setup latexmk using a .latexmkrc configuration file for "precompiling" the preamble of my documents, and reusing that during reruns.

According to a comment by David Carlisle, using precompliation doesn't really save enough time to be worth the bother [he] wrote mylatex.ltx back in 1988.

However, I have a lot of externalized graphics in my documents, and references inside pgfplots, so the loading times grow quite fast in my case, and I want to explore how much precompilation could speed up the entire process.

Requirements

My current requirements are:

1. preamble and document must be able to reside inside the same document, for this question let it be mwe.tex.
2. All compilation must be done from one single call to latexmk.

Literature Review

In particular, I have looked into the following questions and answers:

• https://tex.stackexchange.com/a/37730 is a bit dated, but the general principle can be taken to more recent versions of LaTeX. The main problem with this proposed solution is, that as stated, [y]ou'll need to run pdflatex -ini ... manually the first time, to create the .fmt file, which contradicts my second requirement.

• https://tex.stackexchange.com/a/269052 shows how the preamble my be dumped from one single file, which is

  pdflatex -ini \&pdflatex mylatex.ltx mwe.tex
  

  In particular, this is helpful in view of requirement 1.

• After own research, I have found that a system call

  pdflatex -ini \&pdflatex mylatexformat.ltx mwe.tex
  

  would be the more recent way to do this.

My custom minimal non-working example

Using the previous information and shortcomings in view of my requirements, I intend so sole this problem in the following steps:

1. Modify the $pdflatex variable in .latexmkrc to require a precompiled preamble mwe.fmt file.
2. Use add_cus_dep('tex', 'fmt', 1, 'precompilepreamble'); to add a dependency.
3. Define the perl subroutine precompilepreamble to create the mwe.fmt file if necessary.

I expect the program to behave in the following way:

1. Call latexmk, which loads my custom .latexmkrc configuration.
2. First run of pdflatex fails, and will log the missing required file mwe.fmt.
3. latexmk decides an additional run is needed, and will look up the dependencyies for .fmt.
4. Accordingly, latexmk will run the custom rule I added previously to generate mwe.fmt.
5. Afterwards, latexmk will run my modified $pdflatex again and continue sucessfully.

Attached you find the required files:

mwe.tex

\documentclass{scrbook}
\usepackage{pgfplots}
\usepackage{amsmath}
\usepackage{siunitx}
\usepackage{lipsum}
\begin{document}
\title{Shakespeare's Externalize}
\subtitle{To externalize or not to externalize?}
\subject{Case Studies}
\date{\today}
\maketitle{}
\lipsum[1-20]
\end{document}

.latexmkrc

# ensure_path( 'TEXINPUTS', './src/' );
# ensure_path( 'TEXINPUTS', '../src/' );
Set the program used to generate the PDF
1: pdflatex
2: postscript conversion, don't use this
3: dvi conversion, don't use this
4: lualatex
5: xelatex
$pdf_mode = 1;
Add custom dependency.
latexmk checks whether a file with ending as given in the 2nd
argument exists ('toextension'). If yes, check if file with
ending as in first argument ('fromextension') exists. If yes,
run subroutine as given in fourth argument.
Third argument is whether file MUST exist. If 0, no action taken.
add_cus_dep('tex', 'fmt', 1, "compilepreamble");
show custom dependencies for debugging
show_cus_dep();
parse missing files
push @file_not_found, '^I can't find the format file `(.*)'!';
sub compilepreamble {
    my ( $base) = @_;
    # just output a simple debug string to console
    print ("**** compilepreamble: ( $base ) ****\n");
# create the fmt file
$ret = system("pdflatex",
    "-jobname", "$base",
    "-ini",
    "&pdflatex" ,
    "mylatexformat.ltx",
    $base
     );

# if there were errors, return the error codes
return $ret;

}
modify $pdflatex to use custom format
$pdflatex = 'pdflatex  -fmt mwe %B %O %S';

Observed Behaviour

Instead of pdflatex failing, it tries to run mktexfmt:

$  latexmk -g
Custom dependency list:
  tex fmt 1 compilepreamble
Rc files read:
  .latexmkrc
Latexmk: This is Latexmk, John Collins, 7 Jan. 2023. Version 4.79.
Force everything to be remade.
Latexmk: applying rule 'pdflatex'...
Rule 'pdflatex':  Reasons for rerun
Category 'other':
  Rerun of 'pdflatex' forced or previously required

Run number 1 of rule 'pdflatex'

Running 'pdflatex -fmt mwe   -recorder  "mwe.tex"'
This is pdfTeX, Version 3.141592653-2.6-1.40.24 (TeX Live 2022) (preloaded format=mwe)
 restricted \write18 enabled.
kpathsea: Running mktexfmt mwe.fmt
mktexfmt: mktexfmt is using the following fmtutil.cnf files (in precedence order):
...
I can't find the format file `mwe.fmt'!
Latexmk: fls file doesn't appear to have been made.
Latexmk: Getting log file 'mwe.log'
Latexmk: Couldn't read log file 'mwe.log':
  No such file or directory

Question

1. How can I prevent latexmk, or rather, pdflatex from trying to run mktexfmt?
2. The line I can't find the format file `mwe.fmt'! is only output to console and no mwe.log is generated, which makes reruns impossible, and does not detect my custom dependency. How can I change this behaviour?
3. I could inject a custom subroutine into $pdflatex, which would then check if the format file exists, and possibly create it if not. This could even be combined with a hashing process, to ensure only recompilation if needed and ... oh my, I notice I am copying the functionality of latexmk. So, is there a way to do this in latexmk in the first place?

I'm also looking forward to hearing other suggestions.

Answer 1

1. How can I prevent latexmk, or rather, pdflatex from trying to run mktexfmt?

$ pdftex --no-mktex=fmt --fmt mwe file
This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) (preloaded format=mwe)
 restricted \write18 enabled.
I can't find the format file `mwe.fmt'!

shows a run that does not trigger mktexfmt

2. The line I can't find the format file `mwe.fmt'! is only output to console and no mwe.log is generated, which makes reruns impossible, and does not detect my custom dependency. How can I change this behaviour?

I don't think that is changeable

3. I could inject a custom subroutine into $pdflatex, which would then check if the format file exists, and possibly create it if not.

You could, but (without the option in (1.) above) pdftex is already doing that check and calling mktexfmt in the case the file does not exist so a simpler approach is probably to configure mktexfmt so that it builds mwe.fmt when requested. (texdoc mktexfmt has some information on how to set up the fmtutil config)

Answer 2

The excellent answer by David Carlisle has lead me to discovering a way how my goals can be achieved using pure latexmk, so I am going to share my solution and see how it might be able to be improved.

Solution

As noted by David Carlisle,

  $ pdftex --no-mktex=fmt --fmt mwe file 
  This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) (preloaded format=mwe)
    restricted \write18 enabled. 
  I can't find the format file `mwe.fmt'!

shows a run that does not trigger mktexfmt

and it seems I was not able to read the document right to put in the option --no-mktex=fmt.

As noted by David Carlisle as well, this run does not log to a respective log file, so no rerun for latexmk is triggered.

My solution revolves around chenging the output stream from stdout to a logfile, which can be entirely achieved in .latexmkrc:

mwe.tex

\documentclass{scrbook}
\usepackage{pgfplots}
\usepackage{amsmath}
\usepackage{siunitx}
\usepackage{lipsum}
\usepackage{geometry}
\begin{document}
\title{Shakespeare's Externalize}
\subtitle{To externalize or not to externalize?}
\subject{Case Studies}
\date{\today}
\maketitle{}
\lipsum[1-30]
\end{document}

.latexmkrc

# Set the program used to generate the PDF
# 1: pdflatex
# 2: postscript conversion, don't use this
# 3: dvi conversion, don't use this
# 4: lualatex
# 5: xelatex
$pdf_mode = 1;
Add custom dependency.
latexmk checks whether a file with ending as given in the 2nd
argument exists ('toextension'). If yes, check if file with
ending as in first argument ('fromextension') exists. If yes,
run subroutine as given in fourth argument.
Third argument is whether file MUST exist. If 0, no action taken.
add_cus_dep('tex', 'fmt', 1, "compilepreamble");
show custom dependencies for debugging
show_cus_dep();
parse missing files
push @file_not_found, '^I can't find the format file `(.*)'!';
modify all tex engines to run with my customized instruction set
in particular:
- suppress mktexfmt using --no-mktexfmt=fmt
- request use of custom format using --fmt=%B
- call additional options provided to latexmk using %O
- call the source file %S
set_tex_cmds("--shell-escape  --interaction=batchmode  --file-line-error --synctex=1 --no-mktex=fmt --fmt=%B  %O %S > %B.log");
custom sub handling the loading of the preamble and dumping it into $base.fmt
sub compilepreamble {
    my ( $base) = @_;
    # just output a simple debug string to console
    print ("**** compilepreamble: ( $base ) ****\n");
# create the fmt file
$ret = system("pdflatex",
    "-jobname", "$base",
    "-ini",
    "&pdflatex" ,
    "mylatexformat.ltx",
    $base
     );

# if there were errors, return the error codes
return $ret;

}

Comments

1. My solution seems create the fmt on first run, and reuse it in subsequent runs.
2. My solution does trigger a rebuild of the mwe.fmt file each time one calls latexmk after the file mwe.tex has changed. It does not make a distinction, whether the change has been made in the document or the preamble. This is a bit of a shortcoming, however, in my usecase it can be neglected, as

• most content resides in their own files and is placed using \input{}
• the relevant savings in compilation time will occurr during reruns for externalization of graphics.

If anyone wants to suggest improvements, I'll gladly hear them.