162

I've got a latex project that looks like this:

project/
   |-- main.tex
   |-- main.bib
   |-- preamble.tex
   |-- preamble.fmt
   |-- makefile

The preamble (preamble.tex) is being precompiled into preamble.fmt. main.bib is generated using the file contents environment in main.tex.

This is what is in my makefile

TEX = pdflatex -shell-escape -interaction=nonstopmode -file-line-error
PRE =  $(TEX) -ini -job-name="preamble" "&pdflatex preamble.tex\dump"
BIB = bibtex

.PHONY: all view

all : main.pdf

view :
    open main.pdf

main.pdf : main.tex preamble.fmt main.bbl main.blg
    $(TEX) main.tex

main.bbl main.blg : main.bib main.aux
    $(BIB) main

main.aux : main.tex
    $(TEX) main.tex

main.bib : main.tex
    $(TEX) main.tex

preamble.fmt : preamble.tex
    $(PRE) preamble.tex

The problem is here that bibtex relies on main.aux to be generated, and main.aux is regenerated every pdftex run. This leads to regeneration of the bibtex files every run, which causes makefile to run $(TEX) main.tex a second time (it thinks the .bbl and .blg files are changed, because it looks at their edit time).

So, basically every time I call make all latex is compiled twice, even if there are no changes to references made anywhere in the document (thus making this unnecessary).

Is there a way I can tell make that it only compiles twice if there is an actual change to the .aux and the .bib files. Perhaps by checking of md5 sums?

I'm kind of new to the whole makefile thing, so I thought I'd ask here. It could be off-topic, but I thought the latex gurus that reside on this site might have an answer.

lockstep
  • 250,273
romeovs
  • 9,102
  • 2
    Have you considered rubber, latexmk or similar tools? Also, why are you building a format? – Joseph Wright Jan 11 '12 at 11:13
  • I only saw after posting the question that there is the {makefile} tag, which I'm now scanning for useful code. I'll leave this question up for now. – romeovs Jan 11 '12 at 11:14
  • 2
    @JosephWright The format is nice because I only need to compile the preamble once, which save me a lot of compiling time. See for instance this answer I once posted. I'm looking into these alternatives now, but I'm integrating some other stuff too (like c++ code that generates data which I input into the tex file) and I don't know if that can be easily done using latexmk or rubber. – romeovs Jan 11 '12 at 11:17
  • 1
    Don't add main.aux to the dependencies of $(BIB) (I don't know if its the right terminology). – egreg Jan 11 '12 at 13:26
  • @egreg this won;t do, because bibtex does need the main.aux file to be present. Either way, I still would need to compile main.tex before bibtex is run to see if main.bib (which is generated in the main.tex run) has changed. The problem is in how make checks if a file has changed. Perhaps this question should be migrated to the Stack Overflow site? – romeovs Jan 11 '12 at 14:24
  • If you can offer some sample code for this case, I could try and see how scons copes with it. – Faheem Mitha Jan 13 '12 at 12:38
  • See also http://stackoverflow.com/questions/1240037/recommended-build-system-for-latex – koppor Sep 24 '15 at 09:03

4 Answers4

187

latexmk is the answer you are looking for.

LaTeX is notoriously difficult to "get right" using a Makefile, because it might take multiple compiler passes - updating e.g. .aux files - to get the finished results. Getting this right in a general Makefile (as opposed to one tailored to a specific document) is very hard, which is why there are pre-made solutions. Of these, latexmk comes included with your average LaTeX distribution, which is why I consider it first choice.

The trick is to provide a Makefile rule for every custom step x-to-TeX (or x-to-PDF or whatever) you might have, and having latexmk figure out all the LaTeX-related stuff while relying on the Makefile for the rest (via -use-make).

# You want latexmk to *always* run, because make does not have all the info.
# Also, include non-file targets in .PHONY so they are run regardless of any
# file of the given name existing.
.PHONY: MyDoc.pdf all clean

# The first rule in a Makefile is the one executed by default ("make"). It
# should always be the "all" rule, so that "make" and "make all" are identical.
all: MyDoc.pdf

# CUSTOM BUILD RULES

# In case you didn't know, '$@' is a variable holding the name of the target,
# and '$<' is a variable holding the (first) dependency of a rule.
# "raw2tex" and "dat2tex" are just placeholders for whatever custom steps
# you might have.

%.tex: %.raw
        ./raw2tex $< > $@

%.tex: %.dat
        ./dat2tex $< > $@

# MAIN LATEXMK RULE

# -pdf tells latexmk to generate PDF directly (instead of DVI).
# -pdflatex="" tells latexmk to call a specific backend with specific options.
# -use-make tells latexmk to call make for generating missing files.

# -interaction=nonstopmode keeps the pdflatex backend from stopping at a
# missing file reference and interactively asking you for an alternative.

MyDoc.pdf: MyDoc.tex
        latexmk -pdf -pdflatex="pdflatex -interaction=nonstopmode" -use-make MyDoc.tex

clean:
        latexmk -CA

This setup works flawlessly for anything referenced via \include.

However, \include might not be appropriate in every case. For one, it is not nestable (i.e. an \included file may not \include another). It also adds an automatic \clearpage to your document, i.e. \included content starts a new page. It also has advantages, like resulting in shorter re-build times if contents are modified, but sometimes you need nesting, or the referenced file's contents should be embedded in a page.

You need \input for this.

Sadly, \input breaks the build. If pdflatex encounters a missing \input file, it generates an error (instead of a warning like with \include), and stops compiling. Yes, latexmk will generate the file and re-start pdflatex, but this is inefficient, and breaks completely if you have multiple such file references, because eventually the compile will end with a "too many re-runs" message.

John Collins' answer to a question by me regarding this problem provides a workaround:

\newcommand\inputfile[1]{%
    \InputIfFileExists{#1}{}{\typeout{No file #1.}}%
}

This macro generates a warning instead of the error of a straight \input, and allows latexmk to generate all missing files in the first pass.

Note: A rule with the generic target %.pdf: %.tex gives you trouble once you start using \includeonly in your document, for reasons internal and complex. That's why I used a specific rule instead of a generic one.

There is actually one alternative to latexmk that I can also recommend. In case you are looking at a more involved project setup, you might consider CMake, for which Kenneth Moreland has done the excellent UseLATEX.cmake module.

This, however, is a bit too involved to give a how-to in the scope of this answer.

DevSolar
  • 7,857
  • haha, this actually works! sweet. I'm going to mess around with it a bit and probably accept! +1 – romeovs Jan 11 '12 at 15:09
  • since latexmk apparently doesn't know about the pre formatted preamble. I needed to do main.pdf: preamble.fmt. Otherwise all is fine! – romeovs Jan 11 '12 at 15:19
  • 3
  • Actually, if you turn the recorder option on in latexmk, it does know about the format file. 2. If you are using a makefile, it's a good idea to have all the source files listed in the makefile. Latexmk can set this up automatically; see the section "USING latexmk WITH make" in the latexmk documentation.
    • – John Collins Jan 11 '12 at 20:59
  • @romeovs: Please note the complete overhaul of the answer. Thanks to John Collins, it now includes a fully functional example. – DevSolar Jan 13 '12 at 08:17
  • @JohnCollins: I hope the link gets you lots of upvotes. Thank you for that macro solution! – DevSolar Jan 13 '12 at 08:18
  • MyDoc.pdf is a real file being produced---you shouldn't be listing it as .PHONY. – uckelman Feb 25 '14 at 12:34
  • @uckelman: MyDoc.pdf is not only depending on MyDoc.tex, as the Makefile states, but also on the various generated / included files. Only latexmk knows which ones that would be. So we need to run latexmk in any case so it can inspect if anything important has changed. You can use certain latexmk options to embed its dependency logic into the Makefile, but I did not do this in my example for simplicity's sake. Listing MyDoc.pdf in .PHONY is therefore the right thing to do; latexmk will return immediately if nothing needs updating. – DevSolar Feb 25 '14 at 13:37
  • I guess it makes more sense to me to use dep files as described in latexmk's man page, since it's only a few lines in the Makefile. – uckelman Feb 25 '14 at 15:00
  • @uckelman: Maybe. The Makefile posted was basically verbatim what I used myself at that time (and still use), with comments added. At some point I'm happy to leave "well enough" alone, especially when considering a tradeoff between "simple" and "polished" in a SE answer. Including dependency files isn't that easy to explain, and somewhat OT for tex.SE. – DevSolar Feb 25 '14 at 15:38
  • Thanks for your answer. But I have another problem could you look at it? It is in the comments to my question. – quapka Apr 20 '14 at 21:13
  • So, I have come to a solution and posted it to my question. – quapka Apr 26 '14 at 19:56
  • Is latexmk also usefull when you have Rnw files for R / knitr in the project? – buhtz Dec 12 '16 at 09:24
  • @buhtz: I have no knowledge about R / knitr. Can you put those into a make rule? Then yes. – DevSolar Dec 12 '16 at 09:27
  • It looks I am trying to achieve something similar here. Is it better to use latexmk, basic make or bash+git to compile Tex files in a project? – hhh Apr 17 '20 at 19:40
  • 2
    @hhh: IMHO, version control (git/svn) and building should be two completely separate steps. Consider a build from an exported working directory, or a build made after your origin repo went the way of the Dodo. Or a build that runs on sources intentionally different from what is in the repo. A build should still be possible, and not change the sources present. I.e., trying to get git involved in this is a bad idea IMHO. – DevSolar Apr 17 '20 at 20:44
  • @DevSolar I want to save time, not wanting to compile each single page of large project, just the changed parts. Does latexmk avoid unnecessary compilation? – hhh Apr 17 '20 at 21:44
  • @hhh It avoids unnecessary recompilation where possible. You can help with that by modularizing your document, or by compiling only the subsection you are currently interested in (\includeonly). But that would be a different question. – DevSolar Apr 18 '20 at 13:26
  • @DevSolar I compiled everything with latexmk and some python, here. It was a nuisance to try to get Makefile work with Bash and LatexMk, so decided to go with Python. I had some problems with enviroment variables in Bash that I do not have in Python. – hhh Apr 18 '20 at 16:02
  • Thanks for the answer, by the way, what do %.tex: %.raw and %.tex: %.dat do? I commented them out and the make still works. – zyy Aug 14 '21 at 05:44
  • @zyy: They are placeholders, showcasing how you would add any custom steps that generate LaTeX source from some other input. If you don't have any of those, just leave the lines out. – DevSolar Aug 14 '21 at 12:23
  • 1
    @DevSolar I see, so those rules generate .tex files that may be included by MyDoc.tex. Thanks! – zyy Aug 14 '21 at 15:55