TikZ externalize while using -output-directory

Question

I'm using tikzexternalize to speed things up and to have a clean working directory, And I'm using -output-directory on compilation to have a clean working directory (actually I use cluttex, but I've seen the problem with just -output-directory as well).

So my tex code looks something like this

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=figures/] % activate and define figures/ as cache folder
\begin{document}
\begin{tikzpicture}
    \node[draw] {lore ipsum};
\end{tikzpicture}
\end{document}

and compile with pdflatex -shell-escape -output-directory='tex-aux' main.tex (actually I'm using LuaLaTeX, but the problem remains when using normal (pdf)latex).

The error I get is

 \write18 enabled.
entering extended mode
! I can't write on file `figures/main-figure0.log'.
(Press Enter to retry, or Control-D to exit; default file extension is `.log')
Please type another transcript file namesystem returned with code 256
! Package tikz Error: Sorry, the system call 'pdflatex -shell-escape -halt-on-e
rror -interaction=batchmode -jobname "figures/main-figure0" "\def\tikzexternalr
ealjob{main}\input{main}"' did NOT result in a usable output file 'figures/main
-figure0' (expected one of .pdf:.jpg:.jpeg:.png:). Please verify that you have
enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometimes
 it is also named 'write 18' or something like that. Or maybe the command simpl
y failed? Error messages can be found in 'figures/main-figure0.log'. If you con
tinue now, I'll try to typeset the picture.
See the tikz package documentation for explanation.
Type  H <return>  for immediate help.
 ...
l.12 \end{tikzpicture}

So I know when removing -output-directory='tex-aux' everything works just fine (I even tried manually creating the folder tex-aux/figures just for the case this is the trouble, but it didn't help).

I also know that the hint to check if system calls are enabled shouldn't be the problem, since they are enabled with -shell-escape.

Any suggestions on what the problem might be (and how to solve it)?

There's an alternative solution (to keep clean working directory) of cd to the desired temporary directory, compile the file there (specify full/relative path), then copy the PDF file back. -output-directory just breaks a lot of packages. — user202729, Dec 10 '21 at 13:01
... actually this one can probably be fixed with https://tex.stackexchange.com/a/37825/250119 (although make sure the code is trusted before compiling) — user202729, Dec 10 '21 at 13:03
Oh, didn't knew that, thanks for the hint. Shouldn't be too hard to include this in a Makefile based workflow ;)
Regarding the second suggestion, well if it's not too hard to do this cd I think it's not worth adding anything which has security implications. Thanks again, would you mind writing a short answer, so I can accept it? ;) — atticus, Dec 10 '21 at 13:05
No, not for me openout_any=a pdflatex -shell-escape -output-directory='tex-aux' main.tex gives the same error. I think I'll go with the cd solution (so now I'm trying to avoid that cluttex adds the output-directory option ^^ But if that doesn't work, I think I'll have to use latexmk or something else again...) — atticus, Dec 10 '21 at 13:15
Maybe I'll write one on Prevent pdflatex from writing a bunch of files - TeX - LaTeX Stack Exchange instead, after I test it out. Or you can if it it work for you. — user202729, Dec 10 '21 at 13:23
Well I think I'll use cd tex-aux && cp ../main.tex ./ && lualatex -recorder -halt-on-error -interaction=nonstopmode -file-line-error -shell-escape -jobname='main' "main.tex" && cp $@ ../ for now ($@ comes from the Makefile target). Not pretty but functional for now. But I'm hoping there will be a fix for cluttex (at least some way of disabling the use of output-directory in the latexengine command) since this way I'm missing automatic recomilation (if one run isn't enough) — atticus, Dec 10 '21 at 14:16
Hm well cdonly works if there are no \input statements. For now I think I'll have to cope with all the temporary files in my workin directory — atticus, Dec 10 '21 at 21:13
Actually the issue might be uncreated directory pgfplots - Error using tikz externalize: can't write md5 file - TeX - LaTeX Stack Exchange rather than openout_any. (see also luatex - TikZ externalize + LuaLaTeX + Output Dir + Precompile Preamble = Fail - TeX - LaTeX Stack Exchange ) — user202729, Dec 11 '21 at 00:06
For the cd issue see Using relative paths for \include commands - TeX - LaTeX Stack Exchange, or something in \input and absolute paths - TeX - LaTeX Stack Exchange — user202729, Dec 11 '21 at 00:13
Well the cd solution might work if one adds .. to \input@path on compilation (possibly necessary for the tikzexternalize command as well). But I'll have to see and test what I'm going to use. — atticus, Dec 12 '21 at 13:04
I've added the two possible solutions I came up with based on our discussion/your links. What do you (@user202729) think about the first one? (I personally think it's nice but on the other hand I'm very unsure if it should work since in this case the usage of the paths is a bit strange) — atticus, Dec 12 '21 at 20:28
Why not post them as self-answers? (also, fix your smart quotes. Use ` ... ` for inline code format and ' for apostrophe.) (not an expert on this. Go ask the actual experts.) — user202729, Dec 13 '21 at 03:29

score 0 · Answer 1 · answered Dec 13 '21 at 10:37

Solution1: using cluttex --output-directory=tex-aux -e lualatex -shell-escape main.tex with

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize[prefix=tex-aux/]
\begin{document}
\begin{tikzpicture}
    \node[draw] {lore ipsum};
\end{tikzpicture}
\end{document}

but it's important that prefix equals output-directory and the folders output-directory and output-directory/prefix exist. So far this looks to me like the cleanest way (not much configuration effort) but I'm not sure if this will work on all setups because we`re fooling the extern lib with the prefix.

Solution2: use something like

cd tex-aux && \
cp ../main.tex ./ && \
lualatex -recorder -halt-on-error -interaction=nonstopmode -file-line-error -shell-escape -jobname='main' "\makeatletter\def\input@path{{..}}\makeatother\input{main.tex}" \
&& cp $@ ../

($@ for the output file comes from make)

Here we're adding the parent directory to the path (see https://tex.stackexchange.com/a/24827/206293) so that `main.tex´ is found not only on the direct call in the compilation command but also in the compilation commands issued by tikz-extern lib.

Solution3: Looking at the cluttex code normally it should modify TEXINPUTS (add the directory before the cd) in combination with the cd approach when using the flag --change_directory. Currently this doesn't work completely, but one might use this approach manually (or maybe this pull request get through and one can use cluttex directly)

TikZ externalize while using -output-directory

1 Answers1

Linked