lualatex gives compile error when using -jobname with tikz externalization of images

Question

Using TL 2023, this command works

lualatex --shell-escape  --file-line-error index.tex

Gives no error:

...
/usr/local/texlive/2023/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg))
===== 'mode=convert with system call': Invoking 'lualatex -shell-escape -halt-o
n-error -interaction=batchmode -jobname "images/my_image_file" "\def\tikzextern
alrealjob{index}\input{index}"' ========
This is LuaHBTeX, Version 1.16.0 (TeX Live 2023) 
 system commands enabled.
[1{/usr/local/texlive/2023/texmf-var/fonts/map/pdftex/updmap/pdftex.map}<./imag
es/my_image_file.pdf>] (./index.aux))

But this gives error

lualatex --shell-escape  --file-line-error -jobname=LUA index.tex

This error happens when the latex file is trying to save tikzpicture to pdf file using the externalization package of tikz. It give this error

/usr/local/texlive/2023/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg))
===== 'mode=convert with system call': Invoking 'lualatex -shell-escape -halt-o
n-error -interaction=batchmode -jobname "images/my_image_file" "\def\tikzextern
alrealjob{LUA}\input{LUA}"' ========
This is LuaHBTeX, Version 1.16.0 (TeX Live 2023) 
 system commands enabled.
./index.tex:15: Package tikz Error: Sorry, the system call 'lualatex -shell-esc
ape -halt-on-error -interaction=batchmode -jobname "images/my_image_file" "\def
\tikzexternalrealjob{LUA}\input{LUA}"' did NOT result in a usable output file '
images/my_image_file' (expected one of .pdf:.jpg:.jpeg:.png:). Please verify th
at you have enabled system calls. For pdflatex, this is 'pdflatex -shell-escape
'. Sometimes it is also named 'write 18' or something like that. Or maybe the c
ommand simply failed? Error messages can be found in 'images/my_image_file.log'
. If you continue now, I'll try to typeset the picture.
See the tikz package documentation for explanation.
Type  H <return>  for immediate help.
 ...
l.15 \end{tikzpicture}
? 

It is using the jobname which is causing the problem. Otherwise externalization works, and the pdf of the tikzpicture is created correctly.
I needed to use -jobname option because I also build tex4ht in same folder and this option helps keeps auxiliary and temp file names separated and not clash when I compile to pdf and html in same folder.

In images/my_image_file.log it says

......
Inserting `luaotfload.rewrite_fontname' in `luaotfload.patch_font'.
Inserting `tracingstacklevels' in `input_level_string'.
! Emergency stop.
<read *>
<*> \def\tikzexternalrealjob{LUA}\input{LUA}

I think the problem is that it looking for LUA.tex file in the above command, but there is no LUA.tex file. This is supposed to be just the jobname, not not the input file name ! So the above command should have been

\def\tikzexternalrealjob{LUA}\input{index}

Below is MWE to reproduce this. This is on Linux virtual box running latest TL 2023

\documentclass[12pt]{article}
\usepackage{shellesc} %need for lualatex!       
\usepackage{tikz} 
\usetikzlibrary{external}
\tikzexternalize[prefix=images/]
\begin{document}
%this command saves pic to images/my_image_file.pdf
%make sure to create images/ folder first as it will not do it
\tikzsetnextfilename{my_image_file}
\begin{tikzpicture}
    \node[] at (0,0) {some text};
\end{tikzpicture}
\end{document}

When running these commands, please make sure to have images/ folder created and empty each time in order to see the problem.

I found that if I have the image file there before, then no error shows up when using jobname option because it sees the pdf image file already and will not attempt to create it again.

Question is: It is possible to use -jobname with lulatex and also use tikz externalization?

Answer 1

I propose an alternative which is faster, more secure and more robust than the external library. It doesn't require full shell escape (only restricted, which is default) and works in this case out-of-the-box. Oh, and it will make the directory for you.

\documentclass[12pt]{article}
\usepackage[extract=perl]{memoize}
\mmzset{% perl is default
  path={dir=images},
  mkdir,
}
\usepackage{tikz}
\begin{document}
\begin{tikzpicture}
    \node[] at (0,0) {some text};
\end{tikzpicture}
\end{document}

Compiled with

$ TEXMFHOME=/d lualatex -jobname="lua" prawf

Files produced after two compilations:

$ ls lua.* images/
lua.aux  lua.log  lua.mmz  lua.mmz.log  lua.pdf
images/:
lua.EEA66177700E896EF689C961311611CD-E778DCCCB8AAB0BBD3F6CFEEFD2421F8.memo  lua.EEA66177700E896EF689C961311611CD.memo
lua.EEA66177700E896EF689C961311611CD-E778DCCCB8AAB0BBD3F6CFEEFD2421F8.pdf

What you cannot do, at present, is assign custom names. (I've started lobbying for this ;.) The need for them is somewhat less acute because inserting an image at the beginning of the document won't cause all images to change names, but I would prefer my images to include a human-readable name for convenience.

There are three extraction methods. You may need to install a couple of libraries for perl or python, assuming you have the interpreter available already. TeX Live requires perl, so that's the default. The TeX method requires shell escape and is slowest.