22

I'm trying very hard not to make this sound like a rant, but I'm very angry right now.

I was trying to use the TikZ externalization to create a scatter plot for my thesis. There were too many data points for TeX's memory so I switched to LuaTeX. The externalization didn't work.

(MWE like:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{external}
\tikzexternalize
\begin{document}
\tikz\node{Hello};
\end{document}

)

I had past experience with MiKTeX that suggested that changing -shell-escape to -enable-write18 can make a difference, as well as -enable-write18 vs --enable-write18. I spent literally half a day (remember, LuaTeX compilation times are still orders of magnitude slower on MiKTeX) trying to make it work and figure out why it doesn't, changing the command line around as well as the value of tikz/external/shell escape. Nothing worked.

Today I do a Google search and I find the answer at https://github.com/gpoore/minted/issues/112: There is no more \write18 in LuaTeX. You have to \usepackage{shellesc} which adds it back again.

My question is, I'm trying to be a good user here and always RTFM. Where is TFM I should have read to find out that this functionality was removed and how to add it back? To avoid spending half a day thinking I'm doing something wrong? Surely it's not the Github issue of a completely unrelated package?

marczellm
  • 11,809
  • 3
    Drop by in chat and see the LaTeX team rant about the newest LuaTeX features? – Johannes_B Apr 23 '17 at 08:52
  • 6
    \write18 was never part of TeX itself, it's an extension using the fact that anything above 15 is 'free' in standard TeX. LuaTeX is quite deliberately making breaking changes, and up to v1.0 was 'beta' quality only. Given that \write18 is a security risk, the LuaTeX devs felt that it's a 'edge' feature (it's not \def or similar!). Various macro package developers have been working hard to make the changes transparent to end users, but this is non-trivial. I'm not sure what an answer here would be. – Joseph Wright Apr 23 '17 at 08:55
  • 6
    The LuaTeX manual, on page 44 (as of currently) says "Also, we no longer support write18 because os.execute can do the same." The manual also mentions --enable-write18 and --disable-write18 flags though. But see also the warning at LuaTeX documentation page: “The snapshot manual (in pdf format) is automatically updated from the source […] but nothing in it should be considered stable. The way things are right now, the actual binary is the definitive answer to what luatex does or does not do.” – ShreevatsaR Apr 23 '17 at 09:00
  • @JosephWright My gut feeling is that the right way to handle breaking changes like these would be to notify end users with "big red neon signs" that anything and everything may break. And that it's very wrong to 1) push out these changes as an "update" as MiKTeX is doing, in the age where the correct way to keep your software working is to update often, and 2) to encourage everyone on sites like this to switch to LuaTeX just because, as some are doing. – marczellm Apr 23 '17 at 09:00
  • @ShreevatsaR Should I reread the 230 page LuaTeX manual, that contradicts itself within one page, with every software update? – marczellm Apr 23 '17 at 09:03
  • 6
    @marczellm The LuaTeX manual has always been clear about it being beta software (up to v0.95). I'm not sure what Christian Schenk could do about the change here: MiKTeX (unlike TeX Live) does allow changes in engine binaries within the same release. (TeX Live users will only get the update by changing version.) As I've said, from the point of view of the LuaTeX developers, shell escape is not a core feature: they have after all changed the hyphenation approach compared to TeX90! – Joseph Wright Apr 23 '17 at 09:04
  • As I've said, I'm not sure that we here can answer: Hans Hagen and Christian Schenk seem to be the only people qualified to give an answer. (I also note that David Carlisle in particular has worked hard to make this and other changes transparent to users.) – Joseph Wright Apr 23 '17 at 09:05
  • 2
    See also “Can I use LuaTeX today?” on the LuaTeX FAQ page. Here's a screenshot. As I understand it, the LuaTeX team has always been clear about it being in flux, and when they feel comfortable about changing that statement they will do so. (Disclaimer: I am only an occasional user of LuaTeX and all I know about it is from clicking around the web site… but such a note has always been there AFAIK.) – ShreevatsaR Apr 23 '17 at 09:12
  • Switch to a stack of languages with standards, like HTML, SVG, JS, CSS. Literally the only TeX software that is guaranteed to work always the same way is that written by Knuth. – polkovnikov.ph Apr 23 '17 at 23:36
  • Not directly important, but isn't there a simple alternative toThere were too many data points for TeX's memory so I switched to LuaTeX.? Something like pdflatex --extra-mem-top=60000000 -shell-escape -interaction=batchmode my_LaTeX_File.tex (which I use for example, MiKTeX). – Dr. Manuel Kuehner Apr 25 '17 at 19:04

3 Answers3

20

The manual of luatex mentions the change:

Also, we no longer support write18 because os.execute can do the same.

But I doubt very much that even if you had read this that you would have understand the implications and that it could affect tikzexternalize.

The change happened last year and if you search this site for shellesc you will find a number of questions (also involving texlive, it is not a miktex specific problem, Cannot run tikz externalize with lualatex but it used to work). But again: even if you saw one of this questions during the year I wouldn't expect from you to remember them and realize that they applicate to your current problem.

So what could you do? Some tipps:

  • Changes happen, even more with new software like luatex. Accept this.
  • Not every change is obvious and can be learned by reading the manual.
  • Don't try to solve your problems for hours alone. Search a bit and if you have the impression that you don't make any progress then better spent your time to make a sensible minimal example and then ask.
Ulrike Fischer
  • 327,261
  • I wonder what's the status of this issue now (2022-08-16). Like the OP, I've spent quite a lot of time wondering why write18 doesn't have any effect. If it's not part of lualatex, shouldn't it emit a clear error message like "\write18 is not supported" or "\write18 is not defined"? I don't understand why \write18 is silently ignored. – Ryo Aug 16 '22 at 11:53
  • 1
    @Ryo it is not silently ignored. If you do \immediate\write18{blblblblblb} you will find the text either in the log or in a file. If you want that the shellesc package is loaded by default by the latex kernel, make a feature request. – Ulrike Fischer Aug 16 '22 at 12:23
  • Thanks! Would it be possible to make \write18 an obvious error? A feature request for that would be ideal. I'm not sure whether the feature request you recommend is a good idea. While you are correct that the text is there in the log file, 1) it's in the sea of log messages and 2) the printed-out text doesn't make it clear that \write18 is doing nothing but copying the text to the log file. That was the very reason why it took so much time for me to find out the cause of the problem. – Ryo Aug 18 '22 at 02:10
12

It's a bit unfortunate that Christian Schenk deploys in MiKTeX new versions of the binaries (luatex in this case) as soon as they are available.

TeX Live only changes (core) binaries with the yearly release and doesn't modify them during the year (except for very serious bugs, but these are usually caught during the pretest phase).

When LuaTeX 0.87 was released, many changes were made, in particular the one about \write18. If you look at LaTeX News 24, released February 2016, you'll see

enter image description here

and also

enter image description here

marczellm
  • 11,809
egreg
  • 1,121,712
  • 1
    Not quite sure why you blame miktex handling of binaries, the example doesn't work in TL17 pretest either. There too you need to add shellesc and unless shellesc is added to the latex kernel similar questions will pop up for texlive when 2017 gets official. Als shellesc was added to miktex month ago. – Ulrike Fischer Apr 23 '17 at 10:19
  • @UlrikeFischer I don't blame MiKTeX; I just observe that the release of TL 2016 had no problems, provided shellesc is loaded as recommended in the newsletter. – egreg Apr 23 '17 at 10:24
  • 3
    With shellesc it works also with miktex. There is no binary problem here. Tl16, TL17 and miktex behave here quite identical: you need shellesc and shellesc is avalaible. – Ulrike Fischer Apr 23 '17 at 10:28
  • 2
    I accepted this answer because the newsletter is much easier to follow for breaking changes than a mailing list or a 230 page manual. – marczellm Apr 24 '17 at 13:32
10

If using a beta release of software it's best to follow its development channels as breaking changes are expected in beta releases. This was discussed on the luatex list before the change was made. (Not true of all breaking changes in luatex:-). For example this thread from November 2015 http://tug.org/pipermail/luatex/2015-November/005499.html

Without following the mailing list, this change was flagged in ltnews and also in the luatex manual.

That said, in an ideal world an end user would not need to know this at all, we did search ctan for every use of \write18 that we could find and try to reach out to package developers to update packages before the changed luatex was distributed in major distributions. The minted github issue that you referenced being one example of that.

That there are still packages that require the end user to use shellesc (or luatex85) packages is a bit unfortunate but a possibly unavoidable result of the distributed nature of LaTeX package development. Packages are updated as and when the developers have time or inclination to publish an update.

David Carlisle
  • 757,742