Using \input|texlua... to emulate \directlua in pdfTeX

Question

When --shell-escape is enabled, pdfTeX allows to run external programs and input the result expandably with \input|"...". Together with the Lua interpreter texlua, this could be used to provide a dumbed down version of \directlua. However, texlua expects a file name as its argument.

In my setup (Ubuntu 10.04, with up-to-date TeXLive 2012 install), one can run pdftex --shell-escape on

\everyeof{\noexpand}
\message{\input|"echo print\string\(3.4+5.7\string\) > tmp.lua ; texlua tmp.lua" a}
\bye

to get \message{9.1}. Note that the \everyeof{\noexpand} is in part for convenience, it would be possible to make the construction work (not entirely robustly) without this non-expandable step.

This approach is not portable because it relies on echo, which probably doesn't exist in Windows environments. One way around this would be to have a simple Lua script taking the code print\string\(3.4+5.7\string\) as its argument (see a recent question on texlua arguments), then somehow performing it. Is this possible? Otherwise, we need an os-dependent approach.

A second problem is that special characters such as parentheses need escaping. Which ones? In particular, I'm worried about quotes ", which are used to enclose the argument of \input|. Given a list of which escapings to perform, I have some code up my sleeve to do that expandably (see the l3kernel-extras package on the LaTeX3 svn repository).

Basically, my question boils down to: "How can I invoke texlua from pdfTeX with arbitrary Lua code as portably as possible?"

echo certainly exists on Windows, although the syntax is slightly different to on *nix. Take a look for example at the LaTeX3 make.bat files ;-) — Joseph Wright, Aug 01 '12 at 19:13
This seems to be 'works for me' on Windows with an altered second line :\message{\input|"echo print(3.4+5.7) > tmp.lua & texlua tmp.lua" a} (as cmd.exe does not treat parentheses as special and you need to use & to start a second process). Would a solution using ifplatform be acceptable? — Joseph Wright, Aug 01 '12 at 19:22
@JosephWright yes. My slightly longer term plan is to find out how to add \input| support to XeTeX, and have an expandable quasi-\directlua, of course with no support for interacting with TeX directly. I am very much interested in knowing which characters have to be escaped depending on the platform. — Bruno Le Floch, Aug 01 '12 at 19:28
On the special characters business, life is trickier on *nix than on Windows. The Windows version sends everything through to the file exactly 'as is', except for >, < and ^ which are escaped as ^>, ^< and ^^. Unix is more tricky, but this is more about how Unix works than how TeX works. — Joseph Wright, Aug 01 '12 at 19:40

Joseph Wright · Accepted Answer · 2012-08-01T20:25:22.087

The echo built-in is available on both Unix and Windows, and so provided platform detection is available then the problem is in principal one of deciding which characters to escape. For LaTeX, platform detection can be done using the ifplatform package, which provides the \ifwindows switch.

The first character to worry about is the 'chain process' one. On Windows, this is &, whereas on Unix ; is correct. That be handles with something like

\RequirePackage{ifplatform}
\edef\luachainprocess{\ifwindows\string&\else;\fi }

There is then the question of special characters in the echo part. For Windows, the required substitutions are

> replaced by ^>
< replaced by ^<
| replaced by ^|
& replaced by ^&
^ replaced by ^^

with everything else left alone by echo.

On Unix, the appropriate list seems to be

> replaced by \>
< replaced by \<
( replaced by \(
) replaced by \)
& replaced by \&
| replaced by \|
` replaced by \`
\ replaced by \\

(Note: that is with the bash shell, but others may vary.)

As Windows allows for example ^( for (, it would probably be easiest to use a common set of characters for replacement with only the escape character itself varying. (The same is true for Unix with for example \^.)

One thing that is not clear at all is if there is any way to deal with " in either case, as the pipe syntax for \input does not provide an escape for this, so it seems to be misunderstood whatever you do. This is particularly frustrating for the Unix case as generating effectively

 echo "<code>"

would avoid needing to escape anything other than " inside the code.

I'm of course assuming that all of the replacements are on a detokenized basis. — Joseph Wright, Aug 01 '12 at 20:26

score 6 · Answer 2 · answered Aug 01 '12 at 22:16

I found out about the loadstring function in Lua, which converts a string to Lua code. To avoid any escaping issue for the TeX → Lua conversion, the TeX code converts the argument of \texlua to a space-separated list of character codes, which is then given as extra arguments to texlua. The Lua code decodes this using to str.char.

The remaining weak link is that \input|..., just like any other \input or \scantokens, inserts an end-of-file marker, which may not appear in the argument of any command. We need to get rid of it, and one way is to add some material just before, and just after, the end of the file, here \romannumeral\numexpr0\noexpand and \relax (the \noexpand is key). Unfortunately, the code \romannumeral\numexpr0\noexpand is not inserted if there is a Lua error; in this case, the end-of-file marker is not removed, and all hell breaks loose.

\begingroup
  \catcode `@ = 11
  %
  % Get \texlua@input = primitive \input
  \expandafter\ifx\csname @@input\endcsname\relax
    \global\let\texlua@input\input
  \else
    \global\let\texlua@input\@@input
  \fi
  %
  % Write very simple Lua file.
  \newwrite \texlua@write
  \xdef \texlua@file {\jobname.texlua}
  \immediate \openout \texlua@write \texlua@file \relax
  \immediate \write \texlua@write
    {%
      assert(loadstring(string.char(unpack(arg))))()%
      \detokenize{print("\\romannumeral\\numexpr0\\noexpand")}%
    }
  \immediate \closeout \texlua@write
  %
  % Detokenize the argument and convert spaces to category other.
  \gdef\texlua@str#1%
    {%
      \expandafter\texlua@str@i
        \detokenize{#1}\\ \\\texlua@str@ii/ \relax
    }
  \lccode `* = 32
  \lowercase{\gdef\texlua@str@i#1 #2\\#3 {#3\texlua@str@i#1*#2\\{#3} }}
  \gdef\texlua@str@ii/#1#2\\#3\relax{#2}
  %
  % Convert each character to its character code, space-separated.
  % This is not "f-expandable", it is "restricted expandable" in
  % expl3 terms.
  \gdef\texlua@num#1%
    {\expandafter\texlua@num@i\romannumeral-`q\texlua@str{#1}\relax}
  \gdef\texlua@num@i#1%
    {%
      \ifx #1\relax
      \else
        \number `#1 \space
        \expandafter\texlua@num@i
      \fi
    }
  %
  % Calling `\input|`.
  \gdef\texlua#1%
    {%
      \texlua@input|"texlua \texlua@file \space \texlua@num{#1}"%
      \relax
    }
\endgroup

\message{a\texlua{io.write(3+4)}b}

score 4 · Answer 3 · answered Aug 02 '12 at 23:30

In filter module in ConTeXt provides the necessary boilerplate to run the content of an environment or macro through an external command (full disclosure, I am the author of the module). It does not use \input| to read the output, rather assumes that the external program will write the file to an external file and then the user can specify how to read the file (the default is \ReadFile, which is similar to \input).

The example below illustrates how to process using texlua.

% engine=pdftex

\usemodule[filter]

\defineexternalfilter
    [luatex]
    [filtercommand={texlua \externalfilterinputfile > \externalfilteroutputfile}]

\starttext

\inlineluatex{print((3+5)*3)}

\startluatex
  local t = {1,2,3}
  print("Number of entries " .. #t)
\stopluatex

\stoptext

Note: The \inlineluatex macro is not expandable.

Thank you Aditya for your answer. My initial goal was to get an expandable solution, hence the question using \input|. As noted in my (ugly) answer above, the end-of-file marker that (pdf)TeX inserts upon \input makes an expandable solution significantly harder. Also, expandability will be impossible in XeTeX.
So if we ever decide to implement this kind of functionality in LaTeX3, it will have to be unexpandable. Having accepted this, the next question would be what a good interface could be, and you answer answers this neatly with your ConTeXt example. — Bruno Le Floch, Aug 02 '12 at 23:41

Using \input|texlua... to emulate \directlua in pdfTeX

3 Answers3