Workaround for open lualatex bug in \scantokens?

Question

As package author I am in need of writing a manual with lots of examples.

Now I started to make heavy use of lualatex in order to speed up operations which take a long time in TeX (of my package, not the manual) -- and I stumbled about an open ticket (bug?) of lualatex which blocks me when I attempt to activate my code changes in the manual.

Background:

An example in my manual should be something like

\begin{codeexample}[]
<code here>
\end{codeexample}

and should result in (1) the listing in some verbatim output and (2) the result of executing <code>. I want to type the listing once (and once only).

This is a good use-case for \scantokens : I collect the tokens with category code 12 (other) in order to show the verbatim output as code listing. This listing will include syntax highlighting and automatic cross references. Afterwards I use \scantokens in order to reset the category codes to their initial meaning such that I can execute the code (to show the outcome).

This last invocation of \scantokens works flawlessly in pdftex. It does not work at all in lualatex.

The issue is known, see http://tracker.luatex.org/view.php?id=733 and \scantokens in LuaTeX .

While I could write a lua version of my package and translate the manual with the pdftex version, I would rather see the effects and document special cases. Eventually, the lua version might offer unique capabilities. Consequently, I do not want to wait until someone else fixes that open ticket - and I am searching for workarounds.

Does anyone know a workaround?

The problem as such can be formulated very simple as pointed out in the linked tex.se question. I would formulate it as

\documentclass{standalone}

\begin{document}

Here should come text: \scantokens{^^JNamely this text}

\end{document}

pdftex results in

whereas luatex results in

which is the same as in the linked question.

But I need the newlines in my use case.

This here is a reduced (I suppose minimal) example of my use-case in which I collect a listing verbatim and reassign catcodes afterwards in order to execute the listing. This "execute the listing" needs the newlines, otherwise it will fail to work. And: yes, I do not typeset the listing here (it is a minimal example).

\documentclass{article}

\usepackage{pgfplotstable}

\makeatletter
% Define \find@example such that it doesn't destroy catcodes:
\begingroup
\catcode`|=0
\catcode`[= 1
\catcode`]=2
\catcode`\{=12
\catcode `\}=12
\catcode`\\=12 |gdef|find@example#1\end{codeexample}[|endofcodeexample[#1]]
|endgroup

\def\OLDNEWLINE{^^J}%
%% ATTEMPT (1*):
%% This here result in output of the first minimal. But it breaks the
%% second one.
%%\def\OLDNEWLINE{}%

% define \returntospace.
%
% It should define NEWLINE as {}, spaces and tabs as \space.
\begingroup
\catcode`\^=7
\catcode`\^^M=13
\catcode`\^^I=13
\catcode`\ =13%
\gdef\returntospace{\catcode`\ =13\def {\space}\catcode`\^^I=13\def^^I{\space}\catcode`\^^M=13\def^^M{\OLDNEWLINE}}%
% 
% ATTEMPT (2*):
%\gdef\returntospace{\catcode`\ =13\def {\space}\catcode`\^^I=13\def^^I{\space}\catcode`\^^M=13}%
\endgroup

\def\codeexample[#1]{%
    \parindent0pt
    \begingroup%
    \par%
    \medskip%
    \let\do\@makeother%
    \dospecials%
    \obeylines%
    \@vobeyspaces%
    \catcode`\^^M=13 %
    \find@example}
\def\endofcodeexample#1{%
    \endgroup%
    {%
      \returntospace%
      \xdef\code@temp{#1}% removes returns and comments
    }%
    %
    % ATTEMPT (2*): This here fixes the first minimal example together with (2*):
    %\catcode`\^^M=9 % 9 == ignore
    \expandafter\scantokens\expandafter{\code@temp}%
  \end{codeexample}
}

\makeatother



\begin{document}

This should result in the picture:
\begin{codeexample}[]
\begin{tikzpicture}
    \draw[red,->] (0,0) -- (1,1);

    \draw[green] (0,1) -- (1,0);
\end{tikzpicture}
\end{codeexample}

This should result in the table: 
\begin{codeexample}[]
\pgfplotstabletypeset{%
A B
2 3
4 5
}
\end{codeexample}
\end{document}

pdftex generates the expected outcome which is

whereas the luatex output is

I already tried a couple of things as outlined in the code. I already had the idea to use

\scantokens{\def\CONTENT{<sequence of 'other' catcodes}}

and use a "search-and-replace" routine on \CONTENT to replace every occurence of newlines with - well, I did not know with what. And simple search-and-replace routines will fail with curly braces inside of this macro.

I have used Tex live 2013 (2014 does not fit on my hard drive currently). The lua ticket is in state "new" so I do not expect useful updates.

So: does anyone have an idea how I could adopt this codeexample code in order to (a) collect the listing of both examples and (b) execute both listings successfully without modifying the listings as such?

Answer 1

Instead of \scantokens\expandafter{\code@temp}, you can pass the string to Lua, let it split the string at new lines, and tex.print each line. Calling tex.print only once would result in a single line of input, and no amount of catcode trickery would make TeX behave as if there were multiple lines.

I thus use the following Lua code in the full example below:

local s = "\luaescapestring{\code@temp}"
for line in s:gmatch("[^\string\n]+") do
  tex.print(line)
end

As it is, the code is not quite a replacement for \scantokens. A more accurate replacement would be the following, which takes care of the value of \newlinechar. If it is negative (well, I should also test for when it is too big) then output the string directly. Otherwise, set the variable newline to the new line character (as a Lua string), possibly with an escaping %, then define a pattern which matches any sequence of one or more characters other than a new line (actually I just realize that this will fail when there are two new lines in a row), and print each such sequence through tex.print.

\long\def\myscantokens#1{\directlua{%
  local s = "\luaescapestring{\detokenize{#1}}"
  if \the\newlinechar < 0 then
    tex.print(s)
  else
    local newline = string.char(\the\newlinechar)
    if not newline:match("\string\%w") then
      newline = "\string\%" .. newline
    end
    local pattern = "[^" .. newline .. "]+"
    for line in s:gmatch(pattern) do
      tex.print(line)
    end
  end}}

Anyhow, here I'll use the simpler code since it seems to work for your case.

\documentclass{article}

\usepackage{pgfplotstable}

\makeatletter
% Define \find@example such that it doesn't destroy catcodes:
\begingroup
\catcode`|=0
\catcode`[= 1
\catcode`]=2
\catcode`\{=12
\catcode `\}=12
\catcode`\\=12 |gdef|find@example#1\end{codeexample}[|endofcodeexample[#1]]
|endgroup

\def\OLDNEWLINE{^^J}%

% define \returntospace.
%
% It should define NEWLINE as {}, spaces and tabs as \space.
\begingroup
\catcode`\^=7
\catcode`\^^M=13
\catcode`\^^I=13
\catcode`\ =13%
\gdef\returntospace{\catcode`\ =13\def {\space}\catcode`\^^I=13\def^^I{\space}\catcode`\^^M=13\def^^M{\OLDNEWLINE}}%
\endgroup

\def\codeexample[#1]{%
    \parindent0pt
    \begingroup%
    \par%
    \medskip%
    \let\do\@makeother%
    \dospecials%
    \obeylines%
    \@vobeyspaces%
    \catcode`\^^M=13 %
    \find@example}
\def\endofcodeexample#1{%
    \endgroup%
    {%
      \returntospace%
      \xdef\code@temp{#1}% removes returns and comments
    }%
    \directlua
      {%
        local s = "\luaescapestring{\code@temp}"
        for line in s:gmatch("[^\string\n]+") do
          tex.print(line)
        end
      }%
  \end{codeexample}
}
\makeatother



\begin{document}

This should result in the picture:
\begin{codeexample}[]
\begin{tikzpicture}
    \draw[red,->] (0,0) -- (1,1);

    \draw[green] (0,1) -- (1,0);
\end{tikzpicture}
\end{codeexample}

This should result in the table: 
\begin{codeexample}[]
\pgfplotstabletypeset{%
A B
2 3
4 5
}
\end{codeexample}
\end{document}