When passing TeX code to Lua and back to TeX, the # symbol doubles

Question

Consider the following MWE. If I pass some TeX code to Lua (via \directlua) and Lua tries to pass it back to TeX (via tex.sprint), the following problem becomes apparent.

The code below prints

text is  \newcommand {\foo }{##1} \foo  end

As you can see, the # doubles once passed to Lua, and of course this breaks the macro \foo.

It's necessary to wrap the text argument in \detokenize or similar, in order to stop TeX expanding it before passing to to Lua, but both \detokenize and \unexpanded have this issue.

This question: Macro char # doubles when made letter? is probably the same phenomenon. And also How do I expand exactly once a macro which takes an argument.

Suggestions on how to work around this would be appreciated. I'd obviously prefer an approach whereby I wrap the argument passed to luafun inside \NewDocumentCommand in something, rather than adding extra special code to every occurence of # in my text.

And in this context, are there any other unpleasant surprises lurking in the wings with other special TeX characters? If so, please warn me now.

\documentclass[12pt]{article}
\usepackage{xparse}
\usepackage{filecontents}
\begin{filecontents*}{fn.lua}
function luafun (text)
   texio.write_nl(string.format([[text is %s end]], text))
   tex.sprint(text)
end
\end{filecontents*}
\directlua{local scratch = require "fn.lua"}

\NewDocumentCommand{\luafun}{+m}
{
  \directlua{luafun([[\detokenize{#1}]])}
  %\directlua{luafun([[\unexpanded{#1}]])}
}
\begin{document}

\luafun
{
  \newcommand{\foo}[1]{#1}
  %\foo{foo}
}

\end{document}

The correct way to pass TeX tokens to Lua is with \directlua{luafun("\luaescapestring{\unexpanded{#1}}")}. Everything else will suffer from things like the hash doubling problem. — Henri Menke, Jun 28 '19 at 06:42
@HenriMenke I was just posting exactly the same as an answer! — Joseph Wright, Jun 28 '19 at 06:43

score 4 · Accepted Answer · answered Jun 28 '19 at 06:43

4

You need \unexpanded here because of it's toks-like behaviour. You also need to properly escape the tokens, so [[ .. ]] won't work. Try instead

\NewDocumentCommand{\luafun}{+m}
  {%
    \directlua{luafun("\luaescapestring{\unexpanded{#1}}")}%
  }

answered Jun 28 '19 at 06:43

Joseph Wright

259,911
34
706
1,036

Additional spaces in the code also addressed .... – Joseph Wright Jun 28 '19 at 06:43
"toks-like behaviour"? – Faheem Mitha Jun 28 '19 at 06:44
Thanks. I'm relieved the solution is so simple. I had horrid visions of having to pepper my text with abstruse code in order to get it to work. – Faheem Mitha Jun 28 '19 at 06:46
Why are the extra % needed? I realise they get rid of newlines, but why is that a an issue here? – Faheem Mitha Jun 28 '19 at 06:47
"toks" is presumably a reference to the macro \toks, which (I think) is a token register. But the relevance still escapes me. – Faheem Mitha Jun 28 '19 at 06:55
@FaheemMitha standard tex issue try X\luafun{xxx}X without the % you get additional space in the typeset output. – David Carlisle Jun 28 '19 at 07:00
@FaheemMitha Standard TeX behaviour: end-of-line converts to space, so if you don't 'kill' them you get spaces in the output – Joseph Wright Jun 28 '19 at 07:00
@FaheemMitha Yes, I mean that the behaviour is that of a toks register: they have special rules for # tokens, which you need here – Joseph Wright Jun 28 '19 at 07:00
@DavidCarlisle and Joseph: I see. So extra space in the output of \luafun. – Faheem Mitha Jun 28 '19 at 07:01

When passing TeX code to Lua and back to TeX, the # symbol doubles

1 Answers1