9

I don't understand how the token library of luatex works.

\documentclass{standalone}

\begin{document}

\def\mymacro{A}

\directlua{%
  t = token.create ('mymacro')
  token.expand(t)
  token.get_next()
}

\end{document}

The token.expand does not work as I expected: luatex says Undefined control sequence. Why?

With the token.get_next(), I don't know exactly what I should get...

cjorssen
  • 10,032
  • 4
  • 36
  • 126

3 Answers3

8

Unfortunately token.expand is currently broken. See the bug report which derives from an example exactly like yours.

Andrew Swann
  • 95,762
  • Update: The bug report says "closed" but the asker's code still raises the same error. – user202729 Nov 15 '21 at 07:39
  • @user202729 Then it would be good for you to report it as a bug. Note that it was closed several years ago, so a recent change may have broken this again. – Andrew Swann Nov 15 '21 at 07:46
  • 1
    @user202729 The token library got completely replaced, so just because it was not intended in the old library does not mean that it's not intended behavior now. Since the only documentation for token.expand is: "The token.expand function will trigger expansion but what happens really depends on what you’re doing where." it's arguably doing exactly what's documented :P I don't know what the old token.expand did, but the current one doesn't even take an argument. It expands what's already in the input stream. – Marcel Krüger Nov 15 '21 at 18:07
  • @MarcelKrüger Indeed further down the bug report it says "the old token interface is soon obsolete so there will be no more changes to it". – Andrew Swann Nov 16 '21 at 11:16
3

token.expand is never useful, and is not fixed even 10 years passed, since every API from luatex is isolated from variables like cur_cmd. But you can write you own expand

%!TEX program=lualatex
\documentclass{standalone}
\directlua{
local exp_after = token.create('expandafter')
local lbrace = token.create(string.byte('{'))
local rbrace = token.create(string.byte('}'))
function expand()
  token.put_next(lbrace, exp_after, rbrace)
  token.scan_toks(false, true)
end
}
\begin{document}

\def\mymacro{ABC}%


\directlua{
  local t = token.create'mymacro'
  token.put_next(t)
  expand()
  token.get_next()
}


\end{document}

If you need take care of the catcodes of braces and redefinition of \expandafter, the code is much longer.

local lbrace = token.new(string.byte'{', 1)
local rbrace = token.new(string.byte'}', 2)

function expand()
  local cur_exp_after = token.create('expandafter')
  local cur_let = token.create('let')
  local prefix = '^'..'^@'
  while token.is_defined(prefix..'expandafter') or token.is_defined(prefix..'let') do
    prefix = prefix..'^'..'^@'
  end
  local undefined = token.create(prefix..'expandafter')


tex.enableprimitives(prefix, {'expandafter', 'let'})
  local exp_after, let = token.create(prefix..'expandafter'), token.create(prefix..'let')
  token.put_next(lbrace, exp_after, rbrace)
  token.scan_toks(false, true)
  tex.runtoks(function()
    token.put_next(let, exp_after, undefined, let, let, undefined)
  end)
end
Tuff Contender
  • 401
  • 1
    So... what does the code inside this answer do exactly...? – user202729 Jul 02 '22 at 11:41
  • @user202729 token.scan_toks(false, true) scans a fully expanded <general text>. The code is essentially the first expansion of \expanded{\expandafter}\mymacro. – Tuff Contender Jul 02 '22 at 11:44
  • Okay I see. With the expandafter the net result will be that the following tokens in the input stream will be expanded exactly once. (so e.g. before the expand() call, the following tokens in the input stream are \mymacro \par \end{document}; after the expand() call, the following tokens will be ABC \par \end{document} (space characters are only for illustration) Not sure if that's what token.expand() is meant to do (since it's broken), but alright. – user202729 Jul 02 '22 at 11:45
  • Side note, it would be more robust to specify the catcode for the { and } tokens explicitly. – user202729 Jul 02 '22 at 12:35
  • @user202729 Thank you! Now I've change it a little to treat their catcodes carefully. – Tuff Contender Jul 02 '22 at 13:49
  • I didn't expect to handle undefined meaning of \expandafter though, but sure it works. // didn't test but the second code feels a bit weird • token.create cannot be used for tokens not in the hash table so you need to enableprimitive before...? • the cur_* tokens defined initially is undefined • I'm not sure if that ^^@ will be interpreted as-intended from Lua side, maybe string.char(0)? – user202729 Jul 03 '22 at 00:59
  • @user202729 It is literally ^^@. Once a token is added to the hash table, it is never removed, so token.create always returns a valid token. But after \let\expandafter\relax, token.create'expandafter' gets a token meaning \relax. – Tuff Contender Jul 03 '22 at 02:36
  • Okay, I think the unexpected part here is that undefined is the permanently undefined token (relies on token.create returns that token instead of the "expected" token when it's not in the hash table); nevertheless if the token \^^@expandafter is previously in the hash table but undefined, then after the function is called \^^@expandafter's meaning will remain \expandafter instead of undefined. (just for academic purposes. Obviously normally assuming \expandafter is the built-in should be is sufficient) – user202729 Jul 03 '22 at 03:30
  • @user202729 \expandafter's meaning is never changed. \^^@expandafter's state is changed as "not in the hash table" -> "primitive \expandafter" -> entry in hash table pointing to a eqtb item meaning "undfined control sequence". – Tuff Contender Jul 03 '22 at 03:51
1

Alternatively it could be implemented with tex.runtoks like this

tex.runtoks(function()
    tex.sprint {token.create "expandafter"}
end)

This relies on some implementation details of LuaTeX however.

Explanation: tex.runtoks takes a function that prints out some tokens to be executed immediately; internally it's done by putting ⟨the tokens⟩ ⟨internal \endlocalcontrol token⟩ in front of the input stream, executing a "local run".

So the input stream would have the format

\expandafter ⟨\endlocalcontrol⟩ ⟨tokens in the input stream⟩

First \expandafter is expanded which expands the tokens. Then ⟨\endlocalcontrol⟩ is executed which exits tex.runtoks.

Note that the token represented by ⟨\endlocalcontrol⟩ is an internal token which might show up as BAD depends on the circumstances. It has the same meaning as the normal \endlocalcontrol token however.

user202729
  • 7,143