Building a token list in lua

Question

In TeX, a 'write' node is inserted with, say:

\write1{\string\doit{\the\lastypos}}

With pure luatex, a node could be created with:

local n = node.new(8, 1)
n.stream = 1
n.data = <token-list>

According to the manual, the <token-list> is a table representing the token list to be written (with a list of triplets). I couldn't find any documentation about how this list is built. I discovered a string is accepted, but it gets converted to a string'ed list of chars (much like \meaning), so \the\lastypos is written verbatim, not evaluated.

I've found a workaround, shown in the following piece of code:

\setbox0\hbox{\write1{\the\lastxpos}}

\directlua{

  for _,d in ipairs(tex.box[0].head.data) do
    texio.write(' ** ' .. d[1] .. '/' .. d[2] .. '/' .. d[3])
  end

}

I define a box with a \write and then inspect the node. In the real code, instead of printing it I pass it to n.data and primitives work as expected (with some problems in user defined macros).

My question is: how to generate in lua the token list to feed the data field? [Edit. Please, note my questions is not about \lastypos, but about building an arbitrary token list for the data field. Remember also that, because of the asynchronous nature of TeX, page numbers and the like are not known when the 'write' node is created, only when the 'write' is actually output.]

Here is a latex file to make some experiments, with a lua file named extra.lua:

\documentclass{article}

\def\donothing#1{}

\directlua{
  require'extra'
}

\setbox0\hbox{\write1{\string\donothing{\the\lastypos}}}

\begin{document}

\directlua{

for _,d in ipairs(tex.box[0].head.data) do
  texio.write(' +++ ' .. d[1] .. '/' .. d[2] .. '/' .. d[3])
end

}

\copy0
\copy0
\copy0

\end{document}

The lua file:

local n = node.new(8, 1)
n.stream = 1
n.data =  'abcd#&\\the\\lastxpos' 

for _,d in ipairs(n.data) do
  texio.write(' *** ' .. d[1] .. '/' .. d[2] .. '/' .. d[3])
end

you can use tex.lastypos to avoid needing to tex-expand \the\lastypos and put the value straight in the whatsit. — David Carlisle, Mar 07 '19 at 15:19
\lastypos is just an example. There is no \savepos in the examples after all. — Javier Bezos, Mar 07 '19 at 15:33
@DavidCarlisle Furthermore, this would write the value of lastypos when the whatsit is created, not when it's output, after a save_pos node. — Javier Bezos, Mar 07 '19 at 15:48
sure but you could arrange the timing differently, but as you say without any savepos in the example not clear what the timing should be. — David Carlisle, Mar 07 '19 at 16:07

score 5 · Accepted Answer · answered Mar 07 '19 at 17:26

LuaTeX has the token.create function to create a token uservalue. They can be combined into a token list by putting them into a table. For \string\donothing{\the\lastvpos} this would be:

tl = {
  token.create'string',
  token.create'donothing',
  token.create(string.byte'{'),
    token.create'the',
    token.create'lastypos',
  token.create(string.byte'}')
}

Normally the references to tokenlists in the LuaTeX documentation mean this kind of table, but you need a different kind: A table of tables of numbers. Finding these numbers isn't easy, but you can convert token lists in the format above into this other format (Here I am using a little trick: {0, v.tok} is interpreted in the same way as if we would have split v.tok properly into three parts):

\directlua{
local function convert_tl(list)
  local new = {}
  for i,v in ipairs(list) do
    new[i] = {0, v.tok}
  end
  return new
end

local n = node.new(8, 1)
n.stream = 3
n.data = convert_tl{
  token.create'string',
  token.create'donothing',
  token.create(string.byte'{'),
    token.create'the',
    token.create'lastypos',
  token.create(string.byte'}')
}

tex.box[0] = node.hpack(n)
}
\copy0
\copy0

results in the output

\donothing{0}
\donothing{0}

Wow! I wouldn't say the v.tok is a little trick. But my first tests worked, indeed. — Javier Bezos, Mar 07 '19 at 17:58
@JavierBezos A little warning: Do not rely too much on reading the tokenlist, newer LuaTeX version only provide the expanded string if you try to read from .data. — Marcel Krüger, Mar 07 '19 at 18:20
Bad news, except if you can assign a string to data (which in turn gets tokenized). Otherwise something like x.data = y.data won't work, which is very very counterintuitive. — Javier Bezos, Mar 07 '19 at 18:33
A little question: do you know if a deep copy of a 'write' node will preserve the internal structure of the data field? — Javier Bezos, Mar 08 '19 at 18:09

Building a token list in lua

1 Answers1