How do I define a new TeX register with arguments in LuaTeX?

Question

Some time ago I tried to implement the \pdfadjustinterwordglue from pdfTeX in LuaTeX using a callback. Getting the callback to work is pretty simple, but there is another major problem in the way which I was so far not able to solve. The culprit is in the way parameters for spacing are set, which is for example

\knbscode\font`r=1100

This is a regular TeX assignment and this can be implemented in Lua using

\def\knbscode{\directlua{

local fid = token.scan_int()
local chr = token.scan_int()
local str = token.scan_string()

knbscode[fid] = knbscode[fid] or {}
knbscode[fid][chr] = knbscode[fid][chr] or {}
local int = string.gsub(str,"=","")
knbscode[fid][chr][field] = tonumber(int)

}\fontid}

That is to say, I look ahead for an integer, which is the font ID, another integer which is a character, and “string”, which according the LuaTeX manual is

returns a string given between {}, as \macro or as sequence of characters with catcode 11 or 12

The mindful reader might already notice a problem here because \knbscode\font`r=1100abc should parse by the normal TeX rules but will fail in this case. Luckily the usage of register assignment is sort of regular in TeX and such a situation rarely occurs.

The next big problem is that registers are not write-only. They can also be read using:

\the\knbscode\font`r

Of course, in this case, \knbscode should still scan for the font ID and the character but not for the assignment. My idea to solve this was to overload \the to peek at the next token and if it is \knbscode, delegate to another function. On the Lua end this looks like this

local t = lua.get_functions_table()
t[1] = function()
    local next = token.get_next()
    if next.csname == "knbscode" then
        token.put_next(token.create("theknbscode"))
    elseif next.csname == "stbscode" then
        token.put_next(token.create("thestbscode"))
    elseif next.csname == "shbscode" then
        token.put_next(token.create("theshbscode"))
    else
        token.put_next{ token.create("normalthe"), next }
    end
end

and on the TeX end I define \the to refer to this Lua function

\luadef\the1

such that it can expand within a single step. Except that it doesn't. My new \the does not expand within a single step because instead of putting the number from the register in the input stream I put a token which has to be expanded again.

Maybe this approach is just wrong altogether, so the question is relatively general:

How do I define a new TeX register with arguments in LuaTeX?

I actually have asked the same question some time ago on the LuaTeX mailing list with a simplified example, so I just got the answer: “well, use a normal register, duh” (https://tug.org/pipermail/luatex/2019-January/007040.html).

---

Below I have a full MWE with which you can play around.

\documentclass{article}
\usepackage{luacode}

\newcount\pdfadjustinterwordglue

\begin{luacode}

local subtypes = node.subtypes("glue")

local knbscode = {}

local t = lua.get_functions_table()
t[1] = function()
    local next = token.get_next()
    if next.csname == "knbscode" then
        token.put_next(token.create("theknbscode"))
    elseif next.csname == "stbscode" then
        token.put_next(token.create("thestbscode"))
    elseif next.csname == "shbscode" then
        token.put_next(token.create("theshbscode"))
    else
        token.put_next{ token.create("normalthe"), next }
    end
end

function set_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()
    local str = token.scan_string()

    knbscode[fid] = knbscode[fid] or {}
    knbscode[fid][chr] = knbscode[fid][chr] or {}
    local int = string.gsub(str,"=","")
    knbscode[fid][chr][field] = tonumber(int)
end

function get_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()

    knbscode[fid] = knbscode[fid] or {}
    knbscode[fid][chr] = knbscode[fid][chr] or {}
    tex.sprint([[\numexpr]] .. (knbscode[fid][chr][field] or 0) .. [[\relax]])
end

local microtype_spacing = function(head, tail)
    head, tail, success = node.kerning(head, tail)
    if tex.count.pdfadjustinterwordglue > 0 then
        for space in node.traverse_id(node.id("glue"), head) do
            if subtypes[space.subtype] == "spaceskip" then
                local prev = node.prev(space)
                if prev.id == node.id("glyph") then
                    local knbs = knbscode[prev.font]
                    if knbs and knbs[prev.char] then
                        local f = font.getfont(prev.font)
                        local em = f.parameters.quad

                        local width = knbs[prev.char].width or 0
                        local stretch = knbs[prev.char].stretch or 0
                        local shrink = knbs[prev.char].shrink or 0

                        local glue = node.new(node.id("glue"))
                        glue.width = width*em/1000
                        glue.stretch = stretch*em/1000
                        glue.shrink = shrink*em/1000

                        head = node.insert_before(head, space, glue)
                    end
                end
            end
        end
    end
end

luatexbase.add_to_callback("kerning", microtype_spacing, "microtype_spacing")

\end{luacode}

\protected\def\knbscode{\directlua{set_knbscode("width")}\fontid}
\protected\def\stbscode{\directlua{set_knbscode("stretch")}\fontid}
\protected\def\shbscode{\directlua{set_knbscode("shrink")}\fontid}

\let\normalthe\the

\def\theknbscode{\normalthe\directlua{get_knbscode("width")}\fontid}
\def\thestbscode{\normalthe\directlua{get_knbscode("stretch")}\fontid}
\def\theshbscode{\normalthe\directlua{get_knbscode("shrink")}\fontid}

\luadef\the1

\pdfadjustinterwordglue=1

% Bogus values, just for demonstration
\knbscode\font`r=1100
\stbscode\font`r=10
\shbscode\font`r=10

\knbscode\font`r 1100
\stbscode\font`r 10
\shbscode\font`r 10

\begin{document}

\the\knbscode\font`r
\the\stbscode\font`r
\the\shbscode\font`r

\input lorem

\end{document}

Answer 1

tl;dr: I do not think defining TeX-like registers from Lua is possible, but you can get a bit closer.

Fixing \the is easy: You only have to trigger one level of expansion form Lua. You could try token.expand but that function has a unconventional understanding which token to expand. So it's easier to rely on the standard TeX tool to get one level of expansion: \expandafter: You can insert \expandafter\relax into TeX's input stream, then read the \relax back with expansion.

But there are other problems. Let's assume that we want a "g" to use the same value as a "f". In pdfTeX we could say

\knbscode\font`g=\knbscode\font`f

This wouldn't work with emulated registers like this. This is because of two problems:

Setting \knbscode does only accept strings Lua can convert into numbers and not general TeX numbers and \knbscode works with \the, but still isn't an internal TeX number which can be used in all places where TeX accepts numbers. The first problem can be fixed: To read an optional = followed by an integer, use

token.scan_keyword'=' -- Scans an optional "="
local int = token.scan_int()

The second problem is harder to fix. If you want your register-like Lua value to be usable at all places which accept TeX numbers, you would need some way to determine from Lua if TeX tries to read a register or not. This information isn't available, so I do not think that perfectly emulating a register is possible.

Of course you can say that it's nice to be at least similar, but sometimes having a slightly different interface for the Lua version is better than to have subtly, much harder to detect, differences in usage.

The suggested changes lead to this version:

\documentclass{article}
\usepackage{luacode}
\newcount\pdfadjustinterwordglue
\begin{luacode}
local subtypes = node.subtypes("glue")
local knbscode = {}
-- Let's make sure that no one messes with our primitives:
local function frozentok(name)
  local tok = token.create(name)
  return token.new(tok.mode, tok.command)
end
local the, expandafter, relax = frozentok'the', frozentok'expandafter', frozentok'relax'
local t = lua.get_functions_table()
t[1] = function()
    local next = token.get_next()
    if next.csname == "knbscode" then
        token.put_next(token.create("theknbscode"))
    elseif next.csname == "stbscode" then
        token.put_next(token.create("thestbscode"))
    elseif next.csname == "shbscode" then
        token.put_next(token.create("theshbscode"))
    else
        token.put_next{ next }
    end
    token.put_next{ expandafter, relax, the }
    token.scan_token() -- The scanned token will always be the relax we just wrote, but expandafter does it's magic in the progress.
end
function set_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()
    token.scan_keyword'='
    local int = token.scan_int()
knbscode[fid] = knbscode[fid] or {}
knbscode[fid][chr] = knbscode[fid][chr] or {}
knbscode[fid][chr][field] = int

end
function get_knbscode(field)
    local fid = token.scan_int()
    local chr = token.scan_int()
knbscode[fid] = knbscode[fid] or {}
knbscode[fid][chr] = knbscode[fid][chr] or {}
tex.sprint([[\numexpr]] .. (knbscode[fid][chr][field] or 0) .. [[\relax]])

end
local microtype_spacing = function(head, tail)
    head, tail, success = node.kerning(head, tail)
    if tex.count.pdfadjustinterwordglue > 0 then
        for space in node.traverse_id(node.id("glue"), head) do
            if subtypes[space.subtype] == "spaceskip" then
                local prev = node.prev(space)
                if prev.id == node.id("glyph") then
                    local knbs = knbscode[prev.font]
                    if knbs and knbs[prev.char] then
                        local f = font.getfont(prev.font)
                        local em = f.parameters.quad
                    local width = knbs[prev.char].width or 0
                    local stretch = knbs[prev.char].stretch or 0
                    local shrink = knbs[prev.char].shrink or 0

                    local glue = node.new(node.id("glue"))
                    glue.width = width*em/1000
                    glue.stretch = stretch*em/1000
                    glue.shrink = shrink*em/1000

                    head = node.insert_before(head, space, glue)
                end
            end
        end
    end
end

end
luatexbase.add_to_callback("kerning", microtype_spacing, "microtype_spacing")
\end{luacode}
\protected\def\knbscode{\directlua{set_knbscode("width")}\fontid}
\protected\def\stbscode{\directlua{set_knbscode("stretch")}\fontid}
\protected\def\shbscode{\directlua{set_knbscode("shrink")}\fontid}
\def\theknbscode{\directlua{get_knbscode("width")}\fontid}
\def\thestbscode{\directlua{get_knbscode("stretch")}\fontid}
\def\theshbscode{\directlua{get_knbscode("shrink")}\fontid}
\luadef\the1
\pdfadjustinterwordglue=1
% Bogus values, just for demonstration
\knbscode\fontr=1100 \stbscode\fontr=10
\shbscode\font`r=10
\knbscode\fontr 1100 \stbscode\fontr 10
\shbscode\font`r 10
\begin{document}
\the\knbscode\fontr \the\stbscode\fontr
\the\shbscode\font`r
\input lorem
\end{document}