Obtaining the name of an undefined control sequence in LuaTeX

Question

I want to implement something similar to \detokenize as a Lua function. But there is one case where I am prevented from accomplishing my goal.

\directlua{tex.sprint("[" .. token.get_next().csname .. "]")}\par
\directlua{tex.sprint("[" .. token.get_next().csname .. "]")}\UNDEFINED

The first line above outputs [par] and the other outputs []. Replacing token.get_next().csname by token.scan_csname() doesn't help. How do I obtain the name of an undefined control sequence in LuaTeX?

Answer 1

Note that you don't get an empty string because \UNDEFINED is not defined but because it has never been scanned by TeX (so there is no internal token of that name)

this plain luatex

\long\def\z#1{%
\directlua{texio.write_nl("[" .. token.scan_csname() .. "]")}#1%
\directlua{texio.write_nl("[" .. token.get_next().csname .. "]")}#1%
}

\z\par
\z\UNDEFINED

\bye

produces a log

[par]
[par]
[UNDEFINED]
[UNDEFINED]

showing that just having \UNDEFINED being seen by TeX's #1 argument scanner is enough to get the behaviour you expected.

Answer 2

One possible way to get is to use the expand capability of scan_toks() to expand/execute some following tokens, similar to this answer.

In the example below,

• function f does a simple token.get_next() (which does not handle the case the following token is not in the hash table)
• function g uses scan_toks(false, true) to expand the input, and \immediateassignment combined with \futurelet to force TeX to tokenize the next token.
• function h uses coroutine trick _{(although instead of a coroutine, auxiliary functions are used)} to call the \futurelet instead. \immediateassignment can be added if it needs to work in an expansion-only context; nevertheless in an o-expansion context it will still fail to work.

%! TEX program = lualatex
\documentclass{article}
\usepackage{luacode}
\ExplSyntaxOn
\use_none:n {__unused}  % put the token __unused into the hash table
\cs_new_protected:Npn __h_aux { \directlua{h_aux()} }
\ExplSyntaxOff
\begin{luacode*}

print("\n\n\n\n")

function f()
    print("f: csname =", token.get_next().csname)
end

function g()
    token.put_next {
        token.create(string.byte "{", 1),
        token.create "immediateassignment",
        token.create "futurelet",
        token.create "__unused",
        token.create(string.byte "}", 2),
    }
    token.scan_toks(false, true)
    print("g: csname =", token.get_next().csname)
end

function h()
    tex.sprint {
        token.create "futurelet",
        token.create "__unused",
        token.create "__h_aux",
    }
end

function h_aux()
    print("h: csname =", token.get_next().csname)
end

\end{luacode*}


% normal behavior: tokens never seen before (not in the hash table) results in empty csname
\directlua{f()}\par
\directlua{f()}\undefined
\directlua{f()}\undefineda
\directlua{f()}\undefineda
\directlua{f()}\undefinedb

% use function g instead
\directlua{g()}\par
\directlua{g()}\undefined
\directlua{g()}\undefineda
\directlua{g()}\undefineda

% just checking, undefinedb is still not in the hash table
\directlua{f()}\undefinedb

% use function h instead
\directlua{h()}\par
\directlua{h()}\undefined
\directlua{h()}\undefinedb
\directlua{h()}\undefinedb

\begin{luacode*}
print("\n\n\n\n")
\end{luacode*}
\begin{document}
\end{document}

The output is

f: csname =     par
f: csname =     undefined
f: csname =
f: csname =
f: csname =
g: csname =     par
g: csname =     undefined
g: csname =     undefineda
g: csname =     undefineda
f: csname =
h: csname =     par
h: csname =     undefined
h: csname =     undefinedb
h: csname =     undefinedb

You can see that function g() and h() gets the csname correctly, being \undefineda and \undefinedb.