How can I get the value of a token list (tl) variable in Lua?

Question

For example if I define a token list like this

\tl_set:Nn \l__tmpa_tl {content\someundefinedmacro}

Which function can I use to get the value of the token list?

Answer 1

Use function token.get_macro (which returns a string).

print(token.get_macro("l__tmpa_tl"))

---

For getting also the catcode information I come up with a convoluted way, but it seems that there isn't any better way:

(this method works in an expansion-only context, but it requires exiting to the TeX engine and back, so it cannot be wrapped in a function except if the caller is a coroutine)

%! TEX program = lualatex
\documentclass[12pt]{article}
\usepackage{luacode}
\begin{document}
\ExplSyntaxOn
% ======== Prepare an example \test token list (tl) for getting the value in Lua
\tl_set:Nx \test {
    \char_generate:nn {"41} {8}
    \char_generate:nn {"41} {11}
    \char_generate:nn {"41} {12}
}
\tl_put_right:Nn \test {
    {}$&#^$
    \somethingundefined
    \input
    \scantokens
    \q_nil
    \cs_set:Npn
    \A
    ~
}
\exp_args:NNo \tl_put_right:No \test {
    \char_generate:nn {A} {13} } \exp_args:NNo \tl_put_right:No \test { \char_generate:nn {~} {13}
}
% ======== Define the main function.
\begin{luacode}
function f()
    while true do
        local t=token.get_next()
        if t.csname then
            if t.csname=="q_stop" then
                break
            end
            print(string.format("cs %20s   active=(%s) cmdname=(%s) ", "'" .. t.csname .. "'", t.active, t.cmdname))
        else
            print("char", t.cmdname, string.char(t.mode))
        end
        --print(t.command, t.cmdname, t.tok, t.active, t.expandable, t.protected, t.mode, t.index)
    end
end
\end{luacode}
% ======== Define a helper TeX function.
\cs_set:Npn \callf {\directlua{f()}}
% ======== Actually call the function -- read the definition of the \test macro.
\exp_after:wN \callf \test \q_stop
\ExplSyntaxOff
\end{document}

Output:

char    sub_mark        A
char    letter          A
char    other_char      A
char    left_brace      {
char    right_brace     }
char    math_shift      $
char    tab_mark        &
char    mac_param       #
char    sup_mark        ^
char    math_shift      $
cs 'somethingundefined'   active=(false) cmdname=(undefined_cs)
cs              'input'   active=(false) cmdname=(call)
cs         'scantokens'   active=(false) cmdname=(input)
cs              'q_nil'   active=(false) cmdname=(call)
cs         'cs_set:Npn'   active=(false) cmdname=(long_call)
cs                  'A'   active=(false) cmdname=(undefined_cs)
cs                  '~'   active=(false) cmdname=(call)
cs                  'A'   active=(true) cmdname=(undefined_cs)
cs                  '~'   active=(true) cmdname=(call)

Of course there's also another way of using \tl_analysis_map_inline which is much shorter:

\begin{luacode*}
function f(s, charcode, catcode)
    print(s, charcode, catcode)
end
\end{luacode*}
\tl_analysis_map_inline:Nn \test {\directlua{f("\luaescapestring{\detokenize{#1}}", #2, "#3")}}

Result:

\exp_not:n {A}  65      8
\exp_not:n {A}  65      B
\exp_not:n {A}  65      C
\exp_after:wN {\if_false: }\fi:         123     1
\if_false: {\fi: }      125     2
\exp_not:n {$}  36      3
\exp_not:n {&}  38      4
\exp_not:n {##} 35      6
\exp_not:n {^}  94      7
\exp_not:n {$}  36      3
\exp_not:n {\somethingundefined }       -1      0
\exp_not:n {\input }    -1      0
\exp_not:n {\scantokens }       -1      0
\exp_not:n {\q_nil }    -1      0
\exp_not:n {\cs_set:Npn }       -1      0
\exp_not:n {\A }        -1      0
\exp_not:n {\~} -1      0
\exp_not:n {A}  65      D
\exp_not:n {~}  126     D