tex.hashtokens incomplete?

Question

This came up while trying to answer another question, and concerns tex.hashtokens() in LuaTeX. Consider the following document:

\documentclass{beamer}
\begin{document}
\message{The meaning of (\string\beamertemplatenavigationsymbolsempty) is (\meaning\beamertemplatenavigationsymbolsempty)}
\directlua{
for i,v in pairs(tex.hashtokens()) do
   if v:find('beamertemplatenavigation') then
      print('found:', v)
   end
end
}
\message{The meaning of (\string\beamertemplatenavigationsymbolsempty) is still (\meaning\beamertemplatenavigationsymbolsempty)}
\end{document}

When run with lualatex on my system (“This is LuaHBTeX, Version 1.12.0 (TeX Live 2020)”), this prints (with a newline I added for clarity):

The meaning of (\beamertemplatenavigationsymbolsempty) is (macro:->\setbeamertemplate {navigation symbols}{})
found:  beamertemplatenavigationsymbolsvertical
found:  beamertemplatenavigationsymbolshorizontal
 The meaning of (\beamertemplatenavigationsymbolsempty) is still (macro:->\setbeamertemplate {navigation symbols}{})

The macro \beamertemplatenavigationsymbolsempty clearly exists, so why doesn't the Lua code find it? Have I misunderstood something?

I get similar results with MiKTeX 20.11.

Answer 1

LuaTeX (as well as all other TeX versions derived from web2c after 1995) use a hash table separated into two parts: A normal hash table with up to hash_size (65536) entries and an additional part with hash_extra many entries (hash_extra can be set at runtime, TeXLive sets it to 600000 by default). (The hash_extra part is not really a hash table but more of a linear overflow list, but that's irrelevant right now)

The function tex.hashtokens returns a table which contains only the entries of the "normal" hash table and not the hash_extra part. So in some sense it lists the control sequence names which can be accessed particularly fast, not all of them. I don't know if this behavior is intentional, but I personally think that returning all names would be more useful. In any case, as ShreevatsaR mentioned you can force tex.hashtokens to return all names in a particular run by running LuaTeX as hash_extra=0 lualatex foo.tex and therefore forcing the hash_extra part to be empty. In this case, you can't use more than 65536 different control sequences though.