2

TeX file:

\directlua{require "foo"}
\show\foo
\bye

and the contents of foo.lua:

token.set_macro("foo","\\a\\b\\u\\z")

Here is the result:

> \foo=macro:
->BADa\b \u BADz.
l.2 \show\foo

Why do some backslashes turn into BAD while others work like expected?

I think it might be a bug, but ask here first in case I have a fundamental misunderstanding.

Harald Hanche-Olsen
  • 15,968
  • You get BAD as \a is not defined in plain, while \b is. If you add \def\a{blub} it changes. This looks like a bug to me (and I have no idea how to get \ working). – Ulrike Fischer Dec 10 '19 at 16:05
  • @UlrikeFischer Actually the requirement is slightly weaker: The command sequences do not have to be defined, but they must have appeared at some point. E.g. adding \def\gobble#1{}\gobble{\a\z} would "fix" it too. That's basically what the LuaTeX manual is talking about in "10.6.6 Nota bene". – Marcel Krüger Dec 10 '19 at 16:22
  • @MarcelKrüger yes, I got this. But what about commands like \\ or \!? Whatever I do they end up as BAD!. – Ulrike Fischer Dec 10 '19 at 16:30
  • @UlrikeFischer It seems like the parser has a problem with non-letter control sequences. – Harald Hanche-Olsen Dec 10 '19 at 17:11
  • Related question (regarding how control sequences not in the hash table has surprising behavior: https://tex.stackexchange.com/q/335994/250119) – user202729 Jul 08 '22 at 07:27

2 Answers2

4

The token library pokes into tex internals with almost no checks.

if you do

\def\wibble{\a\z}

\directlua{require "foo"}
\show\foo
\bye

then you get

> \foo=macro:
->\a \b \u \z .

But if \a hasn't been seen by tex at all there is no hash table entry related to \a and nothing to set as the token in the replacement text for \foo

David Carlisle
  • 757,742
  • What can one do if one want \\ in the token list? In latex it is defined, but it still gives BAD when used in the set_macro argument. – Ulrike Fischer Dec 10 '19 at 16:22
  • @UlrikeFischer Specifying luatexbase.registernumber("CatcodeTableOther") as you showed me in chat does put backslash tokens (with catcode other, I am sure) in the list. I wanted control sequences, however … – Harald Hanche-Olsen Dec 10 '19 at 16:25
  • Thanks for the explanation. It appears, however, that I also get BAD tokens in lualatex if I include "\\\\", which is what I originally wanted to do. And that makes no sense, as \\ is known to latex. (I'll experiment some more.) – Harald Hanche-Olsen Dec 10 '19 at 16:28
  • @HaraldHanche-Olsen you might need to ask Hans... (I couldn't get it to work) – David Carlisle Dec 10 '19 at 16:30
  • Okay, I might. But now I have to go solve the original problem by moving it out of the lua realm into latex. – Harald Hanche-Olsen Dec 10 '19 at 16:39
2

Too long for a comment.

It is unlikely that this will be made working, see this comment in the source of lnewtokenlib.c:

this is a relating simple converter; if more is needed one can just use tex.print with a regular \def or \gdef and feed the string into the regular scanner;

0xC0000022L
  • 985
Henri Menke
  • 109,596