8

I know \expandafter can skip the token immediately after it, therefore I try to use it with some combinations of } , \toks and definitions with \def and \let.

The main question the puzzles me is why consecutive uses of \toks like \the\toks0\the\toks0 will extend \expandafter's effect? In theory should it not be last only to the first \the\toks0?

I tried with this:

\toks0{()}
\def\asd{()}
%
1{\def\asd{\TeX}}\asd \par
2{\let\asd\TeX}\asd\par
3{\toks0{\TeX}}\the\toks0\par
4{\toks0{\TeX}}\the\toks0\relax\par
5{\toks0{\TeX}}\the\toks0\the\toks0\relax\par
6{\toks0{\TeX}}\the\toks0\the\toks0\the\toks0\relax\par
7{\toks0{\TeX}}\the\toks0\relax\the\toks0\par
8{\def\asd{\TeX}\expandafter}\asd\par
9{\def\asd{\TeX}\expandafter}\asd\asd\par
10{\let\asd\TeX\expandafter}\asd\par
11{\let\asd\TeX\expandafter}\asd\asd\par
12{\toks0{\TeX}\expandafter}\the\toks0\par
13{\toks0{\TeX}\expandafter}\the\toks0\relax\par
14{\toks0{\TeX}\expandafter}\the\toks0\the\toks0\par
15{\toks0{\TeX}\expandafter}\the\toks0\the\toks0\the\toks0\par
16{\toks0{\TeX}\expandafter}\the\toks0\relax\the\toks0\par
17{\toks0{\TeX}\expandafter}\the\toks0 \the\toks0\par
%
\bye

And the output is like this:

Code output

user13892091
  • 83

2 Answers2

10

After \toks TeX will expand following tokens looking for a <number> to determine which token register to use, so

\the\toks0\the\toks0

will expand the second \the first (while looking for more digits to end the number started by the first 0 token).

As you show, if you have a non expandable token that terminates this <number> the second \the\toks is not expanded until later in the process so for example your test

\the\toks0\relax\the\toks0\par

Note this avoidance of accidentally causing premature expansion due to literal digits is one reason (other than speed) for using numeric tokens such as \z@. Unlike 0, \z@ is a complete <number> so does not force expansion of following tokens looking for more digits. See What does \z@ do?.

Phelype Oleinik
  • 70,814
David Carlisle
  • 757,742
5

Let's look at line 14

{\toks0{\TeX}\expandafter}\the\toks0\the\toks0\par

The expected output is indeed the same as of \TeX\TeX.

You're setting \toks0 in a group, but the \expandafter before } causes the expansion of \the before the group ends. Now \the expands by looking at what comes after it (see The \the command), in this case \toks, so TeX knows it has to scan a number to use the required register. This is still part of the expansion of \the, so TeX finds 0 and, by general rule, it looks for further digits expanding tokens as it goes. Since 0 is followed by \the, this is expanded; the following token is \toks, but since the expansion might reveal other digit, this is again carried on; the number is surely terminated, because the following token is \par. So now you get the current contents of \toks0; now TeX expands \TeX, but this doesn't start with a digit, so TeX is able to decide that the first number was actually 0 and it delivers the current contents of \toks0 (still \TeX). Only after this process the expansion of \expandafter ends, the group is closed and the input stream has

\TeX T\kern -.1667em\lower .5ex\hbox {E}\kern -.125emX\par

In line 17, the space after 0 stops the search for further digits (and is ignored), so you get

\TeX\the\toks0

where \toks0 has the initial value because the group in which it was set has already been ended.

Let's now look at the other lines. In 1–7, the scope of the redefitions ends with }, so you get the value of \asd or \toks0 valid before the group started.

In line 8, \asd is expanded before } is scanned, so the value current at expansion time is used, hence you get \TeX. In line 9 the first \asd after } is expanded before the group ends, but the second only when the group has ended, so you get \TeX().

The difference between line 8 and 10 is that the input stream will contain }\TeX\par in the first case and

}T\kern -.1667em\lower .5ex\hbox {E}\kern -.125emX\par

in the second case, but the final output will be the same.

When 0 is followed by \relax, the scan for further digits ends, so the behavior of line 14 doesn't happen.

Why does TeX scan for further digits? After all 0 is already a number and usually no further digits are allowed. But in the TeX language 01 is an integer denotation and its value is 1. Any amount of leading zeros is allowed to denote an integer (or the numeric part of a dimension).

egreg
  • 1,121,712