The puzzling thing about \newtoks and the token list

Question

The code is:

\documentclass{article}
\newtoks \test
\begin{document}
\test={123}
\test=\test{666}
\test=\test{hhh}
\the\test


\end{document}

and the result is: 666 hhh 123, as shown in the picture below.

My question is: why the result is "666 hhh 123" instead of "123 666 hhh" and why there are spaces between 666 and hhh, as well as hhh and 123?

I'm a beginner in TeX and LaTex and I've got some knowledge about tokens, token lists and \newtoks, but the behavior of the above code really confuses me. What is going on? I hope the experts can give me some advice, and I'd like to thank you in advance.

Answer 1

The code is:

\documentclass{article}
\newtoks \test
\begin{document}
% this assigns 123 to \test and makes a space token (ignored in v mode)
    \test={123}
% this assigns \test to \test (no-op) typesets 666 in a group then a non ignored space
    \test=\test{666}
% this assigns \test to \test (no-op) typesets hhh in a group then a non ignored space
    \test=\test{hhh}
% thistypesets the contents of \test which is 123
    \the\test
\end{document}

You don't say, but I guess you are tying to append to the token list

\documentclass{article}
\newtoks \test
\begin{document}
\test={123}%%%
\test=\expandafter{\the\test 666}%%%
\test=\expandafter{\the\test hhh}%%%
\the\test


\end{document}

Answer 2

After \newtoks\test you can assign something to this token register using the syntax

\test={123}

(the = is optional); the tokens you write between braces should be balanced with respect to {...}.

You can ask TeX to deliver the contents of the register using

\the\test

but there is no direct provision for appending tokens to the already stored contents. One has to exploit the way such assignments are performed. After \test= TeX will expand tokens in order to look for the { that delimits the token list to be assigned (and TeX will ignore space tokens and \relax during the process, but this is just a technicality).

In particular, if you want to append tokens, you can do

\test=\expandafter{\the\test abc}

because TeX will do expansion and \expandafter will act on the token immediately following {, which in this case is \the that will deliver the contents of \test.

If you want to prepend tokens, you have to use a scratch token register: for instance

\toks0={uvw}
\test=\expandafter{\the\toks0\expandafter\space\the\test}

What does this do? As before, the first \expandafter triggers expansion of \the. Now \the finds the primitive \toks that needs to be followed by a number; 0 is found, but TeX will continue expansion until finding something that cannot be interpreted as a digit; it finds \expandafter that jumps over \space and expands \the\test. Then it will expand \space and the resulting token will be ignored because it follows 0, which will be the number TeX was looking for.

The \begingroup\edef\x{\endgroup...}\x method ensures that the definition of \x will disappear as soon as \x is expanded, so we're covering our tracks.

There is a simpler method for prepending tokens:

\test={123}
\toks0={uvw}
\begingroup\edef\x{\endgroup\test={\the\toks0 \the\test}}\x

This works because \edef will skip over unexpandable tokens, in this case \test, but expands \the; however, tokens resulting from \the<token register> will not be expanded further in \edef.

With e-TeX extensions (available with pdftex, xetex and luatex), you can do without a scratch token register:

\test={123}
\begingroup\edef\x{\endgroup\test={\unexpanded{uvw}\the\test}}\x

Of course you want to define macros for these jobs:

\def\append#1#2{#2=\expandafter{\the#1#2}}
\def\prepend#1#2{%
  \begingroup\edef\x{\endgroup
    #1={\unexpanded{#2}\the#1}%
  }\x
}
\newtoks\test
\test={123}
\append\test{abc}
\prepend\test{uvw}
\the\test
\bye

will print

uvw123abc

Answer 3

You are doing:

\test={123}       % the value 123 is saved to \test
\test=\test{666}  % \test=\test (assigning similar to a=a) is done,
                  % then 666 is printed in group
\test=\test{hhh}  % \test=\test is done and then hhh is printed in group
\the\test         % the value of \test, i.e. 123, is expandend and printed