\expandafter with \uppercase ordering

Question

As I understand it, the behavior of \expandafter is such that if \a and \b are macros, then \expandafter\a\b will first expand \b then \a.

But my understanding must be wrong, because I don't understand why the following commands

\def\xx{xx}
\expandafter\uppercase{\xx}
\bye

produce "xx" instead of "XX".

After Googling around, it seems that the right way to do this is to replace the second line with

\uppercase\expandafter{\xx}

which indeed works. But \expandafter takes in two arguments; if the first argument is \xx is the first argument, then what is the second?

In any case, since in this case \xx needs to be expanded after \uppercase, why doesn't the first version work -- how can the ordering of the commands be explained?

Answer 1

The \expandafter primitive expands after exactly one token: it does not take two arguments. As such

\expandafter\uppercase{\xx}

skips over \uppercase and tries to expand {, which is unexpandable and so nothing happens. Thus this is entirely identical to simply writing

\uppercase{\xx}

Thus in general what you have to do to expand the argument after a control sequence is to use two \expandafters

\expandafter\uppercase\expandafter{\xx}

That would be required if \uppercase where a macro, but here we are talking about a primitive. Like several other TeX primitives, \uppercase has a formal syntax which allows expansion before the opening brace. Probably a more common case is something like a toks assignment, where

\def\xx{xx}
\toks0=\expandafter{\xx}
\showthe\toks0

gives

> xx.

The fact that these primitives accept <general text> is allowed is detailed formally in The TeXbook and other reference works (I like TeX by Topic).

Answer 2

Fully discussing the topic would be really too long, so I'll describe some general facts to give the flavor.

Some TeX primitives require very specific tokens to follow them. To make some examples, I'll mention \over that can be followed by anything legal in math mode and \uppercase that, instead, wants to see after it a <general text>. So in the case of \over, no expansion is made immediately, but only after TeX has started a new math list for the denominator. On the other hand, a <general text> is defined as

<filler>{<balanced text><right brace>

[276] (boldface numbers in brackets will refer to a page of the TeXbook), which implicitly means that \uppercase triggers expansion in order to find { (an explicit or implicit character with category code 1). The <filler> is an arbitrary sequence of space and \relax tokens that will be removed. An example of this is [374]

\uppercase\expandafter{\romannumeral\n}

that produces the representation of the value stored in \n in uppercase Roman numerals. Another example is in Get the lion to run in loops. Tersely.

Another example is \left that the syntax rules require to be followed by a <delim> [292], which in turn is [289]

<delim> → <filler>\delimiter<27-bit number> | <filler><letter> | <filler><otherchar>

(in this case, the <letter> or the <otherchar> should have a nonnegative \delcode, of course). Thus something like \left\space[ will work, because \left is looking for a specific token (either \delimiter or a character token with category code 11 or 12) and will expand \space and then ignore the result which fits as <filler>. A similar situation happens with _ and ^ (see Why does `x_\text y` work?).

One should always have in mind the list of places where TeX doesn't expand tokens, which is found at [215]. So, for instance, TeX will not expand tokens when doing \def or \let. The syntax of \let allows <one optional space> after the (optional) = (more precisely <equals>), but it doesn't do expansion: so

\let\foo=\space\relax

will make \foo equivalent to \space, not to \relax. This is because \let can have any token after <equals>, not a specific one.

The “specific token” required by a primitive can be fairly general; for instance, \the allows a very large set of tokens after it, and it will expand tokens looking for them. The construction

\toks@=\expandafter{\the\toks@ x}

will append x to the previous contents of \toks@. What if we want to append the contents of another token register, say \mytoks? Easy:

\toks@=\expandafter{\the\expandafter\toks@\the\mytoks}

Explanation: the outer \expandafter will trigger the first \the; this action is performed because in a token register assignment TeX is looking for {, so it expands tokens as it goes; now \the wants to see a specific token, so it expands in order to find it; since \expandafter is expandable (with empty expansion), it triggers it, which will expand the second \the. A more complicated version is

\edef\x{\the\toks@\the\mytoks}\toks@=\expandafter{\x}

or

\begingroup\edef\x{\endgroup\toks@={\the\toks@\the\mytoks}}\x

(the latter will not leave a definition for \x).

The construction \expandafter\uppercase{\xx} will do nothing different from \uppercase{\xx} because \expandafter will try to expand {, which is unexpandable. There is no general way to force a one level expansion of more than one token, which wouldn't make much sense anyway. If you have to uppercase a list of characters possibly hidden in macros, then only full expansion makes sense:

\begingroup\edef\x{\endgroup\uppercase{<tokens>}}\x

(again for not leaving behind a definition for \x) or

\begingroup\edef\x{<tokens>}\uppercase\expandafter{\expandafter\endgroup\x}

A simpler command using a scratch macro would be

\edef\next{<tokens>}\uppercase\expandafter{\next}

but I'd not recommend it (troubles are behind the corner if by chance \next had previously be \let to an unexpandable token). In LaTeX one should use \protected@edef, instead of \edef.