5

Many of TeX's primitives require a <general text> as their argument (e.g. \uppercase), which is defined as (TeXbook p. 276)

<general text> -> <filler>{<balanced text><right brace>

where <filler> is an arbitrary sequence of spaces and \relax commands. Now I would like to find out in what way it is useful that such a <filler> can be inserted.

Could someone give an example where the presence or absence of a <filler> makes any difference, or, if such a situation shouldn't exist, explain why the fact that a <filler> could be inserted is advantageous?

Edit: In the original version of the question, I claimed that

[...] constructs such as

\uppercase\expandafter{\romannumeral ...

are already made possible by the fact that the left brace may be implicit, so that TeX is expanding to find one.

Thanks to a discussion with frougon, I have come to realize that this is a misunderstanding. Here is what I currently believe to be true:

  • TeX really just always expands expandable tokens whenever expansion is not inhibited (i.e., except from the situations in TeXbook p. 215)
  • \expandafter is expandable and expansion (immediately) after \uppercase and the like is not inhibited (only after the opening brace was found), which is the full reason for why expansion takes place in this situation
  • the syntax rules are not an inherent property of the language that could be found explicitly in tex.web; in fact, syntactic correctness (only spaces and \relax tokens allowed before the {) matters only once something unexpandable was found

As a consequence:

  • the fact that the syntax requires a (possibly empty) <filler> and a (possibly implicit) opening brace has nothing to do with whether or not things are expanded
  • instead, the fact that a <general text> allows an implicit brace and \def requires an explicit one merely emerges from the phenomenon that expansion is inhibited after a \def, but not before the <balanced text> of a <general text>

Basically it's not the case that implicit braces in the syntax rules imply expansion, but the other way around: Expansion implies that the braces may be implicit. Also, implicit braces are defined as control sequences that were \let or \futurelet to a category 1 token, not just anything expanding to one, so it wouldn't make sense to say that TeX fully expands tokens in order to find a brace as I originally thought.

This does not affect the question itself, though; it just illustrates how I was mistaking the descriptive nature of the grammar rules for a normative one.

wave
  • 841
  • Maybe it is to allow constructs like \toks0=\csname maybeexpandafter\endcsname{\foo}. If \maybeexpandafter is not defined, the \csname will turn it into \relax. – Henri Menke Mar 21 '20 at 06:18
  • @HenriMenke interesting idea. correct me if I'm wrong, but that seems like a rare thing to do to me, or at least too rare to justify the inclusion of such a rule based on that alone (as there are other ways to achieve the same effect). – wave Mar 21 '20 at 06:34
  • No idea, I've just made this up. I've not seen such a construct yet. – Henri Menke Mar 21 '20 at 06:37
  • Did you see https://tex.stackexchange.com/a/44459/4427 ? – egreg Mar 21 '20 at 10:43
  • @egreg I did wonder when writing my answer if I should mention the \relax inserted by incomplete if constructs but I couldn't think of a plausible looking example where you could generate a relax from an incomplete if token in a filler position:-) – David Carlisle Mar 21 '20 at 10:46
  • @egreg I did, but I guess I don't really see how it implies any kind of usefulness of that syntactic feature. I thought that \if\lowercase\fi leads to an insertion of \relax simply because \lowercase is unexpandable and \fi is encountered before two unexpandable tokens were found, right? – wave Mar 21 '20 at 11:04

3 Answers3

8

Space is perhaps easier to justify as it makes some tex primitives that take braced arguments act a little bit more like macros where \frac {1} {2} the space token before the second brace is ignored. It is also why spaces after = in \count0 = 2 are ignored, however in both cases there are perhaps more natural places in the grammar where spaces could have been absorbed.

\relax is harder to justify but I would guess it is for

\insert <8 bit number> <filler> {...}

where it means that if you have a habit or a macro that always terminates numbers with \relax you can have

\insert 250\relax {...}

But to be honest it mostly looks like an early syntax idea from the start of the TeX design that doesn't really make sense the way TeX finally came out.

Quite often the tex source (tex.web) is more informative than the texbook for this sort of thing but the word filler does not appear and all it says that I can see is

@ The |scan_left_brace| routine is called when a left brace is supposed to be
the next non-blank token. (The term ``left brace'' means, more precisely,
a character whose catcode is |left_brace|.) \TeX\ allows \.{\\relax} to
appear before the |left_brace|.

which doesn't give much of a clue as to why \relax is allowed.

David Carlisle
  • 757,742
  • I guess it is safe to say that the <filler> is really not a (useful) syntactic feature rather than a harmless addition making TeX complain less if, as you say, you have a habit of inserting \relax in various places – wave Mar 22 '20 at 19:38
4

The grammar given in The TeXbook documents what the program (TeX) does, rather than the program being written to match a specific grammar. (Although I can't definitively prove this, this is fairly clear (IMO) from looking at both; also consider these stories.) So as pointed out by David Carlisle's answer, the relevant part seems to be scan_left_brace in the TeX program's source code (available as a book, with texdoc tex, or online):

Sections 403 and 404

So your question seems to boil down to why scan_left_brace (as called from scan_toks, which is called when acting on \uppercase etc.) ignores both spaces and \relax tokens. Some clues are available in TeX's change log (available with texdoc errorlog or online) (to make sense of it see also Notes on the Errors of TeX and the paper The Errors of TeX, reprinted with additions and corrections as Chapters 10 and 11 of Literate Programming). If you look for the relevant sections 403 and 404, you see:

  • [19 May 1978] Change 251 [Algorithmic Anomaly]. Skip past blanks in the scan_math procedure. [This blank-skipping will eventually go into scan_left_brace.] [Affects §403.]

  • [5 Mar 1981] Change 498 [Cleanup for Consistency]. Allow optional space before a required left brace, e.g., \if AA {...}. [See Change 251.] [Affects §403.]

  • [28 May 1983] Change 699 [Cleanup for Consistency]. Ignore ‘\relax’ as if it were a space, in math mode and in a few other places where \relax would otherwise be erroneous. [Affects §404.]

(FWIW, The TeXbook's preface is dated June 1983; it was first published in 1984.)

I think this may give some clue towards the question (if I understood it). At least for spaces, there are many examples for which it makes sense. When it comes to \relax, note that just like the §404 "Get the next non-blank non-relax non-call token" there's another §406 "Get the next non-blank non-call token" where \relax is not ignored, and it's used in another bunch of places (like scan_optional_equals). So you might ask why each of those places chooses to ignore \relax rather than not, and it may be hard (for me) to explain each of those choices (though it must have been a conscious choice as it would have been so easy to choose otherwise).

ShreevatsaR
  • 45,428
  • 10
  • 117
  • 149
  • 1
    so after some further research it seems quite clear to me that there is a pattern as to when §404 is used and when it isn't; it turns out that it occurs pretty much everywhere where TeX does not expect a single token or a numeric constant, which are both pretty reasonable of course. – wave Mar 22 '20 at 19:30
2

Based on the helpful answers by David Carlisle and ShreevatsaR, I have a tentative suggestion to make:

If TeX expects a specific syntactical quantity, it ignores \relax tokens unless there are good reasons not to ignore them.

Let me demonstrate this by evaluating in which situations a \relax is ignored (after possible expansion) and in which it is not. First of all, here are the situations mentioned in the TeXbook where a <filler> may occur:

  1. before a (due to expansion possibly implicit) left brace, for example in a <general text>, a <box specification>, after \insert, \vadjust and \noalign;
  2. when assigning a <token variable> another <token variable> (or a <general text>, as above);
  3. in math mode as part of a <math field> or a <delim>.

However, going through the list of situations where TeX's §404 "Get the next non-blank non-relax non-call token" is called (which can be seen in ShreevatsaR's answer), there are actually a few more, that as far as I know are not mentioned in the TeXbook:

  1. before any <box> and <box or rule> (which only occurs in <leaders>);
  2. after any \global, \long and \outer prefix;
  3. in <leaders> before the <horizontal skip> or <vertical skip>.

By looking at the syntax rules, I can only find a few situations in which a \relax actually is not ignored or even disallowed:

  1. when a single <token>, including a <control sequence> is expected (such as after \show, \afterassignment, \let, \def, or \chardef);
  2. as part of a <file name>;
  3. when something is expected that may consist of <digit>s (or <hex digit>s or <octal digit>s) and/or TeX's keywords (pt, at, by and so on), which includes <number>, <dimen>, <mudimen>, <glue>;
  4. when nothing specific is expected (e.g. when reading the next command, some sort of <material> or parameter and replacement texts).

And here are the good reasons I can think of for each of those points:

  1. \relax might be the argument of the preceding command in this case.
  2. Being unexpandable, it can only serve as a delimiter for the filename.
  3. One of the most important uses of \relax is to delimit numbers, as in \def\a{\count0=3\relax} \a0 to prevent that a value of 30 is assigned. This functionality would be lost if \relax could be part of such an assignment. Now the crucial part is that \relax is significant whenever a numeric constant may come next, not only when there actually is one. This is why for two \toksdef tokens \a and \b, both \a=\b and \a=\relax\b are legal (as no numbers are expected), but if they were \countdef tokens, only the first would be allowed.
  4. Among other things this allows \relax to be part of the parameter text of a macro definition.

To summarize, it seems to me that if some specific syntactical quantity is expected, \relax is ignored if it could not act meaningfully as a delimiter.

wave
  • 841