6

I've noticed that TeX doesn't always save a macro's text as typed. For instance, the following plain TeX manuscript:

\def\a#1{\relax#1}%
\font\myfont=cmtt12\myfont%
\meaning\a%
\bye

produces the output

macro:#1->\relax #1

Note that the output shows a space between '\relax' and '#1' where none exists in the manuscript. This discrepancy can have a dramatic effect on the execution of the manuscript.

Consider, for instance, the following manuscript.

\def\a#1{\def\b#1{Hello, world!}}%
\a c%
\bc%
\bye

If \a's text were saved as-is, this manuscript would compile successfully, the second line would define the macro \bc, and the third line would output: Hello, world!. Instead, the compilation aborts with the following error:

! Undefined control sequence.
l.3 \bc

It therefore behooves to know: what transformations exactly does a macro's text undergo before it is saved to memory?

Evan Aad
  • 11,066
  • what do you mean by "manuscript"? – jarnosc Aug 17 '17 at 18:45
  • @erreka: A program written in the TeX language. This is the terminology used in the TeXbook by the creator of TeX. – Evan Aad Aug 17 '17 at 18:52
  • I suspect you are taking a Knuthian way of saying as something normative or even mandatory, along the lines of one of your other questions. A "manuscript" (or more precisely, a "typescript" for a computer document), in common parlance, as you probably know, is the general term for the original of your document. In TeX's case, as in many a text with markup, has text interspersed with code/markup. In your example you are not marking up any text, so strictly speaking it is only "code". – jarnosc Aug 17 '17 at 19:06

2 Answers2

8

Basically: What TeX stores is not text (sequence of characters) but tokens: after your \def it remembers that the definition of \a has the control sequence \relax followed by macro parameter 1, and when you ask for this token list to be printed, it shows you a space after the \relax token, just for clarity.

I'm not sure about the “user”-level description of this as described in The TeXbook etc., but I can point you to the “internal” representation as described in TeX: The Program. See Part 20 (Token Lists) of texdoc tex. A token is either a (command code, character) pair, or a control sequence. With that background:

  • The \def you type gets stored as a sequence of tokens:

part 1

  • Then the above token list, when you ask for the definition of \mac to be shown, gets shown as:

part 2

(In this example the text happens to be identical, but note that you could have left out the space after the \b in the input text, and the internal token list representation would have been the same, so you would still have a space after the \b if you asked for it to be printed.)

ShreevatsaR
  • 45,428
  • 10
  • 117
  • 149
  • Thanks. This was illuminating. However, I'm still left wondering: why, then, does the following manuscript: \def\a#1{\def\b#1{Hello, world!}}\a c\bc\bye results in an error (! Undefined control sequence. l.3 \bc)? I'd expect it to output: Hello, world!. – Evan Aad Aug 16 '17 at 09:05
  • I've added the contents of my comment to my question. – Evan Aad Aug 16 '17 at 09:13
  • @EvanAad Your input definition becomes \def\b c{...} so whenever you invoke \b it looks for a c to parse its input. \b c would work but \b won't – percusse Aug 16 '17 at 09:41
  • @percusse: This would make sense, except how do you deduce from ShreevatsaR's answer that my input definition becomes \def\b c{...}? – Evan Aad Aug 16 '17 at 09:45
  • 1
    @EvanAad Maybe you should wait a bit longer before accepting answers. :-) Anyway, from a close look at the example above I can deduce that after \def\a#1{\def\b#1{Hello, world!}}, the definition of \a is stored as the token list (match #, end_match, \def, \b, out_param 1, {, letter h, … etc., so that when TeX expands \a c, these two tokens get replaced by the token list \def, \b, letter c, {, … etc. You can deduce from here what the result of expanding \def operating on that stream of tokens will be. – ShreevatsaR Aug 16 '17 at 19:01
6

The code

\def\a#1{\def\b#1{Hello, world!}}%
\a c%
\bc%

defines \a with one parameter; upon calling \a c, #1 becomes c, so what's executed is

\def•\b•c•{•H•e•l•l•o•,• •w•o•r•l•d•!•}

which defines \b with parameter text c and replacement text Hello, world!. (• is used here to separate tokens, for better clarity.)

If you do \show\a, you get

> \a=macro:
#1->\def \b #1{Hello, world!}.

which uses a different representation (a space after a control word), but this amounts to exactly the same thing: \b has already been tokenized. If you want to define \bc, you need

\def\a#1{\expandafter\def\csname b#1\endcsname{Hello, world!}}

Now \a c will define \bc to have replacement text Hello, world!.

As said before, the representation with \show or \meaning adds a space after control words, which is unconsequential if you copy and paste the shown code, as spaces are ignored in that position. This way you can see the division into tokens. Of course, it's possible to be misguided if something strange is done:

\catcode`z=12
\def\a#1{\def\z#1{Hello, world!}}
\show\a

would display

> \a=macro:
#1->\def \z#1{Hello, world!}.

but \a c would not define a macro \zc, because #1 is still a separate parameter token.

No transformation whatsoever is done to the tokens during \def, except for tokenization (which fixes category codes, by the way).

The category code of the characters forming the name of a macro are not recorded in any way. However, TeX will add a space in the \show/\meaning representation only after control words, not control symbols. So if you add

\catcode`z=11
\show\a

to the code above, the second \show will output

> \a=macro:
#1->\def \z #1{Hello, world!}.

because at this point, \z is to be considered a control word. This has more to do with input from a .tex file, rather than with tokens: always remember that already stored tokens are not touched by category code changes, but input ones are.

A real world example is the LaTeX macro \@. In code where the category code of @ is 11 (which is frequently done for defining “private” macros), \@ still has the same definition as when @ has category code 12. However, care is needed when using \@ in code, because

\@x

is read in as two tokens in normal situations (catcode of @ is 12), but just one when the catcode of @ is 11. Not a big deal, but something to be aware of.

egreg
  • 1,121,712
  • Thanks. The words "\b has already been tokenized" were the key to my understanding of this part of my question. – Evan Aad Aug 16 '17 at 09:55
  • Can you please explain your last example? What did changing z's catcode accomplish? Does the inner macro have an empty name? If not, what is the inner macro's name? Why didn't \show\a add a space between \z and #1 in the console output? Why would \a c not define a macro \zc? – Evan Aad Aug 16 '17 at 10:08
  • @EvanAad In the representation by \show, a control symbol is not followed by a space, but this has no other consequence, as \z has already been tokenized as *one* token and #1 is a parameter token, representing the argument (as a sequence of tokens) at macro call. – egreg Aug 16 '17 at 10:11
  • Thanks. Just let me understand, is the following statement correct? A macro's name is either a non-empty string of catcode-11 tokens, or a single catcode-12 token, and the catcodes of the tokens that make up the name are saved with the macro. – Evan Aad Aug 16 '17 at 10:14
  • 1
    @EvanAad No, the catcodes of the tokens that make up the name are not saved with the macro. – egreg Aug 16 '17 at 10:20
  • But if z's catcode isn't saved with the macro, then the macro you defined has the exact same internal representation if you comment-out the first line (cathode\z=12). However if you do so, the console output shows a space between the\zand the#1`. – Evan Aad Aug 16 '17 at 10:26
  • @EvanAad I see no space between \z and #1 in the \show output. Of course, if you change back the category code of z to 11, \z becomes a normal control word, exactly because category codes of characters forming the name are not saved. So long as the catcode of z remains 12, \z is a control symbol (with the relative rules for tokenization); if it becomes 11 again, \z is a control word. – egreg Aug 16 '17 at 10:28
  • This was my point! If z's catcode is not saved with the macro, then why do the outputs differ when you change z's catcode? – Evan Aad Aug 16 '17 at 10:29
  • 2
    @EvanAad Because z still has category code 12 at the point of \show and TeX doesn't add a space after a control symbol in the output of \show or \meaning. The key is the category code at the point of \show. – egreg Aug 16 '17 at 10:31