3

The example comes from another thread with some modification:

\documentclass{article}
\newcommand\abc{\\xyz}
\newcommand\foo[1]{\def\abc{#1}\ifx\abc\empty T\else F\fi}
\begin{document}
\tableofcontents
\section{\foo{}}
\end{document}

After 2-round compilation, the toc file contains:

\contentsline {section}{\numberline {1}\def \\xyz{}F}{1}%

According to that answer, the expansion of \foo{} is likely:

  • \def is copied verbatim.

  • \abc{#1} becomes \\xyz{} because \abc is \\xyz and #1 is empty.

  • \ifx\abc\empty T\else F\fi evaluates to F because \abc is non-empty.

Totally, they match the toc content, when evaluated in the "expansion-only" mode.

But isn't this mode a total nonsense? The definition of \foo clearly states that the macro \abc is something being defined here, but the tex engine is substituting this macro with an existing definition. This is not what the code is meant to be. The code doesn't mean to define \\xyz.

I think the culprit here is this "expand-but-not-execute" mode. I don't see why such partial expansions will ever become meaningful. I believe the only way to correctly process a list of macros is to expand and execute all of them, in order. So, my question is very simple: Why do we need this "expansion-only" mode?

Cyker
  • 667
  • 1
    Are you looking for \section{\protect\foo{}}? (I cannot really answer why \section was defined in the way it is. However, \protect allows you to avoid the complications you are describing here.) –  Jun 12 '20 at 05:32
  • @Schrödinger'scat I replied you in that thread with a new example where a single \protect won't work. This thread is related but more fundamental. I think I do have to understand why \section is defined in the way it is. To be honest, I think this is a huge bug. I wouldn't think it works in this way. But there may be different ideas. – Cyker Jun 12 '20 at 05:35
  • 3
    Yes, but this is precisely why it is hard to answer your questions, and which is why I only post comments. For you it seems to be important to know why TeX is doing what it is doing, and for me it is important to know why you are asking these questions, i.e. what you really want to know. Obviously there must be some prescriptions that allow one to get the "contents" of the macro at the time \section was "executed", and there must be means to deal with delicate cases. These means exist, whether the way things are done is always the most intuitive way is a different question. –  Jun 12 '20 at 05:46
  • @Schrödinger'scat I agree there needs to be "contents" of a macro. But that is either the body text copied verbatim, or a fully expanded and executed result, not to be a result obtained by evaluating only a portion of the macro body. – Cyker Jun 12 '20 at 07:17
  • 1
    But you can't just expand all macros. What if the macro is \textsf, say? –  Jun 12 '20 at 07:20
  • @Schrödinger'scat I mean if you expand it then you also execute it. Or you can leave it as is. I don't know how exactly \textsf is defined. But if it is used in moving arguments then I guess it is fine just leave it there. – Cyker Jun 12 '20 at 07:27
  • 1
    We are talking about what LaTeX writes to the toc file, aren't we? What should it do with \textsf{pft} or \textcolor{\mycolor}{abc}? Wouldn't it be reasonable if it expanded \mycolor in the second case, but kept \textcolor unexpanded? Or \textsuperscipt{\mysuperscript}? You can't just "expand" these fully when you write them to the toc file. –  Jun 12 '20 at 07:32
  • @Schrödinger'scat In this case I'd rather writing the macro definition verbatim into toc. The correct way of reusing this macro is to save its definition together with its context. But in reality it could be difficult to save the context. On the other hand, I don't see why it isn't possible to expand both \textcolor and \mycolor if you really want to do it. – Cyker Jun 12 '20 at 07:50
  • 4
    the expansion only mode is the normal execution model for every other macro processor eg the C language #define system. The really weird tex-specific execution model is the intertwined macro expansion and evaluation that it normally does, with macro expansion happening "just in time" to produce enough tokens for the next primitive execution. – David Carlisle Jun 12 '20 at 08:21
  • 2
    Execution has side effects. If you would e.g. change a counter or a def in a \write or an \edef their state would be different later on. That is normally not something that you want when you simply write to a file – Ulrike Fischer Jun 12 '20 at 08:38
  • See also macros - Why isn't everything expandable? - TeX - LaTeX Stack Exchange – user202729 Jul 03 '22 at 08:50

3 Answers3

4

Welcome to the wonderful world of \edef and \write.

If you want to make a table of contents, you have to collect material that refers to the titles and write them down in an auxiliary file that can be input at the next run.

The main problems with \section{<text>} are:

  1. you need that the value of \thesection is immediately computed;
  2. conversely, the value of \thepage should only be computed when the page containing the section title is shipped out.

What happens is that TeX doesn't know what page it is on when \section is found. Indeed, it might have already processed material that almost fills a page and only after processing \section and the following paragraph TeX will realize this title has to be moved on the next page.

So, what does LaTeX do in this case? Two actions: first it should temporarily set \thepage to mean \relax and perform something like

\edef\x{\write\tocfile{\contentsline{section}{\numberline{\thesection}<text>}{\thepage}}
\x

(this is a simplification; actually the .toc file is written at end document, because it is not known where the table of contents should be typeset).

However, a problem immediately pops up: many of those macros must not be expanded at the time of \edef. OK, we could do

\edef\x{\write\tocfile{\noexpand\contentsline{section}{\noexpand\numberline{\thesection}<text>}{\noexpand\thepage}

(no need to add \noexpand in front of \write and \tocfile because they're unexpandable by construction). Well, yes and no: what's in <text>?

The title could contain something the user wants to be expanded (the lecture number, for instance, with a counter independent on the section counter) or something that the user doesn't want to be expanded: it's your case, isn't it?

The situation is far more complex than hinted above: macros such as \textbf do very complicates things and should never be expanded during the process of writing the .toc file. Therefore LaTeX uses \protected@edef and \protected@write that are wrappers around \edef and \write that take care of the protection mechanism necessary for avoiding premature expansion of those commands.

It's left to the user to decide whether something should be expanded or not when writing to the .toc file. How? With \protect in front of a token that should not be expanded or by exploiting the general protection mechanism: if you do

\DeclareRobustCommand{\foo}[1]{\def\abc{#1}\ifx\abc\empty T\else F\fi}

then your .toc file will contain

\contentsline {section}{\numberline {1}\foo {}}{1}%
egreg
  • 1,121,712
  • So your idea is: TeX gives users the ability to expand, but if users don't want it then they can choose not to expand using \protect or \DeclareRobustCommand. The default is to expand and so requires extra work if users want it otherwise. – Cyker Jun 15 '20 at 03:41
3

Only classical TeX has expansion-only mode. But all TeX engines today includes eTeX extension (version 2 from 1998) where the primitives \detokenize or \unexpanded allow to suppress the expansion.

The \section macro can be defined without problems described in your question. We need to save three different information to a reference file (toc for example) when \section macro is processed.

  • The title of the section: unexpanded.
  • The number of the section (2.3, for example): immediately expanded.
  • The page number: expandend later when \shipout is in process.

Suppose, that we have saved the title in \sectitle, the number in \secnumber and we have to create a macro which does the \write part of the \section macro. We can do:

\edef\tmp{\string\Dosection{\secnumber}{\detokenize\expandafter{\sectitle}}}
\expandafter\write\expandafter\tocfile\expandafter{\tmp{\the\pageno}}

Now, if you write \section{Text \foo{xy}, hello} then you get in the toc file:

\Dosection{1.1}{Text \foo{xy}, hello}{13}

Exactly what you want. You need not use \protect nor \protected\def of the used \foo macro.

But if you are using LaTeX's \section macro then have more problems. The reason is: LaTeX does not use \detokenize (the primitive from 1998) when does \write in its section macro. Because it is older than this primitive.

wipet
  • 74,238
2

Let's clarify some things first:

  • Expansion is manipulation of the input stream. This can happen either by expanding a macro (i.e. replacing it by its definition, possibly with arguments) or by expanding an expandable primitive (like \ifx).
  • Execution is manipulating the properties of the TeX engine. This can happen through non-expandable primitives (like \def) or assignments. These properties affect how the TeX engine behaves in the future.
  • In the usual metaphor, both expansion and execution happen in the mouth of the TeX engine. This is, however, not everything TeX does (otherwise we would never get a document). When the engine finds the letter a, for example, at the start of the input stream, it will swallow it. That is, it will (usually) put the glyph "a" of the currently selected font in the document (skipping a lot of detail here).

Now, we often want to store some piece of input stream away for later use, most notably in macros (for use in the same TeX run) and in files (usually for use in a later TeX run)1. Let's concentrate on the former.

Rather than just storing some piece of input string in a macro that we typed in the source code before compiling, it is often useful to store the expansion of something instead (like the current page number), which we can do with \edef. Note that we are "storing pieces of input stream", nothing else. We cannot store the state of the entire TeX engine in a macro (what would expanding that even mean?). It doesn't make sense to say "let's store the execution of this". Thus, \edef fully expands its argument, but does not execute anything (as does \write).

As noted by David Carlisle in his comment, this really is the "normal" mode for a macro expansion language. All that other business of execution and swallowing the TeX engine does is something else entirely. Only doing expansion should thus be quite natural.

1This is not really the same, of course, as the definition of a macro consists of tokens but the content of a file consists of characters, but let's not get hung up on that.

If you think about it, this is really the only way to go about it. In your question you proposed also doing execution in these contexts, but what would that even mean? If I did

\edef\foo{\def\baz{asdf}}

what would be the expansion of \foo? Should \baz be defined now? What should

\write\@auxout{\color{blue}Hello}

write to the aux file? Should text following this \write be blue?

I'm sure one could find more absurd examples, but in the end it comes down to this: You are storing a piece of input stream, so only input stream manipulation (i.e. expansion) should be performed during this operation. If some definition is necessary for the expansion you are aiming for, do that outside of this operation, e.g.

{\def\foo{baz}
  \immediate\write\@auxout{I just wanted to say \foo.}
}

As a final note, there are of course other places where an expansion-only context occurs. For example, when TeX expects a number, it will expand everything until it finds something unexpandable that can't be part of the number anymore. Also doing execution there would not make any sense.

schtandard
  • 14,892