Does TeX always insert the macro \par?

Question

When the input processor encounters two characters with category code 5 in a row (in other words, a blank line), it inserts the \par macro.

When \vbox{Abc.} ends, TeX ends the current paragraph, but not by inserting the \par macro. It seems that TeX is inserting the \par primitive instead. Am I understanding correctly what TeX is doing in this case? What about other places where TeX inserts \par? When is it the macro \par, and when the primitive?

\catcode`@=11
\let\@@par\par
\def\par{\typeout{Macro!}\@@par}

Abc.

\vbox{Abc.\tracingall}

Answer 1

There are exactly 7 places in the TeX program where TeX executes the paragraph builder internally, i.e. turning a horizontal list (if there is one under construction) into a paragraph. This happens not by inserting a \par token into the input stream but by executing the procedure end_graf implemented in module §1096 in the TeX program.

This procedure does nothing if TeX is not in horizontal mode, more or less nothing if in horizontal mode but with an empty list (null paragraphs are ignored by TeX), and executes the procedure line_break otherwise (and that one does all the magic about adding parfillskip penalties, etc.).

The seven places are

• at the end of the internal vertical structures like the closing brace of a \vbox but also \noalign or \vcenter or an alignment cell
• when the primitive par_end is sensed (which is available on the macro level initially as the meaning of the token \par)
• immediately after an output routine (OR) has ended (thus any horizontal list started in the OR will not continue with material from the galley but form a paragraph by its own)

In none of these cases there is a \par token inserted (that might be subject to redefinitions); instead, the end_graf procedure is executed!

\par tokens are only inserted when in horizontal mode and primitives incompatible with horizontal mode (like \vskip, \hrule, ... the full list in is §1094 in the TeX code) are encountered. And of course in the tokenization process when TeX replaces two endline chars by \par (i.e., making empty lines equivalent to \par).

Answer 2

According to The TeXbook, the rule for end-of-lines is as follows

If TeX sees an end-of-line character (category 5), it throws away any other information that might remain on the current line. Then if TeX is in state N (new line), the end-of-line character is converted to the control sequence token \par (end of paragraph); if TeX is in state M (mid-line), the end-of-line character is converted to a token for character 32 ( ) of category 10 (space); and if TeX is in state S (skipping blanks), the end-of-line character is simply dropped.

Here, it's important to note that the TeX inserts the \par token, and not the \par primitive. That means that \par has to be defined under all circumstances. Knuth illustrates this point when discussing commands which force horizontal mode:

The appearance of a <vertical command> in restricted horizontal mode is forbidden, but in regular horizontal mode it causes TeX to insert the token \par into the input; after reading and expanding this \par token, TeX will see the <vertical command> token again. (The current meaning of the control sequence \par will be used; \par might no longer stand for TeX's \par primitive.)

Now, what does not seem to be mentioned anywhere is what happens at the end of internal vertical mode. Clearly, the rules above do not allow for the insertion of a \par token as there are no end-of-line markers to deal with. (You do get a \par inserted if there is a blank line between the last material in the \vbox and the end of the box.)

Reading the trace output, there is no mention of the \par primitive at the end of the box, but the shipout does show

....\penalty 10000
....\glue(\parfillskip) 0.0 plus 1.0fil

and of course there is clearly a paragraph built. So my conclusion is that the end of internal vertical mode implicitly inserts a \par primitive, and thus inserts the usual end-of-paragraph material and then runs the paragraph builder.

Answer 3

There is a reason why TeX puts the "original meaning" of \par at the end of a \vbox: consider the following silly input

\vbox{\let\par\empty a}

When TeX finds }, it backs it up, terminates horizontal mode by inserting the "original \par" and then rereads the } that closes the internal vertical mode.

We can emulate what would happen if TeX inserted the current meaning of \par before rereading the } by

\vbox{\let\par\empty a\vskip0pt}

because \vskip inserts a (current) \par and TeX rereads \vskip. The result is an infinite loop!

(Found in a 1993 discussion on comp.text.tex.)