Unbalanced output routine

Question

The TeX primitive \output behaves almost like a toks register, except in the fact that it is always surrounded by braces. For instance, after \output={\plainoutput}, the result of \showthe\output is {\plainoutput}, instead of what we could naively expect (no braces).

The status of the surrounded braces is a little odd. My specific problem happens when replacing the closing brace with an explicit or implicit end-group character which does not directly come from TeX's internal mechanisms.

Specifically, compiling the following example in plain TeX triggers the ! Unbalanced output routine. error at line 6: the first case works, but not the second.

\output{\plainoutput \afterassignment\next \let\next}
\hbox{} \vfill \penalty -10000

\def\Next{\next}
\output{\plainoutput \afterassignment\Next \let\next}
\hbox{} \vfill \penalty -10000

In both cases, I redefine the output routine, then force TeX to call it (with \penalty-10000). I keep the normal \plainoutput routine, and afterwards grab the closing brace in \next, and put it back, either directly (\next), or indirectly (\Next). In the first, direct, case, TeX recognizes the closing brace as being from the \output routine, while in the second case, TeX doesn't, and gets into a state where nothing I type has any effect.

1. Why are the two situations different?

2. Assuming that I have typed the code above in the command line, is there any way to recover?

Answer 1

When one says

\toks0={\plainoutput}\showthe\toks0

TeX answers

> \plainoutput .

If one says after this

\output=\toks0

then \showthe\output gives

> \plainoutput .

If one then says \hbox{}\penalty-10000, then the error message

! Missing { inserted.
<to be read again> 
                   \shipout 
\plainoutput ->\shipout 
                        \vbox {\makeheadline \pagebody \makefootline }\advan...
<*> \hbox{}\penalty-10000

is issued. Now TeX is in internal vertical mode, as \end produces

`! You can't use `\end' in internal vertical mode.`

but typing } doesn't do any good, as we fall in the black hole when TeX doesn't interpret any more token.

It's interesting that TeX adds braces when the assignment to \output is of the form

\output=<general text>

but not when it's like \output=<token register>.

The relevant code for the addition of braces is in module 1226, where Knuth comments "For safety’s sake, we place an enclosing pair of braces around an \output list.", and module 1227.

An interesting module to examine is 1100, which ends with output_group: followed by what is in module 1026.

I realize this is not a full answer: it's only to show that it's better not to monkey with the closing brace. TeX is in a very particular type of group, when it performs the output routine and disturbing it during this task reveals to be quite dangerous.

I've also found a discussion on comp.text.tex that seems to have points in common with this problem.

Answer 2

If you trace through the code in tex.web, you'll find that what is important is routine 1025:

@ @<Fire up the user's output routine and |return|@>=
begin output_active:=true;
incr(dead_cycles);
push_nest; mode:=-vmode; prev_depth:=ignore_depth; mode_line:=-line;
begin_token_list(output_routine,output_text);
new_save_level(output_group); normal_paragraph;
scan_left_brace;
return;
end

As with any other case where we can have the left brace as an implicit {, the right brace has to be either } itself or an implicit } not hidden within a macro. So hiding } inside \Next will fail in exactly the same way as for any other situation. (You can test that the } is not 'special' in any other way by using \def\Next{\egroup}, which works fine.)

You'll also notice that the output group is actually started not by the { but is explicitly coded, as TeX knows that there has to be a group here. scan_left_brace finds an (implicit) left brace, but throws it away.

Answer 3

Delving through tex.web, I see

begin if (loc<>null) or 
 ((token_type<>output_text)and(token_type<>backed_up)) then
  @<Recover from an unbalanced output routine@>;

I don't understand when the loc<>null is true/false. But the rest is easier to understand: the token_type tells us where the token comes from: is it inserted by TeX, is it <to be read again>, etc. Here, output_text is for tokens coming from the \output token parameter, which includes the closing brace, and backed_up is for tokens which are skipped over by \expandafter. I don't know why Knuth chose to allow those two types of tokens (well, output_text is obvious): any other type of token makes the test true, which calls the code for an unbalanced output routine. Thus, to close the output routine ourselves, we can simply use \expandafter\egroup\empty, as below:

\output
  {%
    \plainoutput % Plain TeX output routine
    \expandafter \egroup % End the output routine
    \let \next % Remove the closing brace inserted by TeX
  }
\hbox{} \vfill \penalty -10000

In case the token does not come from the right place, the following code is run

@ @<Recover from an unbalanced output routine@>=
begin print_err("Unbalanced output routine");
@.Unbalanced output routine@>
help2("Your sneaky output routine has problematic {'s and/or }'s.")@/
("I can't handle that very well; good luck."); error;
repeat get_token;
until loc=null;
end {loops forever if reading from a file, since |null=min_halfword<=0|}

After the error, TeX fetches tokens with get_token, until finding null, which presumably (according to the comment) cannot be obtained from user input, thus the user gets stuck.