Is \toks necessary to define \g@addto@macro? Can token registers be avoided in generel?

Question

In LaTeX2e kernel, there is a useful macro \g@addto@macro:

\long\def\g@addto@macro#1#2{%
  \begingroup
    \toks@\expandafter{#1#2}%
    \xdef#1{\the\toks@}%
  \endgroup}

However, I think the definition is rather complex. Why it is not defined as this?

\def\g@addto@macro#1#2{%
  \expandafter\gdef\expandafter#1\expandafter{#1#2}} % or \xdef?

I cannot see any differences between the two implementation except that \par is allowed in the former one.

And we know that etoolbox package and LaTeX3 use another approach with help of eTeX:

% etoolbox
\newcommand{\expandonce}[1]{%
  \unexpanded\expandafter{#1}}
\newrobustcmd{\appto}[2]{%
  \ifundef{#1}
    {\edef#1{\unexpanded{#2}}}
    {\edef#1{\expandonce#1\unexpanded{#2}}}}

% LaTeX3
\cs_new:Npn \exp_not:o #1 { \etex_unexpanded:D \exp_after:wN {#1} }
\cs_new_protected:Npn \tl_put_right:Nn #1#2
  { \cs_set_nopar:Npx #1 { \exp_not:o #1 \exp_not:n {#2} } }

It seems that \toks is not that useful: I cannot find any use of \tex_toks:D, say \toks in source3 at all. So, do we really need \toks registers?

---

NOTE: I'm not asking a question about LaTeX (2e/3) kernel, \g@addto@macro is just an example. This is more a comparison between macro and token registers in TeX.

Answer 1

I'll take the answer in two parts, without e-TeX and with e-TeX, with a couple of subdivisions to the latter.

Without e-TeX, you do need token registers to be able to control expansion. Inside an \edef is the classic example, where

\toks@{\mymacro\myothermacro...}%
\edef\foo{\the\toks@}%

will not expand our macros. That's particularly required when you do not know what the input here might be: if it's known to be only one token then \noexpand would do. The same applies in other expansion contexts such as a \write.

(I'm assuming here that readers know that \the\toks@ will expand only to the content of the toks inside an \edef, and no further. Depending on the nature of \mymacro, etc., it may be critical to avoid exhaustive expansion.)

On the question of why use a toks rather than a simple \expandafter chain, the LaTeX2e sources give the game away:

% \begin{macro}{\g@addto@macro}
% Globally add to the end of a macro.
% \changes{v0.2a}{1993/11/14}{Made global}
% \changes{v0.2w}{1994/01/31}
%     {Use toks register to avoid `hash' problems}
% \changes{v1.0o}{1995/05/17}
%     {Make long for latex/1522}
% \changes{v1.0w}{1996/12/17}
%     {Use \cs{begingroup} to save making a mathord}
% \changes{v1.0x}{1997/02/05}
%     {missing percent /2402}
%    \begin{macrocode}
\long\def\g@addto@macro#1#2{%
  \begingroup
    \toks@\expandafter{#1#2}%
    \xdef#1{\the\toks@}%
  \endgroup}
%    \end{macrocode}
% \end{macro}

Try \g@addto@macro\foo{\def\baz#1{}} or anything else containing a # without the use of a toks and observe the issue: you'd need to double all of the #s.

With e-TeX, the \unexpanded primitive is available. This acts like a toks but without the assignment, which allows a much simpler definition for \g@addto@macro along the lines of the two cited in the question. So in general there is little need to use a toks with e-TeX available: we dropped the entire l3toks module from LaTeX3 as a result.

TeX provides a few token parameters (for example \everypar) which act very much like toks and do have to be handled in a similar way. For LaTeX3, the plan is to provide a macro-like wrapper around all of them, so dealing with this unusual behaviour will be a job for the kernel team alone.

---

There is one place even with e-TeX where toks are useful, and that is when you want to refer to them by register number. The usual way of using a toks is

\newtoks\my@toks

and then use of the control sequence. However, you can also use toks by number

\toks0={tokens}

Normally, that's not a great approach. However, it does not consume a csname in the TeX hash table. So within a group (where we can be sure that we can freely use any register) it is potentially useful to use toks by number. Bruno Le Floch has taken advantage of this method in the l3regex module for LaTeX3, as this avoids heavy use of the hash table for no benefit, and is a case where referencing by number is actually not such a bad idea.