How does one append material to a token list?

Question

Several times, the floatrow package uses \newtoks which (according to this website) means that a token register is allocated. An example code snippet is

\@ifdefinable\FR@everyfloat{\newtoks\FR@everyfloat}
\FR@everyfloat={\let\@footnotetext\@mpfootnotetext
  \def\@mpfn{mpfootnote}\def\thempfn{\thempfootnote}\c@mpfootnote\z@
  \floatobjectset\floatfont}

Suppose I want to add something (e.g., \setlength{\footnotesep}{0.2\baselineskip}) to the definition of \FR@everyfloat. One way I successfully applied in this answer is to overwrite the original definition, i.e., write

\FR@everyfloat={\let\@footnotetext\@mpfootnotetext
  \def\@mpfn{mpfootnote}\def\thempfn{\thempfootnote}\c@mpfootnote\z@
  \floatobjectset\floatfont
  \setlength{\footnotesep}{0.2\baselineskip}}

However, I'd like to avoid duplicating the whole token list content; instead, I want to append material to the list the same way one would use \g@addto@macro (or \appto from the etoolbox package). However, my naïve approach to write

\g@addto@macro{\FR@everyfloat}{\setlength{\footnotesep}{0.2\baselineskip}}

doesn't work -- obviously, token lists must be treated different from macros.

So: How does one append material to a token list?

Answer 1

As others have pointed out, \the\mytoks delivers the contents of the token register \mytoks. When one wants to append something to a token register, the usual way is

\mytoks=\expandafter{\the\mytoks <tokens>}

Such code needs an explanation. A token register's name (not following \the) starts an assignment to that token register. An optional = can follow, and then there must be {<balanced text><right brace>. Every time a required token list has to begin with { (in the simbology of the TeXbook this denotes an explicit token of category 1 or a token \let to it) TeX expands the tokens it finds in order to come to {. Thus, in

\mytoks=\expandafter{\the\mytoks <tokens>}

\expandafter is expanded, delivering the contents of \mytoks and so the assignment can be performed. The assignment is terminated by an explicit right brace. A general macro for this can be

\long\def\appendto#1#2{#1=\expandafter{\the#1#2}}

where #1 is the name of the token register and #2 the token list to append.

A bit more complicated is to prepend something to the contents of a token register. Let's try; parameter #1 will denote a token register, #2 will be the tokens to be prepended:

\long\def\prependto#1#2{\toks0={#2}%
  #1=\expandafter{\the\toks0\the#1}}

Here we're using a dirty trick: \the wants to know what it must act on; it sees \toks which has to be followed by a ; 0 starts a number, but TeX continues to expand to see if other digits follow. So the second \the is expanded.

Limitation: one can't \prependto the token register \toks0.

Is there a way to overcome this limitation? Yes, there is:

\long\def\prependto#1#2{\def\tempa{#2}%
  \edef\tempb{\noexpand\tempa\the#1}%
  #1=\expandafter\expandafter\expandafter{\tempb}}

Suppose we have \mytoks={\A\B\C} and that we call

\prependto\mytoks{\a\b\c}

Then TeX executes \def\tempa{\a\b\c}; we now exploit the fact that the token list resulting from \the\mytoks is not further expanded in an \edef, so this is equivalent to having said

\def\tempb{\tempa\A\B\C}

The next instruction is

\mytoks=\expandafter\expandafter\expandafter{\tempb}

which becomes

\mytoks=\expandafter{\tempa\A\B\C}

and, finally,

\mytoks={\a\b\c\A\B\C}

Of course, the first method is much more efficient. Instead of \toks0 one can use a permanently reserved token register, say `\reservedtoks:

\def\prependto#1#2{\reservedtoks={#2}%
  #1=\expandafter{\the\expandafter\reservedtoks\the#1}}

(exercise: find out why it works). Users wanting to say \prependto\reservedtoks{abc} are on their own. :)

A trickier strategy, that avoids temporary assignments, has been suggested by Bruno Le Floch:

\def\prependto#1{\expandafter\prependtoaux\expandafter{\the#1}#1} 
\long\def\prependtoaux#1#2#3{#2{#3#1}}

Let's see with the same setting as before what happens with \prependto\mytoks{\a\b\c} (each line is what results from the expansion of the preceding one):

\prependto\mytoks{\a\b\c}
\expandafter\prependtoaux\expandafter{\the\mytoks}\mytoks{\a\b\c}
\prependtoaux{\A\B\C}\mytoks{\a\b\c}
\mytoks{\a\b\c\A\B\C}

(the = after \mytoks is optional).

Notes

1. The expansion to find an open brace don't happen for \def because the left brace delimiting the replacement text must be explicit.

2. \mytoks=\bgroup abc} is a valid assignment; on the contrary, \mytoks=\bgroup abc\egroup isn't: TeX will continue to scan tokens until finding an explicit (unbalanced) right brace.

3. \long is needed because the token list to append or prepend could contain \par (thanks to Bruno Le Floch for pointing it out).

Answer 2

Looking at mbork's solution and asking myself "Why isn't this included in the LaTeX kernel?", I discovered that it actually is. A search for #1=\expandafter{\the #1#2} didn't return anything, but a search for #1\expandafter{\the#1#2} (note the missing equal sign and space) returns (citing source2e, chapter 27)

We need a macro to add tokens to a hook.

485 \long\def\addto@hook#1#2{#1\expandafter{\the#1#2}}

This is TeX syntax, but otherwise identical to mbork's code. So the answer to my question is: Use \addto@hook.

Answer 3

Token register expand to their content only if prefixed with \the, otherwise on their own they are not expandable, IIRC. You need to use \expandafter to expand the token register before assigning it to the register again:

Appending works like this:

\mytoks\expandafter{\the\mytoks<new code>}

Here you don't need another \expandafter at the beginning because token register expand the following tokens in their search for a balanced { } list.

Prefixing code is more complicated because you have to jump over it. This is normally done using a second token register:

\temptoks{<new code>}
\mytoks\expandafter{\the\expandafter\temptoks\the\mytoks}

This uses the fact that \the is expanding the next tokens as well to find a register.

Answer 4

What about this?

\documentclass{article}

\newtoks\abc
\newcommand{\appendtotoks}[2]{% #1=toks register, #2-toks to append
  #1=\expandafter{\the #1#2}%
}

\begin{document}

\abc={A. B. C.}
\the\abc

\appendtotoks{\abc}{X. Y. Z.}
\the\abc

\end{document}

Answer 5

WARNING: as Joseph Wright pointed out in the comments section, this won't work in newer versions of expl3.

---

And using LaTeX3 (man, this is my first try of it, so forgive (and correct) me if I get something wrong!):

\documentclass{article}

\usepackage{expl3}

\begin{document}

\ExplSyntaxOn
\toks_new:N\abc_toks
\toks_set:Nn\abc_toks{A. B. C.}
\toks_use:N\abc_toks
\par

\toks_put_right:Nn\abc_toks{X. Y. Z.}
\toks_use:N\abc_toks
\ExplSyntaxOff

\end{document}