display full macro expansion

Question

Intrigued by a strange question (how to produce a [La]TeX representation of a Texinfo source file, under the assumption that the Texinfo source file would yield somehow a lower level document in "pure" TeX [presumably \ { } notation] macros), I wonder now how could I redirect the byproduct of macro expansion in TeX's gullet to, say, stdout or console or a text file, before it is fed to the typesetter (the "stomach"). I do not think anything in the \tracing* family of debugging control sequences does that job, as I'd like to see the actual stream of tokens.

I guess I could do that with some arcane LuaTeX callback, but I wonder if a dirty dekTeX macro (with \immediate\write or something like that) would do the trick.

---

For instance: if we feed pdfinitex with the following minimal input

\pdfoutput 1
\font\myfont ptmr8y
\myfont
Hi!
\end

we may produce a very minimal PDF file (with some errors). I assume these tokens go straight into the typesetter guts, as there are no macros, to produce the PDF output.

If, on the contrary, we run pdftex with the Plain format (loading, presumably, the etex.src macros) with the following input

Hi!
\bye

I conjecture the typesetter shall be fed with at least the following tokens:

\pdfoutput 1
\font\tenrm=cmr10
\tenrm
Hi!
\par \vfill \par \penalty -\@MM \end

How to show (on console, in aux file, anything...) the exact stream of tokens?

Perhaps dekTeX (and derivatives) can't do it; but LuaTeX may have some way.

Answer 1

Have a look at the unravel package. It implements TeX in TeX with a few limitations but it does an exceptional job at following the expansion chain for “normal” TeX code.

In you example you mention \pdfoutput=1 and \font\tenrm.... This you cannot obtain at all, not even with LuaTeX because these things are baked into the format which is just a dump of TeX's memory which is then reloaded from disk.

Unfortunately the unravel package is LaTeX-only. This will generate a lot of output.

\RequirePackage{unravel}
\unravel{
\documentclass{article}
\begin{document}
Hi!
\end{document}
}

Answer 2

1. What is the byproduct of expansion? The product of expansion is unexpandable tokens and/or expandable tokens whose expansion was suppressed for some reason.
2. Assume an expansion-based infinite loop. When is the time to write?
3. The mouth produces tokens. Expansion takes place in the gullet.
4. I don't know of a way of intercepting the token-stream that reaches the stomach feasible in Knuthian TeX.
5. \immediate\write alone does not do the trick: The "argument" of \write is expanded but expansion is not suppressed with tokens where expansion should be suppressed.
   E.g., assume \foo is defined to yield bar.
   \immediate\write\somewritehandle{\def\foo{rab}} yields that \def bar{rab} is written which seems not to be what you want.
6. The following is a problem when not writing all tokens that reach TeX's stomach (writing all tokens that reach TeX's stomach is not possible in Knuthian-TeX) but only some tokens, e.g., tokens within the argument of \write: Writing a set of (unexpandable) tokens to an external file is not sufficient for ensuring that tokenization at the time of reading/processing that file yields the same set of tokens.
   • For example, at the time of writing \escapechar might be changed. E.g. with \escapechar=`\x (unexpandable) control sequence tokens are written with leading x instead of leading backslash.
   • For example a could have category code 12(other). Writing the token sequence \a, b, c, <space token> then results in the character-sequence \abc<space> being written to the text file. If at the time of reading/tokenizing things from the text-file a has the usual category code 11(letter), then the character-sequence is tokenized as the token \abc. The <space> is not tokenized at all because after tokenizing a control word token the status of TeX's reading-apparatus is set to S(=skipping blanks).
   • Explicit character-tokens of category 6(parameter) are doubled during writing. (Usually hashes (#) have category code 6.)
   • Depending on the TeX-engine and configuration of TeX's character translation some characters might be written in ^^-notation. If the corresponding part of the resulting file is read/tokenized while ^^-notation is disabled, e.g., while gathering the tokens that belong to the argument of a command \verb, or while gathering the tokens that belong to the "body" of an environment \begin{verbatim}..\end{verbatim}, then Bob is not your uncle any more.