19

Chapter 7 of The TeXbook famously talks of TeX as having 'eyes', 'mouth' and 'stomach' and goes on to talk of TeX's 'digestive processes' and keeps reusing these terms.

Chapter 8 introduces talk of TeX's 'gullet', apparently a reference back to this extended metaphor, without further explanation. The term sees use in other works: is there an accepted definition for the term?

egreg
  • 1,121,712
Charles Stewart
  • 21,014
  • 5
  • 65
  • 121
  • Well you first need to know what a gullet is :-) Quoting: "expandable tokens are converted to unexpandable ones in TeX's "gullet" – Sean Allred Nov 07 '15 at 13:37
  • 1
    @Sean: A gullet is the tube running from the mouth to the stomach, which is a bit helpful, but it doesn't exactly determine where the boundaries between the three 'organs' are to be drawn. – Charles Stewart Nov 07 '15 at 13:40
  • Yes :) so continuing along this line of thought: if you know that the stomach processes unexpandable tokens and that the mouth makes tokens out of characters, you have only the expansion process left to join the two. – Sean Allred Nov 07 '15 at 13:46

3 Answers3

20

The terminology isn't that useful really but as far as I can make out.

The mouth is where tokens get recognised so abc becomes three character tokens, and \abc becomes one control token

Unexpandable tokens are handled in the stomach so the gullet is basically the expansion process.

The texbook only uses gullet twice (and tex.web not at all):

expandable tokens are converted to unexpandable ones in TeX's ''gullet'' by a process similar to regurgitation.

and

What would be wrong with

 \def\%{\char`%}

? The % would be treated as a comment character, because its category code is 14; thus, no % token or } token would get through to the gullet of TeX where numbers are treated.

So as well as macros being replaced by their replacement text, the two tokens `a being interpreted as a number (the character code of a) is also being seen as a gullet process. Conversely \char itself is stomach.

David Carlisle
  • 757,742
19

I don't think that the comparison TeX with a living being having eyes, mouth, stomach and bowels should be taken so seriously as to find an analogy with other body parts.

I like better the description Victor Eijkhout makes in “TeX by Topic” of TeX as operating on four levels (see pages 21 and 22):

The four levels are (corresponding roughly to the ‘eyes’, ‘mouth’, ‘stomach’, and ‘bowels’ respectively in Knuth’s original terminology) as follows.

  1. The input processor. This is the piece of TeX that accepts input lines from the file system of whatever computer TeX runs on, and turns them into tokens. Tokens are the internal objects of TeX: there are character tokens that constitute the typeset text, and control sequence tokens that are commands to be processed by the next two levels.

  2. The expansion processor. Some but not all of the tokens generated in the first level – macros, conditionals, and a number of primitive TeX commands – are subject to expansion. Expansion is the process that replaces some (sequences of) tokens by other (or no) tokens.

  3. The execution processor. Control sequences that are not expandable are executable, and this execution takes place on the third level of the TeX processor. One part of the activity here concerns changes to TeX’s internal state: assignments (including macro definitions) are typical activities in this category. The other major thing happening on this level is the construction of horizontal, vertical, and mathematical lists.

  4. The visual processor. In the final level of processing the visual part of TeX processing is performed. Here horizontal lists are broken into paragraphs, vertical lists are broken into pages, and formulas are built out of math lists. Also the output to the dvi file takes place on this level. The algorithms working here are not accessible to the user, but they can be influenced by a number of parameters.

Under this interpretation, the “gullet” would be the channel by which levels 2 and 3 communicate in both directions, for example when tokens sent to level 3 by \lowercase or \uppercase return to level 2 after having been converted according to the values in the \lccode and \uccode arrays. However, I think this would add nothing to the understanding of how things work.

There is another aspect that's not covered by the living being allegory: the process whereby TeX sends to level 2 the macro replacement texts. Level 3 is responsible for storing macro definitions, which level 2 accesses at.

Finally, I would reject comparing the process whereby DVI or PDF files are produced, along with the log and write streams, to another body part that I won't name. Such an analogy would be ridiculous.

Community
  • 1
egreg
  • 1,121,712
  • 2
    It's not so ridiculous when you look at some of my first documents :-) – Sean Allred Nov 07 '15 at 15:10
  • 1
    @SeanAllred When my advisor saw the first paper I typeset for him with TeX, back in 1986, he was amazed! :P – egreg Nov 07 '15 at 15:14
  • Well, perhaps you didn't have such a deep-seated attachment to the Wrong Way espoused by overzealous humanities professors :-) It took a short while for me to realize that if a task is so hard using a given tool, perhaps it's not the way the tool was meant to be used. – Sean Allred Nov 07 '15 at 15:16
  • @SeanAllred Well, in those days the Wrong Way wasn't still so fashionable: papers used to be typewritten. – egreg Nov 07 '15 at 15:20
  • Precisely; I don't imagine there was much room for fine-tuning. (If I understand correctly, this is a large reason of why TeX was created in the first place!) The possibility for customization affords also the possibility for really bad customizations. – Sean Allred Nov 07 '15 at 15:22
  • It seems that Eijkhout denotes eyes plus mouth as one processor (1). This brings a bit of obfuscation in the terminology. For example in my book, there is input processor which reads lines of the source, removes the spaces at the end of lines and does an reencoding (from EBCDIC to ASCII for example or by encTeX). Then token processor makes tokens, which is exactly state automaton. – wipet Nov 08 '15 at 19:26
  • @wipet The “eyes” are level 1. There is communication also between levels 1 and 3, because assignments performed in level 3 can influence level 1. Level 2 is macro expansion. I don't think there's any confusion between them. You are dividing level 1 into two parts, which is also sensible: just a different point of view, possibly influenced by your work on encTeX. – egreg Nov 08 '15 at 20:42
  • (Sorry for the crude comment…) It seems DEK thought of the ridiculous/crude joke too: I just noticed an index entry in TeX: The Program (/texdoc tex), between default_skew_char_code and define: it goes to the section 597 which contains procedure write_dvi. – ShreevatsaR May 19 '17 at 18:58
7

Quoting from TeX For The Impatient, page 16, emphasis added:

The eyes transform an input file into a sequence of characters. The mouth transforms the sequence of characters into a sequence of tokens, where each token is either a single character or a control sequence. The gullet expands the tokens into a sequence of primitive commands, which are also tokens. The stomach carries out the operations specified by the primitive commands, producing a sequence of pages. Finally, the intestines transform each page into the form required for the .dvi file and send it there. These actions are described in more detail in Section 4 under “anatomy of TeX” (p. 46).

So, TeX's 'gullet' effectively performs expansion.

Sean Allred
  • 27,421
  • 1
    On the other hand, we see on page 379 “Careful study of this example shows that TeX’s mouth is capable of doing recursive operations, given sufficiently tricky macros.”, which hints at the mouth being where expansion is carried on. – egreg Nov 07 '15 at 14:00
  • 1
    @egreg What can I say? It would seem TeX has a very short neck. – Sean Allred Nov 07 '15 at 14:02