Whatsits: when are they used in practice?

Question

This question came up in discussions surrounding the question How to replace a large block of text by an empty block of the same size?, which basically asks for a vmode analog of \phantom. My question thus focuses on vertical mode only.

One approach is to collect the material to be hidden in a vertical box, then dissect the vertical box, replacing every box with contents by a box of the same size without contents. The only tools that TeX gives us are \lastskip and \unskip for skips, \lastkern and \unkern for kerns, \lastpenalty and \unpenalty for penalties, and \lastbox (as \setbox0\lastbox) for boxes. There is, in particular, no way of removing whatsits such as produced by \write, and \special.

1. What primitives produce whatsits nodes? When are they used in real world documents (in vertical mode)? Color changes? TikZ drawings?
2. (bonus) Are there other kinds of nodes in vmode, apart from skips, kerns, penalties, boxes, and whatsits? What primitives produce those? When do they appear in practice?

Answer 1

If whatsits are aptly named is a matter of opinion, as I think they would fit better in TeX's semantics if they were called afterallnodes; whatsits represent commands whose execution is delayed or are special commands associated with a particular device or system and are not part of TeX's normal processing flow.

It is interesting to investigate Knuth's rationale for introducing them. In a meeting with NTG members on March 13th, 1996 Knuth in reply to a question said:

I tried to make the programs so that they would have logical structure and it would be easy to throw in new features. This hasn’t happened anywhere near as often as I thought because people were more interested, I think, in inter-changeability of what they do; once you have your own program, then other people don’t have it. Still, if I were a large publisher, and I were to get special projects— some encyclopaedia, some new edition of the Bible, things like that — I would certainly think that the right thing to do would be to hire a good programmer and make a special computer system just for this project. At least, that was my idea about the way people would do it. It seems that hasn’t happened very much, although in Brno I met a student who is well along on producing Acrobat format directly in TeX, by changing the code. And the Omega system that you mentioned, that’s 150,000 lines of change files [laughter].

I built in hooks so that every time TeX outputs a page, it could come to a whatsit node and a whatsit node could be something that was completely different in each version of TeX. So, when the program sees a whatsit node, it calls a special routine saying, ‘how do I typeset this whatsit node?’ It’ll look at the sub-type and the sub-type might be another sub-type put in as a demo or it might be a brand new sub-type.

A whatsit can appear in either a horizontal or a vertical list and has no dimensions. It signifies an operation that should be delayed as it doesn't fit in its ordinary scheme of things. The paragraph builder and the page builder scan lists submitted to them and execute certain types of whatsit. They are useful when associated with specific implementations.

The more common whatsits are the ones associated with the main vertical list:

(a) delayed writes generated by \write. The token list of a delayed \write is not written-out until the surrounding material of a \write makes it to the output routine where a \shipout is executed. Therefore, the write token list has to be stored on the main vertical list.

(b) specials that use the \special command. The token list of a special command is stored with the main vertical list because the token list needs to be written to the dvi file. This happens, as in the case of write at the time of shipout.

Practical implementations can be found in postcript, pdf, color drivers and graphics programs. An interesting read is always the hyperref manual. The package uses \specials extensively to implement the interface between TeX commands and the PDF page description language. They are very simple to write:

    \immediate\special{!pdfpagelabels #2}%

To summarize it is a free for all hook/interface. Why they were called whatsit -- my guess is that it was a Knuth (ala \fi) whatisit. This simple innocent special command enabled TeX to survive and adapt over the years, producing output from postcript to PDF and introducing color and graphics.

Answer 2

I use whatsit nodes quite a lot for everything that has to do with PDF. I don't use the hyperref package for example and do everything by myself, so a whatsit is needed for links, color and rules. As an example see my LuaTeX wiki article on TeX without \TeX.

Answer 3

There's also inserts, see the log from this plain tex:

edit: and also marks: add \mark{foo} to the test below

\tracingonline1000
\showboxbreadth=100
\showboxdepth=1
\setbox2=\hbox{b}
\setbox0=\vbox{
a

\write-1{hello}

$$b$$

\special{foo}

\topinsert
aa
\endinsert

c}

\showbox0

\bye


(./wh.tex
> \box0=
\vbox(47.30554+0.0)x469.75499
.\hbox(4.30554+0.0)x469.75499, glue set 444.75497fil []
.\write-{hello}
.\glue(\parskip) 0.0 plus 1.0
.\glue(\baselineskip) 12.0
.\hbox(0.0+0.0)x469.75499, glue set 449.75499fil []
.\penalty 10000
.\glue(\abovedisplayshortskip) 0.0 plus 3.0
.\glue(\baselineskip) 5.05556
.\hbox(6.94444+0.0)x4.29166, shifted 232.73167 []
.\penalty 0
.\glue(\belowdisplayshortskip) 7.0 plus 3.0 minus 4.0
.\special{foo}
.\insert253, natural size 16.30554; split(0.0,16383.99998); float cost 0 []
.\glue(\parskip) 0.0 plus 1.0
.\glue(\baselineskip) 7.69446
.\hbox(4.30554+0.0)x469.75499, glue set 445.31055fil []

! OK.
l.20 \showbox0