TeX's reading rules (text to tokens)

Question

Are TeX's reading rules, which define the conversion from text to tokens, and which are described on pp. 46-48 of Knuth's The TeXbook (1996), describe correctly and completely the reading rules of the TeX systems available in TeXLive 2017, namely pdfTeX, XeTeX and LuaTeX? If not, where can I find a lucid, up-to-date description of the reading rules pertaining to these systems?

You've asked a few questions about TeX90 vs newer engines, which suggest you are unclear on the fact that both pdfTeX and XeTeX are directly based on the Knuth code whilst LuaTeX is still very much a TeX system (with some deliberate non-compatibility). The differenced between the three 'common' engines are covered in https://tex.stackexchange.com/questions/222286/what-are-the-incompatibilities-of-pdftex-xetex-and-luatex, which as you'll see makes no mention of most TeX concepts as they apply unchanged. — Joseph Wright, Jun 21 '17 at 10:45
@JosephWright: What's the distinction between "based on the Knuth code" vs. "very much a TeX system"? — Evan Aad, Jun 21 '17 at 11:39
Both pdfTeX and XeTeX are still written in WEB, and until relatively recently were .ch files against tex.web itself. (Both are now separate .web. files.) Importantly, they make no breaking changes compared other than those which are required to extend the engine. (For example, as XeTeX is a Unicode engine it has to alter the range of valid values in many places.) On the other hand, LuaTeX is written in C and more important does make breaking changes (see details in the LuaTeX manual). However, it is still fair to describe it as a TeX system: it has macros, catcodes, ... — Joseph Wright, Jun 21 '17 at 12:03
Note that LuaTeX's C was written starting from pdfTeX (WEB source) mechanically converted to C (web2c), but this has been heavily revised so the 'link' to the origins is broken. On the other hand, one can (somewhat) usefully examine pdftex.web or xetex.web in comparison to tex.web to see how they diverge. — Joseph Wright, Jun 21 '17 at 12:04

score 7 · Accepted Answer · answered Jun 21 '17 at 12:19

All TeX-derived systems uses the same approach for this area and indeed for a large part of their operation.

In a modern TeX distribution, other than the command tex itself, no other command uses Knuth's TeX90 code unmodified. The 'standard' engine for provision of .e.g. etex and latex is pdfTeX. This is possible as pdfTeX is a drop-in replacement for TeX90, provided one does not actively use the e-TeX extensions or pdfTeX additions. (These can be turned off at the command line or when building a format.) The full detail of e-TeX and the pdfTeX additions to e-TeX are covered in the relevant manuals.

It's therefore reasonable to take pdfTeX as the reference and to examine what is different in XeTeX and LuaTeX: this is covered in detail in What are the incompatibilities of pdftex, xetex and luatex?. Most notably, both XeTeX and LuaTeX retain the vast majority of the approach documented in The TeXbook. Where a document is not exploiting engine-specific features or very specific Unicode aspects, XeTeX will give the same result as pdfTeX (and thus as TeX90). LuaTeX is slightly different but this is at the level of preventing its use in archival processing of existing inputs. In terms of 'will LuaTeX process this .tex file in a way similar to TeX90' the answer is 'yes'.

TeX's reading rules (text to tokens)

1 Answers1