31

Trying to understand TeX's rules for parsing dimensions, I came up with the following weird behaviour:

\dimen0=1\noexpand\noexpand\noexpand\noexpand\noexpand\empty pt

is valid, but

\dimen0=1\noexpand\noexpand\noexpand\noexpand\noexpand\noexpand\empty pt

yields

! Illegal unit of measure (pt inserted).
<to be read again> 
                   p
<to be read again> 
                   t
<*> ...oexpand\noexpand\noexpand\noexpand\empty pt

?

What causes the difference?

Bruno Le Floch
  • 44,937
  • I need 7 \noexpand to get the error. – Alain Matthes Jul 10 '12 at 13:59
  • Same here. Either it is a stack/register limitation or maybe it tries to \noexpand what is after/in the definition/instance of \empty and depending on the version may or may not fail. Adding more \noexpands seems to always produce the error which suggests it is a parser limitation. – AbstractDissonance Jul 10 '12 at 14:01
  • 4
    It's six with Knuth's TeX, but seven with e-TeX. – egreg Jul 10 '12 at 14:02
  • ooohhhhh in T:2011 I get an error with tex but not pdftex – David Carlisle Jul 10 '12 at 14:05
  • It is platform-dependent. I would go with @AbstractDissonance that it's due to either a stack or parser limitation. Unlike \expandafter, there is a limit to the chain-length of \noexpand. Or can \noexpand become a frozen \relax? – Ahmed Musa Jul 10 '12 at 17:07
  • @AhmedMusa See what I found in my answer below. It ends up not being a stack limitation (that would have surprised me a bit, since TeX can happily \expandafter thousands of tokens. Another example is that \toks0\noexpand\noexpand...\noexpand{} will work fine. – Bruno Le Floch Jul 11 '12 at 23:03
  • @BrunoLeFloch: That was an interesting effort by you. – Ahmed Musa Jul 12 '12 at 18:07

1 Answers1

28

This was tricky to track down. Notice that we get an error after different numbers of \noexpand depending on the unit, which means that TeX does a different number of expansions depending on the unit. To experiment with this, let's set the active character ~ to be \noexpand, then try various assignments, increasing the number of ~ until TeX complains. Here are the maximum numbers of ~ for the original TeX engine:

\let~\noexpand
\dimen0=1~~\dimen1
\dimen0=1~~~em
\dimen0=1~~~~ex
\dimen0=1~~~~~true~pt % (and others)
\dimen0=1~~~~~~pt
\dimen0=1~~~~~~~in
\dimen0=1~~~~~~~~pc
\dimen0=1~~~~~~~~~cm
\dimen0=1~~~~~~~~~~mm
\dimen0=1~~~~~~~~~~~bp
\dimen0=1~~~~~~~~~~~~dd
\dimen0=1~~~~~~~~~~~~~cc
\dimen0=1~~~~~~~~~~~~~~sp
\dimen0=1~~~~~~~~~~~~~~~ % crashes after reading the last ~.

Clearly, all units are not created equal. The key is found in the tex.web file: the code TeX uses for scanning a unit is copied below. Let's comment on the case of

\dimen0=1~~~~~~~pt

for instance, which leads to an error (with 7 ~). TeX reads the number 1, expands the first ~ which temporarily turns the second ~ to \relax, stopping the expansion of 1: the number is complete. The:

@<Scan units and set |cur_val| to $x\cdot(|cur_val|+f/2^{16})$...@>=

This is essentially a "title".

if inf then @<Scan for \(f)\.{fil} units; |goto attach_fraction| if found@>;

Dimension assignments do not allow infinite glue, so this part is skipped.

@<Scan for \(u)units that are internal dimensions;
  |goto attach_sign| with |cur_val| set if found@>;

This scans for a unit such as \dimen..., expanding our second ~ which turns the next ~ to \relax (note that in the first line of our example file the ~ acts on \dimen and does nothing, so TeX sees \dimen). Then TeX looks for em, then for ex, "killing" two more ~. We are left with 3.

if mu then @<Scan for \(m)\.{mu} units and |goto attach_fraction|@>;

Skipped: this is not a muskip assignment

if scan_keyword("true") then @<Adjust \(f)for the magnification ratio@>;
@.true@>

TeX tries to scan true and fails again, leaving us with 2 ~.

if scan_keyword("pt") then goto attach_fraction; {the easy case}
@.pt@>

TeX looks for pt, expands, but the next ~ turns the last ~ to \relax, hiding the pt which follows. Yet another failure.

@<Scan for \(a)all other units and adjust |cur_val| and |f| accordingly;
  |goto done| in the case of scaled points@>;

The code called there tries various units one by one: in, pc, cm, mm, bp, dd, cc, sp. Our last ~ does nothing to the following p, and TeX keeps seeing a p, trying to match it with the units. When scanning for pc, TeX recognizes the p, but is then disappointed by the t. Eventually, TeX gives up. Note that the error message in this case shows both p and t as "to be read again". If you add two more ~, TeX hasn't reached the p yet when trying to parse pc, so only the p will be "to be read again".

attach_fraction: if cur_val>=@'40000 then arith_error:=true
else cur_val:=cur_val*unity+f;
done:

Rest of the code, we don't care here.

As a side note: since eTeX scans a few additional units than TeX, the following crashes at 14 ~ with TeX but 18 with eTeX:

\let~\noexpand
\dimen0=1~~~~~~~~~~~~~~~~~sp
Bruno Le Floch
  • 44,937