{\phantom{X}} sometimes higher than X

Question

Why is {\vphantom{X}}^2 (%8) higher than any of the other options? In particular, why is it higher than %5, %6, and %7. The behavior is normal if I use a lower-case character instead of the X.

\documentclass{minimal}
\begin{document}
  \fbox{$X$}%1
  \fbox{${X}$}%2
  \fbox{$\vphantom{X}$}%3
  \fbox{${\vphantom{X}}$}%4

  \fbox{$X^2$}%5
  \fbox{${X}^2$}%6
  \fbox{$\vphantom{X}^2$}%7
  \fbox{${\vphantom{X}}^2$}%8
\end{document}

Answer 1

For the same reason

\documentclass{article}
\begin{document}
\fbox{${\vphantom{X}}^{\rlap{\vrule height0pt depth0.1pt width 2cm}2}$}%
\fbox{${\kern0ptX}^2$}
\end{document}

produces

The rule is just to show the height is the same.

Actually the comparison should be with

\fbox{${\vphantom{X}}^{\rlap{\vrule height0pt depth0.1pt width 2cm}2}$}%
\fbox{$\hbox{$X$}^2$}

that yields the same output.

When TeX has to add a superscript to a box (and a subformula in braces containing more than a single math character counts as a box), it has no clue about what character to append the superscript to, only the box's height.

When you do \vphantom{X} in math mode, TeX builds a box with \setbox0=\hbox{$X$}, then sets \wd0=0pt and does \box0. Apart from setting the width, it's the same as doing \hbox{$X$}.

Appending a superscript field to a single math character is different, because now TeX knows much more about the object.

Let's examine a simpler case

\documentclass{article}
\begin{document}

\showoutput

$\vphantom{X}^2$

${\vphantom{X}}^2$

\end{document}

The relevant bits of the log file are

....\mathon
....\hbox(6.83331+0.0)x0.0
....\hbox(4.51111+0.0)x4.48613, shifted -3.62892
.....\OT1/cmr/m/n/7 2
....\mathoff

and

....\mathon
....\hbox(6.83331+0.0)x0.0
.....\hbox(6.83331+0.0)x0.0
....\hbox(4.51111+0.0)x4.48613, shifted -4.36111
.....\OT1/cmr/m/n/7 2
....\mathoff

If we add \showlists before the two closing $ characters, we get

### math mode entered at line 6
\mathchoice
D\mathord
D.\hbox(6.83331+0.0)x0.0
T\mathord
T.\hbox(6.83331+0.0)x0.0
S\mathord
S.\hbox(4.78334+0.0)x0.0
s\mathord
s.\hbox(3.41667+0.0)x0.0
\mathord
^\fam0 2

for the first formula and

### math mode entered at line 8
\mathord
.\mathchoice
.D\mathord
.D.\hbox(6.83331+0.0)x0.0
.T\mathord
.T.\hbox(6.83331+0.0)x0.0
.S\mathord
.S.\hbox(4.78334+0.0)x0.0
.s\mathord
.s.\hbox(3.41667+0.0)x0.0
^\fam0 2

for the second formula. The difference is subtle: in the first case the superscript is appended to an empty \mathord atom, in the second case to the whole box.

Answer 2

This is just to summarize egreg's detailed analysis in my own words.

The effect is a combination of two things.

First, TeX can apply a superscript either to a character or a box. Character: it knows more and can place the superscript at a better position. Box: it knows nothing more than the dimensions of that box "(and a subformula in braces containing more than a single math character counts as a box)". This explains %5, %6, and %8.

\documentclass{minimal}
\begin{document}
  \fbox{$X^2$}%
  \fbox{${X}^2$}%
  \fbox{${XX}^2$}
\end{document}

Second, writing a superscript directly after a \vphantom somehow is not applied to that \vphantom. This explains %7.

\documentclass{minimal}
\begin{document}
  \fbox{$\vphantom{\rule{1pt}{30pt}}^2$}
  \fbox{$\vphantom{\rule{1pt}{30pt}}{}^2$}
  \fbox{${\vphantom{\rule{1pt}{30pt}}}^2$}
\end{document}