Discrepancy between lualatex and xelatex regarding display math vertical spacing

Question

Consider this

\documentclass[a4paper]{article}
\usepackage[left=0cm,right=12cm]{geometry}
\usepackage{unicode-math}
\setmathfont{Libertinus Math}
\usepackage[grid]{eso-pic}
\begin{document}
Before
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
After
\end{document}

At left, result from xelatex, at right from lualatex:

Testing was done with TL2018 (updated pretest). Do mentally the slight shift upwards of right image to perfectly align it with left. The point is that xetex output is more spread out vertically. In real life this means documents will differ easily in pagination.

It was pointed out in comment that this is already the case due to LuaTeX extending the hyphenation algorithm, whereas XeTeX sticks with the traditional one.

This testing came out of Problem with \underbrace and Libertinus Math

With default fonts, no discrepancy.

Sorry, edit: also discrepancy with commenting out \setmainfont{Libertinus Math} and keeping only unicode-math.

\documentclass[a4paper]{article}
\usepackage[left=0cm,right=12cm]{geometry}
\usepackage{unicode-math}
\usepackage[grid]{eso-pic}
\begin{document}
Before
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
\[ \underbrace{AAA}_{B}\]
After
\end{document}

Editing question title to remove reference to Libertinus Math.

Using only fontspec but not unicode-math outputs are identical. Using fontspec and unicode-math (or only the latter which loads the former) the outputs differ even with no additional font setup.

Same issue when using for example \setmathfont{TeX Gyre Pagella Math}.

---

It is not an issue of unicode-math but an engine “problem” which is shown using this Plain TeX example:

\nopagenumbers
\ifdefined\directlua
  \input luaotfload.sty
\fi

% Load any Unicode math font (we don't need scriptscript here)
\font\mathfont="TeX Gyre Pagella Math:script=math" at 10pt
\textfont0=\mathfont
\font\mathfonts="TeX Gyre Pagella Math:script=math;+ssty=0" at 7pt
\scriptfont0=\mathfonts

% Define the relevant symbols
\Umathcodenum`A="1D434
\Umathcodenum`B="1D435
\def\underbrace#1{\mathop{\Umathaccent bottom 0 0 "23DF {#1}}}

$$ \hbox{Before} $$
$$ \underbrace{AAA}_{B} $$
$$ \underbrace{AAA}_{B} $$
$$ \underbrace{AAA}_{B} $$
$$ \underbrace{AAA}_{B} $$
$$ \underbrace{AAA}_{B} $$
$$ \underbrace{AAA}_{B} $$
$$ \underbrace{AAA}_{B} $$
$$ \underbrace{AAA}_{B} $$
$$ \hbox{After} $$
\bye

Left is LuaTeX, right is XeTeX:

Answer 1

This is not an answer but more of a rhetorical follow-up question which is too long for a comment.

Both XeTeX and LuaTeX will emulate the traditional TeX behaviour of math typesetting in the absence of Unicode math fonts. We can exploit that to draw a side-by-side comparison of the output with and without a Unicode math font.

\nopagenumbers
\ifdefined\directlua
  \input luaotfload.sty
  Lua%
\else
  Xe%
\fi
\TeX

\hbox{
  \vbox{\hsize=50pt
    $$ \hbox{Traditional} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \hbox{After} $$
  }

  \font\mathfont="Latin Modern Math:script=math" at 10pt
  \textfont0=\mathfont
  \font\mathfonts="Latin Modern Math:script=math;+ssty=0" at 7pt
  \scriptfont0=\mathfonts

  \Umathcodenum`A="1D434
  \Umathcodenum`B="1D435
  \def\underbrace#1{\mathop{\Umathaccent bottom 0 0 "23DF {#1}}}

  \vbox{\hsize=50pt
    $$ \hbox{Unicode} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \underbrace{AAA}_{B} $$
    $$ \hbox{After} $$
  }
}
\bye

Here we begin to run into some serious trouble because neither engine lines up with the traditional TeX output. This is partly due to the fact that the metrics are slightly different and the underbrace accent has a radically different shape. But at the same time this discrepancy makes it impossible to say which engine is “right”.

The only thing one could do at this point is ask the developers of both engines to review their code and see whether any wrong scalings creep in somewhere, but that seems a bit pointless to me because it is not really clear what we are looking for. Because XeTeX has kind of a history of broken math, I'd suspect the error with them, but that is purely a guess and might as well be a false accusation.