The mathtools package contains a fix to \underbrace and \overbrace as defined in the LaTeX kernel.
If we compare the two definitions
%%% LaTeX kernel
\DeclareRobustCommand\underbrace[1]{\mathop{\vtop{\m@th\ialign{##\crcr
$\hfil\displaystyle{#1}\hfil$\crcr
\noalign{\kern3\p@\nointerlineskip}%
\upbracefill\crcr\noalign{\kern3\p@}}}}\limits}
%%% mathtools.sty
\def\underbrace#1{\mathop{\vtop{\m@th\ialign{##\crcr
$\hfil\displaystyle{#1}\hfil$\crcr
\noalign{\kern.7\fontdimen5\textfont2\nointerlineskip}%
\upbracefill\crcr\noalign{\kern.5\fontdimen5\textfont2}}}}\limits}
we see that they're almost identical, but mathtools improves the definition by not using a fixed length of 3pt and adjusting it with the current math font setup.
However, the definition of \underbrace in unicode-math is
\UnicodeMathSymbol{"023DF}{\underbrace}{\mathunder}{bottom curly bracket (mathematical use)}
that translates to more low level
\underbrace=\protected macro:
#1->\mathop {\Umathaccent bottom 7\symoperators "023DF\scan_stop: {{}#1}}\limits
What happens if you load mathtools after unicode-math is that the “Unicode” definition of \underbrace is overridden with the fix above done by mathtools.
Note that you get no error: indeed \upbracefill uses math characters that in unicode-math don't correspond to the pieces of brace as in legacy LaTeX math fonts.
In conclusion: mathtools makes a good fix for legacy math fonts, but it shouldn't do it when unicode-math is loaded. Until mathtools is updated, the solution is to load it before unicode-math.
amsmathbeforeunicode-math; the present issue is due tomathtools, though. I think this has already appeared. – egreg Dec 22 '19 at 08:48