This is a direct follow-up question from my previous one: Overwrite matrix to work inside align (to use mtpro2 parentheses and braces). Although this question is self-contained, I encourage anyone who is interested to read that post first.
My goal is to overwrite the mathtools package’s dcases* and friends, so that large curly braces from the MathTime Pro 2 font can be used automatically. Note that although the “complete” version of MTPro2 is non-free, the “lite” version (which contains large curly braces) is freely available.
Note that pmatrix, Bmatrix and cases from amsmath; and pmatrix* and Bmatrix* from mathtools were taken care of in my own answer, which was based on @egreg’s excellent answer with two noticeable improvements.
Requirements
For dcases* and friends, there is only one requirement:
- Use large curly braces if package option
curlybracesis passed tomtpro2. Otherwise, with optionmorphedbracesorstraightbraces, use large straight braces.
Useful information
Large curly braces are accessed via the mtpro2 command
\LEFTRIGHT<left delimiter><right delimiter>{<contents>}% `mtpro2' exclusive
while large straight braces can be obtained simply by
\left<left delimiter> <contents> \right<right delimiter>
Furthermore, the 3 brace-shape package options are declared as
% From `mtpro2.sty'
\DeclareOption{curlybraces}{\let\mtp@br=c}
\DeclareOption{morphedbraces}{\let\mtp@br=m}
\DeclareOption{straightbraces}{\let\mtp@br=s}
My attempt
The ideas are the same as in @egreg’s answer and my own: 1) Store the contents of the cases in a box \mtp@casesbox. 2) Print the box with \LEFTRIGHT (when curlybraces is set).
However, mathtools creates dcases* and friends differently from how amsmath redefines cases. So, I used two new building macros \MTP_MT_start_cases:nnn and \MTP_MH_end_cases:, with two new commands \mtp@newcases and \mtp@renewcases. As the names suggest, the two commands are made internal on purpose.
\documentclass{article}
\usepackage{mathtools}% loads `amsmath'
\usepackage{newtxtext}
\usepackage[scaled=0.861,lining]{FiraMono}
\def\bracesshape{curlybraces}% change here to obtain different braces
% curlybraces
% morphedbraces
% straightbraces
\usepackage[lite,\bracesshape]{mtpro2}
% Patches begin
\makeatletter
\newsavebox{\mtp@casesbox}
% Activate `mathtools' syntax
\MHInternalSyntaxOn
% Curly braces are used only if `curlybraces' is set
% From `mtpro2.sty': \DeclareOption{curlybraces}{\let\mtp@br=c}
\MH_if_meaning:NN \mtp@br c
\def\MTP_MT_start_cases:nnn #1#2#3{ % #1=sep,#2=lpreamble,#3=rpreamble
\RIfM@\else
\nonmatherr@{\begin{\@currenvir}}
\fi
\MH_group_align_safe_begin:
\setbox\mtp@casesbox=\hbox\bgroup$% <- put contents in `\mtp@casesbox'
\vcenter \bgroup
\Let@ \chardef\dspbrk@context\@ne \restore@math@cr
\let \math@cr@@\AMS@math@cr@@
\spread@equation
\ialign\bgroup
\strut@#2 \strut@
#3
\crcr
}
\def\MTP_MH_end_cases:{\crcr\egroup
\restorecolumn@
\egroup
$\egroup% <- close the `\hbox'
\MH_group_align_safe_end:
}
\newcommand*\mtp@newcases[6]{% #1=name, #2=sep, #3=lpreamble, #4=rpreamble, #5=left, #6=right
\newenvironment{#1}
{\MTP_MT_start_cases:nnn {#2}{#3}{#4}}
{\MTP_MH_end_cases:\LEFTRIGHT#5#6{\copy\mtp@casesbox}}
}
\newcommand*\mtp@renewcases[6]{
\renewenvironment{#1}
{\MTP_MT_start_cases:nnn {#2}{#3}{#4}}
{\MTP_MH_end_cases:\LEFTRIGHT#5#6{\copy\mtp@casesbox}}
}
\mtp@renewcases{dcases}{\quad}{%
$\m@th\displaystyle{##}$\hfil}{$\m@th\displaystyle{##}$\hfil}{\lbrace}{.}
\mtp@renewcases{dcases*}{\quad}{%
$\m@th\displaystyle{##}$\hfil}{{##}\hfil}{\lbrace}{.}
\mtp@renewcases{rcases}{\quad}{%
$\m@th{##}$\hfil}{$\m@th{##}$\hfil}{.}{\rbrace}
\mtp@renewcases{rcases*}{\quad}{%
$\m@th{##}$\hfil}{{##}\hfil}{.}{\rbrace}
\mtp@renewcases{drcases}{\quad}{%
$\m@th\displaystyle{##}$\hfil}{$\m@th\displaystyle{##}$\hfil}{.}{\rbrace}
\mtp@renewcases{drcases*}{\quad}{%
$\m@th\displaystyle{##}$\hfil}{{##}\hfil}{.}{\rbrace}
\mtp@renewcases{cases*}{\quad}{%
$\m@th{##}$\hfil}{{##}\hfil}{\lbrace}{.}
\MH_fi:
% Deactivate `mathtools' syntax
\MHInternalSyntaxOff
\makeatother
% Patches end
\newcommand*\showopendelimitersizes[1]{%
#1\bigl#1\Bigl#1\biggl#1\Biggl#1}
\begin{document}
\section*{\texttt{dcases*} in \texttt{align} work?}
\verb|dcases*| and friends from \verb|mathtools| work,
iff \verb|curlybraces| is set.
\subsection*{Package \texttt{mtpro2} options: \texttt{lite,\bracesshape}}
\begin{align*}
\showopendelimitersizes{\lbrace}
\begin{dcases*}
\int_a^b f(x) \, \mathrm{d}x & Nothing to see here \\
\sum_{n=1}^\infty \frac{1}{n^2} & Otherwise
\end{dcases*}
\end{align*}
With \verb|morphedbraces| or \verb|straightbraces|,
errors appear.
\end{document}
Problems
My implementation works if and only if the package option curlybraces is used. If the package option morphedbraces or straightbraces is used, the following errors appear:
% Hit <compile>
! Package amsmath Error: \begin{document} allowed only in math mode.
See the amsmath package documentation for explanation.
Type H <return> for immediate help.
...
l.21 \nonmatherr@{\begin{\@currenvir}}
% Hit <return>
! Only one # is allowed per tab.
l.31 #
3
% Hit <return>
! Extra \fi.
l.64 \MH_fi:
% Hit <return>
! Undefined control sequence.
\dcases* ->\MTP_MT_start_cases:nnn
{\quad }{$\m@th \displaystyle {##}$\hfil ...
l.85 \end{align*}
...
- Major question: Why do these errors appear? Added: My educated guess: The errors are likely to arise from how TeX deals with conditionals.
- Minor question: Did I insert
\setbox\mtp@casesbox=\hbox\bgroup$and$\egroupto the correct spots? I ask this, because in the definition of the original\newcasesand\renewcases,\left#4comes after\MH_group_align_safe_begin:but\right#6also comes after\MH_group_align_safe_end:?! This is rather surprising: I thought\right#6should come before\MH_group_align_safe_end:.
Alternative solutions are more than welcome (to this question as well as to my previous one)!

\RIfM@with its expanded form\relax\ifmmodeas defined inamsgen.dtx:\def\RIfM@{\relax\ifmmode}. But, of course, your solution requires less digging. – Ruixi Zhang Sep 05 '18 at 14:52