Value of LuaTeX’s \mathstyle in \frac (de)nominator

Question

This was fixed in version from 2017/06/15 (available on CTAN).

This is a followup question of How to transfer math style in \hbox_set?.

While the code I posted there as an answer using LuaTeX’s \mathstyle primitive it doesn’t work inside of the (de)nominator of a \frac. If a \frac happens to appear in \textstyle the (de)nominator actually is set in \scriptstyle (as far as I can tell), but \mathstyle stell equals 2.

% !TeX program = luatex
\documentclass{article}

\usepackage{parskip,xparse,xcolor}
\usepackage{amsmath,lualatex-math}

\ExplSyntaxOn\makeatletter

\cs_new:Npn \tobi_save_math_style: {
   \int_case:nn { \mathstyle } {
      { \displaystyle } { \tl_set:Nn \l_tobi_saved_math_style_tl { \displaystyle } }
      { \textstyle } { \tl_set:Nn \l_tobi_saved_math_style_tl { \textstyle } }
      { \scriptstyle } { \tl_set:Nn \l_tobi_saved_math_style_tl { \scriptstyle } }
      { \scriptscriptstyle } { \tl_set:Nn \l_tobi_saved_math_style_tl { \scriptscriptstyle } }
   }
}

\NewDocumentCommand { \boxtest }{ m }{
   \mode_if_math:TF {
      % aktuellen Mathemodus ermitteln und für später sichen
      \tobi_save_math_style:
      % Box mit entsprechender Formel speichern
      \hbox_gset:Nn \l_tmpa_box {
         \begingroup
            \(
               \m@th
               \tl_use:N \l_tobi_saved_math_style_tl
               #1
            \)
         \endgroup
      }
   } {
      \hbox_set:Nn \l_tmpa_box { #1 }
   }
   \fbox { \box_use:N \l_tmpa_box }
   [\tl_to_str:n { \mathstyle } = \mathstyle; saved style = \tl_to_str:N \l_tobi_saved_math_style_tl] % for testing!
}

\cs_new:Npn \tobi_genfrac:nnnn #1#2#3#4 {
%   \colorlet { l_tobi_current_color_before_frac } { . }
%   \begingroup
%      \color { #2 }
      \genfrac { } { } { } { #1 } {
%         \color { l_tobi_current_color_before_frac }
         #3
      } {
%         \color { l_tobi_current_color_before_frac }
         #4
      }
%   \endgroup
}

\RenewDocumentCommand { \frac } { O{.} m m } {
   \tobi_genfrac:nnnn { } { #1 } { #2 } { #3 }
}

\makeatother\ExplSyntaxOff

\begin{document}

OK: $a^2 = a^{\boxtest{2}}$

OK: $\displaystyle \frac{1}{2} = \frac{\boxtest{1}}{2}$

OK: $\frac{1}{2} = \boxtest{\frac{1}{2}}$

OK: $\displaystyle \frac{1}{2} = \boxtest{\frac{1}{2}}$

wrong: $\frac{1}{2} \neq \frac{\boxtest{1}}{2}$

wrong: $\genfrac{}{}{}{}{1}{2} \neq \genfrac{}{}{}{}{\boxtest{1}}{2}$

\end{document}

So … is it possible to capture the correct style in this case too?

Update: I added an example using \genfrac, which is used in my real document do get a coloured fraction line.

Answer 1

\frac{a}{b} is essentially {a \over b} and the big problem with the infix \over is that it makes tracking the current style hard luatex does not change that.

to get a version in which \mathstyle is set you need \Ustack{a \over b} so in latex you would have to redefine \frac and related commands to use \Ustack.

One possibility would be to load lualatex-math which redefines \frac in this way.

Answer 2

This is what I came up with …

% !TeX program = luatex
\documentclass{article}

\usepackage{parskip,xparse,xcolor}
\usepackage{amsmath,lualatex-math}

\ExplSyntaxOn\makeatletter

\cs_new:Npn \tobi_save_math_style: {
   \int_case:nn { \mathstyle } {
      { \displaystyle             } { \tl_set:Nn \l_tobi_saved_math_style_tl { \displaystyle             } }
      { \crampeddisplaystyle      } { \tl_set:Nn \l_tobi_saved_math_style_tl { \crampeddisplaystyle      } }
      { \textstyle                } { \tl_set:Nn \l_tobi_saved_math_style_tl { \textstyle                } }
      { \crampedtextstyle         } { \tl_set:Nn \l_tobi_saved_math_style_tl { \crampedtextstyle         } }
      { \scriptstyle              } { \tl_set:Nn \l_tobi_saved_math_style_tl { \scriptstyle              } }
      { \crampedscriptstyle       } { \tl_set:Nn \l_tobi_saved_math_style_tl { \crampedscriptstyle       } }
      { \scriptscriptstyle        } { \tl_set:Nn \l_tobi_saved_math_style_tl { \scriptscriptstyle        } }
      { \crampedscriptscriptstyle } { \tl_set:Nn \l_tobi_saved_math_style_tl { \crampedscriptscriptstyle } }
   }
}

\NewDocumentCommand { \boxtest }{ m }{
   \mode_if_math:TF {
      % aktuellen Mathemodus ermitteln und für später sichen
      \tobi_save_math_style:
      % Box mit entsprechender Formel speichern
      \hbox_gset:Nn \l_tmpa_box {
         \begingroup
            \(
               \m@th
               \tl_use:N \l_tobi_saved_math_style_tl
               #1
            \)
         \endgroup
      }
   } {
      \hbox_set:Nn \l_tmpa_box { #1 }
   }
   \fbox { \box_use:N \l_tmpa_box }
%   [\tl_to_str:n { \mathstyle } = \mathstyle; saved style = \tl_to_str:N \l_tobi_saved_math_style_tl] % for testing!
}

% from amsmath.dtx, ll. 696--701
\DeclareRobustCommand{\genfrac}[6]{{%
\@mathstyle{#4}%
\genfrac@choice o{#1}%
{\Ustack{\begingroup#5\endgroup\ifx @#3@\@@over\else\@@above\fi#3\relax#6}}%
\genfrac@choice c{#2}%
}}

\cs_new:Npn \tobi_genfrac:nnnn #1#2#3#4 {
%   \colorlet { l_tobi_current_color_before_frac } { . }
%   \begingroup
%      \color { #2 }
      \genfrac { } { } { } { #1 } {
%         \color { l_tobi_current_color_before_frac }
         #3
      } {
%         \color { l_tobi_current_color_before_frac }
         #4
      }
%   \endgroup
}

\RenewDocumentCommand { \frac } { O{.} m m } {
   \tobi_genfrac:nnnn { } { #1 } { #2 } { #3 }
}

\setlength{\fboxsep}{1pt}
\setlength{\fboxrule}{0.3pt}

\makeatother\ExplSyntaxOff

\begin{document}

OK: $a^2 = a^{\boxtest{2}}$

OK: $a^{2^2} = a^{2^{\boxtest{2}}}$

OK: $\displaystyle \frac{1}{2} = \frac{\boxtest{1}}{2}$

OK: $\frac{1}{2} = \boxtest{\frac{1}{2}}$

OK: $\displaystyle \frac{1}{2} = \boxtest{\frac{1}{2}}$

OK: $\frac{1}{2} = \frac{\boxtest{1}}{2}$

OK: $\frac{1}{2} = \frac{1}{\boxtest{2}}$

OK: $\genfrac{}{}{}{}{1}{2} = \genfrac{}{}{}{}{\boxtest{1}}{2}$

OK: $\binom{1}{2} = \binom{\boxtest{1}}{2}$

OK: $\binom{1}{2} = \binom{1}{\boxtest{2}}$

OK: $\displaystyle\binom{1}{2} = \binom{\boxtest{1}}{2}$

OK: $\displaystyle\binom{1}{2} = \binom{1}{\boxtest{2}}$

OK: $\sum_{n=1}^\infty = \sum_{\boxtest{n=1}}^{\boxtest{\infty}}$

OK: $\displaystyle\sum_{n=1}^\infty = \sum_{\boxtest{n=1}}^{\boxtest{\infty}}$
\end{document}