The following MWE points out that, without fleqn class option, math internal mode is detected ("i" and "o") but, as soon as fleqn class option is enabled, this mode is not detected anymore ("i" and "i").
Do you see what's going on?
\documentclass
%[fleqn]
{article}
%
\newcommand{\test}{%
\relax%
\ifinner%
i%
\else%
o%
\fi%
}
%
\begin{document}
$\test$
%
\begin{equation}
\test
\end{equation}
\end{document}
\ifinner does not really test anything useful in math mode whether or not fleqn is used.
If you want to distinguish displaystyle then unfortunately you need to use \mathchoice.
The following produces t inline and d in display whether or not the [fleqn] option is used.
\documentclass
[fleqn]
{article}
%
\newcommand{\test}{%
\mathchoice
{\displaystyle d}%
{\textstyle t}%
{\scriptstyle s}%
{\scriptscriptstyle 2}%
}
%
\begin{document}
$\test$
%
\begin{equation}
\test
\end{equation}
\end{document}
\displaystyle vs non \displaystyle matters, isn't amsmath's \if@display reliable?
– Denis Bitouzé
Mar 31 '16 at 15:56
$$ or eqnarray or any other non-amsmath math construct then yes.
– David Carlisle
Mar 31 '16 at 15:58
\ifinnerhas never been a good test for being in inline or display math. – egreg Mar 31 '16 at 13:41fleqnyou have forequationsomething like$\m@th\displaystyle \hskip\mathindent\bgroup.... Withoutfleqnyou have plain$$.... – Henri Menke Mar 31 '16 at 13:42amsmath's\if@displayis more reliable but strange anyway, at least for me :) – Denis Bitouzé Mar 31 '16 at 13:46fleqnyou will end up in$$...$$, withfleqnyou will end up in$...$. For more look atarticle.clsandfleqn.clo. – Henri Menke Mar 31 '16 at 13:49\ifinneris a useful test in math mode. – David Carlisle Mar 31 '16 at 14:06