Short answer: no — there is no way to do this correctly in all situations or in one LaTeX pass.
After a lot of learning and experimentation, it seems that the only way around this is something like Manuel's answer and comment here. There is no such thing as \whatIsTheMathStyleHere (unless we are on LuaTeX). The \mathchoice command is a TeX primitive. The choice of the four options is made late. To understand what "late" means, we need to appreciate that TeX has a few digestive stages: eye, mouth, gullet, stomach, each of which transforms the logical document into a form closer to the final visual document, working through a subset of the semantically meaningful tokens. Sometimes it goes back to the previous stage. (But doesn't its almost too-good-to-be-true typesetting intelligence that we've come to expect justify some degree of complexity?) So, the stuff surrounding \mathchoice and inside of its arguments gets macro-expanded before \mathchoice itself is applied. In fact, as far as I can tell, not just expansion, but other TeX commands, such as \ifx, get processed before \mathchoice, so within each stage of processing there might be several stratified substages?
Manuel's alternative, called \mathcase, uses a clever trick, where a nanoscale space is inserted, of a size dependent on the math style. And then it checks the size of the space and determines the style. How can that work, if the final math style everywhere is supposed to be determined only after the document becomes a fully resolved structure? It does so using the zref-savepos package. Reading the documentation for zref, there is a note:
The page position is not instantly known. First the page must be constructed by TEX’s asynchronous output routine. Thus the time where the position is known is the page shipout time. Thus a reference system where the information is recorded in the first run and made available for use in the second run comes in handy.
In other words, the correct math style is determined at shipout, at which point it can't be used in any \if or other decision-making structures, but it is stored in the AUX file (as you can see by inspecting the AUX file for the example below), and becomes available during the second pass.
Unfortunately, this causes the following sort of problem, with cascading math styles propagating down a nested series of \mathcases: if the nesting is N levels deep, then the document is not guaranteed to be stable until it is compiled N times over. For example:
(This is from Manuel's answer, where I just changed the document body.)
\documentclass{scrartcl}
\usepackage{mathtools}
\usepackage{zref-savepos}
\newcount\mmstynum
\newcount\tmpnum
\protected\def\getmsty
{\global\advance\mmstynum1 %
\expandafter\getstyA\expandafter{\number\mmstynum}}
\protected\def\getstyA#1%
{\zsaveposx{mmsty-#1-a}%
\mathchoice{\kern4sp}{\kern3sp}{\kern2sp}{\kern1sp}%
\zsaveposx{mmsty-#1-b}
\tmpnum=\numexpr
\zposx{mmsty-#1-b} - \zposx{mmsty-#1-a}
\relax
\ifcase\tmpnum\or\kern-1sp \or\kern-2sp \or\kern-3sp \or\kern-4sp \fi}
\protected\def\mstycase
{\ifcase\tmpnum\expandafter\gobblefour\or\expandafter\usefourth\or
\expandafter\usethird\or\expandafter\usesecond\or
\expandafter\usefirst\else\expandafter\gobblefour\fi}
\protected\def\mathcase{\getmsty\mstycase}
\long\def\gobblefour#1#2#3#4{}
\long\def\usefourth#1#2#3#4{#4}
\long\def\usethird#1#2#3#4{#3}
\long\def\usesecond#1#2#3#4{#2}
\long\def\usefirst#1#2#3#4{#1}
\def\tmpa{If this prints it means it doesn't work}
\newcommand*\foo
{\mathcase % compare with \mathchoice
{\def\tmpa{displaystyle}}
{\def\tmpa{textstyle}}
{\def\tmpa{scriptstyle}}
{\def\tmpa{scriptscriptstyle}}
\text\tmpa}
\let\savedstyle\displaystyle
\def\thisstyle#1{\begingroup\mathcase{\let\savedstyle\displaystyle}{\let\savedstyle\textstyle}{\let\savedstyle\scriptstyle}{\let\savedstyle\scriptscriptstyle}#1\endgroup}
\begin{document}
\[
\sum \scriptstyle \sum \thisstyle{%
\fbox{\( \sum \savedstyle \sum \thisstyle{%
\fbox{\( \sum \savedstyle \sum \thisstyle{%
\fbox{\( \sum \savedstyle \sum \)}%
}\)}%
}\)}%
}
\]
\end{document}
Looking at the PDF in TeXworks viewer while compiling it, we see:
After the first pass:
After the second pass:
After the third pass:
After the fourth pass:
So, the solution trades off the number of passes to improve the speed of each pass. (Though it would be good if it also warned you when the document is not yet stable.) While not a perfect solution, I'm convinced that this is just about the best alternative that is possible, given the nature of TeX (and TeX itself is unlikely to change in this aspect in the foreseeable future).
There is also the mathstyle package (see discussion here), which works around \mathchoice by tapping into all of the math mode delimiters (but not \( apparently!), and injecting some logic to keep track of what the current math style is going to be, out of band with TeX. But, as pointed out in Aditya's comment here, there are cases where it fails to predict TeX's behaviour. For example, with the use of \over, \atop TeX primitives. (The comment is 5 years old though. Maybe it had become better in the meantime?)
So the best answer is to reduce the number of \mathchoice levels of nesting (note that a \mathchoice is contained in \ThisStyle, \text, \mathpalette, etc.) where possible (e.g. I removed a few redundant \TextStyles in the precursor question), or if you can infer the math style using common sense reasoning, you could redefine \mathchoice locally to short-circuit it.
All in all, it's a surprising outcome. This is the first time that I wanted to do something with LaTeX and it turned out to be impractical. Also, while trying to hack around \mathchoice, I've met all the horrors of macro languages, where you never quite know what's going to expand in what order. (Just take a look at the "classical" way being \expandafter\expandafter\expandafter\expandafter\expandafter... in this solution, and you know something's wrong.) Finally, for software from the seventies, the semantics of \mathchoice seem somewhat inefficient, even if its intended purpose is just to typeset a single custom symbol...
\ThisStyle{...\SavedStyle...}is indeed just a more convenient way of performing a\mathchoice. And nesting of them goes as 4^nest-level boxes to be created. I suggest looking at my answer at http://tex.stackexchange.com/questions/172455/necessity-of-nested-text-within-math-mode-for-proper-mathchoice-based-scaling/173070#173070 for ways to cope with it. I say "cope" because there is no fix for the nesting of\mathchoices but sometimes, if you know something about what is being typeset, you can hardwire at least some of the choices. – Steven B. Segletes Feb 17 '16 at 13:23