\mathpalette needs two arguments, the first of which should be a two argument macro. With your code, the call
\mathsbox{2}
expands to
\expandafter\mathpalette\expandafter\math@sbox{2}
and \expandafter does nothing at all, so you get
\mathpalette\math@sbox{2}
Now, \math@sbox is defined with a single argument, which is bad. According to the definition of \mathpalette
% latex.ltx, line 4289:
\def\mathpalette#1#2{%
\mathchoice
{#1\displaystyle{#2}}%
{#1\textstyle{#2}}%
{#1\scriptstyle{#2}}%
{#1\scriptscriptstyle{#2}}}
you now get
\mathchoice
{\math@sbox\displaystyle{2}}%
{\math@sbox\textstyle{2}}%
{\math@sbox\scriptstyle{2}}%
{\math@sbox\scriptscriptstyle{2}}}
This typesets four math subformulas expanding each argument; the first becomes
\sbox0{$\m@th\displaystyle$}{2}
and similarly for the other three. So \box0 is set to a box containing an empty formula.
Even if you define \math@sbox with two arguments, you get nothing really useful, because \box0 will turn out to be void.
If you want to properly save the box in the style current at the time of the definition, you can use the mathstyle package (but it has some quirks, beware).
\documentclass{amsart}
\usepackage{mathstyle}
\makeatletter
\newcommand{\mathsavebox}[2]{%
\global\setbox#1=\hbox{$\m@th\currentmathstyle#2$}%
}
\makeatother
\newsavebox{\mybox}
\begin{document}
\begin{equation*}
x^{\mathsavebox{\mybox}{2}} = a^{\usebox{\mybox}}{}^2
\end{equation*}
\end{document}
The {}^2 is meant to show that the exponent is in the correct style (with your code it isn't).

\ThisStylemacro of thescalerelpacakge can help you, but I don't understand your quesiton fully. What is\mathsboxsupposed to accomplish? – Steven B. Segletes Jan 13 '17 at 17:11{}^2aftera^{\usebox{\mybox}}to see it. – egreg Jan 14 '17 at 00:10\ThisStyle{}"process", here are some relevant answers: http://tex.stackexchange.com/questions/78872/how-to-capture-the-current-math-style/114658#114658 and http://tex.stackexchange.com/questions/43978/proper-use-of-mathchoice/114594#114594 – Steven B. Segletes Jan 14 '17 at 16:51