\mathpalette comes very handy in many situations where similar token lists must be passed to \mathchoice, that is, we want an expansion of the form
\mathchoice
{<something in \displaystyle>}
{<something in \textstyle>}
{<something in \scriptstyle>}
{<something in \scriptscriptstyle>}
However, to allow for maximum flexibility, the first argument to \mathpalette must be a two-argument macro, whose first argument is a math style declaration. Let's see an example, Plain TeX definition of \in:
\def\notin{\mathrel{\mathpalette\c@ncel\in}}
This is followed by
\def\c@ncel#1#2{\m@th\ooalign{$\hfil#1\mkern1mu/\hfil$\crcr$#1#2$}}
Don't look at the complications of \c@ncel but at the fact that the first argument must indeed be a style declaration. If TeX finds \notin it will do \mathrel{\mathpalette\c@ncel\in}, and the subformula inside \mathrel is expanded into
\mathchoice
{\c@ncel\displaystyle{\in}}%
{\c@ncel\textstyle{\in}}%
{\c@ncel\scriptstyle{\in}}%
{\c@ncel\scriptscriptstyle{\in}}
so, in text style the math list to typeset will be \c@ncel\textstyle{\in}, that is
\m@th\ooalign{$\hfil\textstyle\mkern1mu/\hfil$\crcr$\textstyle\in$}
Here \ooalign is the trick that allows for superimposing characters (it's a simple \halign but baseline skip is suppressed). The effect is of superimposing a slighly offset bar to \in.
The call of \mathpalette can also be of the form \def\xyz{\mathpalette\XYZ} and in this case the macro \xyz will appear to take an argument, which will actually be the second argument for \XYZ. And this is why the macro is defined this way.
Your \vfrac macro can be defined via \mathpalette, since the structure is the same; but the two arguments must be gathered before calling \mathpalette and stored in some macros:
\catcode`@=11
\def\vfrac#1#2{\def\vf@num{#1}\def\vf@den{#2}\mathpalette\vfr@c\relax}
\def\vfr@c#1#2{%
\raise.5ex\hbox{$\m@th#1\vf@num$}\!/\!\lower.25ex\hbox{$\m@th#1\vf@den$}}
\catcode`@=12
(As Aditya observes, \m@th should always be called in these cases.) Here we don't need the second argument for \vfr@c, but it must be there anyway; one can use \relax or {}, it's the same: that second argument will never be used.
\vfracusing\def\vfrac#1#2{\mathpalette\relax{\vfr@c{#1}{#2}}}or\def\vfrac#1#2{\mathpalette{\vfr@c{#1}{#2}}\relax}solve your problem? – Werner Nov 10 '11 at 19:15\hboxcontents do not get the correct size (now that I think about it, how could they). I should somehow be able to put the first argument to\mathpaletteas an argument to\vr@cI think. – morbusg Nov 10 '11 at 19:19