The best I could come up with is the following ugliness:
\left( \left. \frac ab \right| a \right)
or
\left( ab \left | a^2 \right. \right)
The downside is that you need to know which of the two parts is the larger one and appropriately enclose that in a \left-\right pair.
The other option would be to manually specify the size of the brackets:
\bigg( \frac ab \bigg| a \bigg)
Not nice, but probably the easiest to write and the hardest to get wrong. The size may need some fiddling, though (but whoever uses LaTeX is probably used to that).
Another, even more hacky solution would be to use an array:
\left( \begin{array}{c|c} \dfrac ab & a \end{array} \right)
\left( \begin{array}{c|c} ab & a^2 \end{array} \right)
This will work the same regardless of which side is larger, but care has to be taken with fractions (hence the \dfrac which forces display size) and you'll get extra space around the array which looks kinda ugly, imho. You can counter that by including negative space:
\left( \!\! \begin{array}{c|c} \dfrac ab & a \end{array} \!\! \right)
\left( \!\! \begin{array}{c|c} ab & a^2 \end{array} \!\! \right)
Two \! seem to be enough to make it good-looking again.