Assume $a>b>0$,
\begin{align}
t &= \sqrt{a^2\sin^2 \theta+b^2\cos^2 \theta} \\
t^2 &= (a^2-b^2)\sin^2 \theta+b^2 \\
\sin^2 \theta &= \frac{t^2-b^2}{a^2-b^2} \\
t^2 &= a^2+(b^2-a^2)\cos^2 \theta \\
\cos^2 \theta &= \frac{a^2-t^2}{a^2-b^2} \\
dt &=
\frac{(a^2-b^2)\sin \theta \cos \theta}
{\sqrt{a^2\sin^2 \theta+b^2\cos^2 \theta}} \, d\theta \\
&=\frac{\sqrt{(t^2-b^2)(a^2-t^2)}}{t} \, d\theta \\
d\theta &= \frac{t \, dt}{\sqrt{(t^2-b^2)(a^2-t^2)}} \\
\end{align}
Now
\begin{align}
d\theta &= \frac{abu}{\sqrt{(a^2 u^2-1)(1-b^2 u^2)}} \, du \\
&= \frac{abu \times (ab\, du)}{\sqrt{(a^2 b^2 u^2-b^2)(a^2-a^2b^2u^2)}} \\
&= \frac{t \, dt}{\sqrt{(t^2-b^2)(a^2-t^2)}} \\
dt &= ab \, du
\end{align}
With boundary conditions:
- $\theta=0$ , $u=\dfrac{1}{r}=\dfrac{1}{b}$
- $\theta=\dfrac{\pi}{2}$ , $u=\dfrac{1}{r}=\dfrac{1}{a}$
Therefore,
$$\fbox{$u=\frac{\sqrt{a^2\sin^2 \theta+b^2\cos^2 \theta}}{ab}$}$$
Alternatively,
\begin{align}
\int \frac{t\, dt}{\sqrt{(a^2-t^2)(t^2-b^2)}} &=
\tan^{-1} \sqrt{\frac{t^2-b^2}{a^2-t^2}} \\
\int \frac{u\, du}{\sqrt{(a^2 u^2-1)(1-b^2 u^2)}} &=
\frac{1}{ab} \tan^{-1}
\left( \frac{b}{a} \sqrt{\frac{a^2 u^2-1}{1-b^2 u^2}} \right)
\end{align}