Following Steinberg's advice, substitute $u = \sin x$. Note that $\cos x = \sqrt{1 - u^{2}}$ and $\tan x = u/\sqrt{1-u^{2}}$.
$$ I = \int_{0}^{\pi/4}\frac{\cos^{3}x + \sin^{3}x}{\sqrt{\cos 2x}}\,\mathrm{d}x = \int_{0}^{1/\sqrt{2}}\left(\frac{1-u^{2}}{\sqrt{1-2u^{2}}} + \frac{u^{3}}{\sqrt{1-u^{2}}\sqrt{1-2u^{2}}}\right)\mathrm{d}u$$
Substitute $x = 1 - 2u^{2}$.
$$ I = \frac{1}{4\sqrt{2}}\left(\int_{0}^{1}\frac{1+x}{\sqrt{x}\sqrt{1-x}} + \frac{1-x}{\sqrt{x}\sqrt{1+x}}\right)\mathrm{d}x $$
Using the beta function, the first term is evaluated as
$$\begin{aligned} I_{1} &= \frac{1}{4\sqrt{2}}\left(\int_{0}^{1}x^{-1/2}(1-x)^{-1/2} + x^{1/2}(1-x)^{-1/2}\right)\mathrm{d}x \\ &= \frac{1}{4\sqrt{2}}\left(\Gamma(1/2)\Gamma(1/2) + \Gamma(3/2)\Gamma(1/2)\right) = \frac{3\pi}{8\sqrt{2}} \end{aligned}$$
The second term is split into two integrals.
$$ I_{2} = \frac{1}{4\sqrt{2}}\int_{0}^{1}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{1+x}} - \frac{1}{4\sqrt{2}}\int_{0}^{1}\frac{\sqrt{x}}{\sqrt{1+x}}\,\mathrm{d}x$$
These integrals are done by substituting $u = \sqrt{x}$. The second integral requires the hyperbolic substitution $u = \sinh y$ and can be done using hyperbolic identity, etc.
$$ \frac{1}{4\sqrt{2}}\int_{0}^{1}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{1+x}} = \frac{1}{2\sqrt{2}}\int_{0}^{1}\frac{\mathrm{d}u}{\sqrt{1+u^{2}}} = \frac{\sinh^{-1}1}{2\sqrt{2}}$$
$$\begin{aligned} -\frac{1}{4\sqrt{2}}\int_{0}^{1}\frac{\sqrt{x}}{\sqrt{1+x}}\,\mathrm{d}x &= -\frac{1}{2\sqrt{2}}\int_{0}^{1}\frac{u^{2}\,\mathrm{d}u}{\sqrt{1+u^{2}}} \\ &= -\frac{1}{2\sqrt{2}}\int_{0}^{\sinh^{-1}1}\sinh^{2}y\,\mathrm{d}y \\ &= -\frac{1}{2\sqrt{2}}\left(\frac{1}{\sqrt{2}} - \frac{\sinh^{-1}1}{2}\right) \end{aligned} $$
The answer is thus
$$ \boxed{I = \frac{3\pi}{8\sqrt{2}} - \frac{1}{4} + \frac{3}{4\sqrt{2}}\,\sinh^{-1}1.} $$
$$\int_0^{\pi/4}\frac{\sin^3(x)}{\sqrt{\cos(2x)}},dx=\int_0^{\pi/4}\frac{\sin(x)(1-\cos^2(x))}{\sqrt{2\cos^2(x)-1}},dx=\int_{1/\sqrt2}^1 \frac{1-t^2}{\sqrt{2t^2-1}},dt$$
– Mark Viola Dec 27 '18 at 01:28