I’m not sure why you gave up on polar coordinates and why you say that the Jacobian is nasty – the Jacobian is well-known and simple, and the integrals all work out nicely.
We have $\mathrm dx\mathrm dy=r\mathrm dr\mathrm d\phi$. Denote the polar coordinates of the two points by $r_1,\phi_1$ and $r_2,\phi_2$. We need two separate cases according as the points are in the same or different octants.
For the same octant, the integral over the region $\phi_2\lt\phi_1\lt\frac\pi4$ is
\begin{eqnarray}
&&
\int_0^\frac\pi4\mathrm d\phi_1\int_0^{\phi_1}\mathrm d\phi_2\int_0^\frac1{\cos\phi_1}\mathrm r_1dr_1\int_0^\frac1{\cos\phi_2}\mathrm r_2dr_2\left(\phi_1-\phi_2\right)
\\
&=&
\frac14\int_0^\frac\pi4\mathrm d\phi_1\frac1{\cos^2\phi_1}\int_0^{\phi_1}\mathrm d\phi_2\frac{\phi_1-\phi_2}{\cos^2\phi_2}
\\
&=&
\frac14\int_0^\frac\pi4\mathrm d\phi_1\frac1{\cos^2\phi_1}\left[\left(\phi_1-\phi_2\right)\tan\phi_2-\log\cos\phi_2\right]_0^{\phi_1}
\\
&=&
-\frac14\int_0^\frac\pi4\mathrm d\phi_1\frac{\log\cos\phi_1}{\cos^2\phi_1}
\\
&=&
-\frac14\left[\tan\phi_1(1+\log\cos\phi_1)-\phi_1\right]_0^\frac\pi4
\\
&=&
\frac\pi{16}+\frac18\log2-\frac14\;.
\end{eqnarray}
For different octants, the integral over the region $\phi_2\lt\frac\pi4\lt\phi_1$ is
\begin{eqnarray}
&&
\int_\frac\pi4^\frac\pi2\mathrm d\phi_1\int_0^\frac\pi4\mathrm d\phi_2\int_0^\frac1{\cos\left(\frac\pi2-\phi_1\right)}\mathrm r_1dr_1\int_0^\frac1{\cos\phi_2}\mathrm r_2dr_2\left(\phi_1-\phi_2\right)
\\
&=&
\frac14\int_\frac\pi4^\frac\pi2\mathrm d\phi_1\frac1{\cos^2\left(\frac\pi2-\phi_1\right)}\int_0^\frac\pi4\mathrm d\phi_2\frac{\phi_1-\phi_2}{\cos^2\phi_2}
\\
&=&
\frac14\int_\frac\pi4^\frac\pi2\mathrm d\phi_1\frac1{\cos^2\left(\frac\pi2-\phi_1\right)}\left[\left(\phi_1-\phi_2\right)\tan\phi_2-\log\cos\phi_2\right]_0^\frac\pi4
\\
&=&
\frac14\int_\frac\pi4^\frac\pi2\mathrm d\phi_1\frac{\phi_1-\frac\pi4+\frac12\log2}{\cos^2\left(\frac\pi2-\phi_1\right)}
\\
&=&
\frac14\int_0^\frac\pi4\mathrm d\phi\frac{\frac\pi4+\frac12\log2-\phi}{\cos^2\phi}
\\
&=&
\frac14\left[\left(\frac\pi4+\frac12\log2\right)\tan\phi-\left(\phi\tan\phi+\log\cos\phi\right)\right]_0^\frac\pi4
\\
&=&
\frac14\log2\;.
\end{eqnarray}
There are $4$ symmetric copies of the first region and $2$ of the second, for a total of
$$
\boxed{\frac\pi4+\log2-1\approx0.4785}\;.
$$
This is not too different from the value $\frac\pi6\approx0.5236$ if the points are picked from the first quadrant of the unit disk.
Here’s Java code that checks the result by simulation.