One can show that
$$
u := \arccos \sqrt{\frac{1-x^2}{1-x^2y^2}} \quad \text{and} \quad
v:= \arccos \sqrt{\frac{1-y^2}{1-x^2 y^2}} \quad \tag 1
$$
is a bijective mapping of the open rectangle
$M :=\{ (x,y) : 0 < x ,y < 1\}$ onto the open triangle
$\widetilde{M} := \left\lbrace (u,v) : u,v > 0, u+v < \frac{\pi}{2} \right\rbrace$.
The inverse mapping is
$$
x := \frac{\sin u}{\cos v} \quad \text{and} \quad y := \frac{\sin v }{\cos u} \tag 2
$$
as shown below.
Since
$$
(u(1, y), v(1, y)) = (\frac \pi 2, 0) \\
(u(x, 1), v(x, 1)) = (0, \frac \pi 2) \\
$$
the mapping cannot be extended one-to-one to the boundary of the square $M$.
Now let $(x, y) \in M$ and define $(u,v)$ by $(1)$. It is clear that
both $u$ and $v$ are in the range $(0, \frac \pi 2)$. To show that
$u + v < \frac \pi 2$, we compute
$$
\cos (u + v) = \cos u \cos v - \sin u \sin v =
\cos u \cos v - \sqrt{1 - \cos^2 u} \sqrt{1 - \cos^2 v} \\
= \sqrt{\frac{1-x^2}{1-x^2y^2}} \sqrt{\frac{1-y^2}{1-x^2 y^2}}
- \sqrt{\frac{x^2(1-y^2)}{1-x^2y^2}} \sqrt{\frac{y^2(1-x^2)}{1-x^2 y^2}} \\
= (1 - xy) \sqrt{\frac{1-x^2}{1-x^2y^2}} \sqrt{\frac{1-y^2}{1-x^2 y^2}}
> 0 \quad .
$$
It follows that $u + v < \frac \pi 2$.
So up to now we know that $(1)$ is a mapping from the square $M$
into the triangle $\widetilde{M}$. To show that the mapping is in fact
bijective, we verify that $(2)$ is the inverse mapping from
$\widetilde{M}$ to $M$.
Let $(u, v) \in \widetilde{M}$ and define $(x, y)$ by $(2)$.
It is clear that both $x$ and $y$ are positive. From
$u + v < \frac \pi 2$ it follows that
$$
\sin u < \sin(\frac \pi 2 - v) = \cos v \quad \Longrightarrow \quad x < 1 \, ,\\
\sin v < \sin(\frac \pi 2 - u) = \cos u \quad \Longrightarrow \quad y < 1 \, .$$
and therefore $(x, y) \in M$.
To verify that $(x, y)$ is mapped back exactly to the given $(u, v)$
is a straightforward calculation and I'll leave that final part to you :)
