\begin{align}J=\int\limits_0^\infty \frac{\ln^2(1+x)}{1+x^2}\ dx\end{align}
Perform the change of variable $y=\dfrac{1}{1+x}$,
\begin{align}J&=\int\limits_0^1 \frac{\ln^2 x}{2x^2-2x+1}\ dx\\
&=\int_0^1 \frac{\ln^2 x}{\Big(1-(1+i)x\Big)\Big(1-(1-i)x\Big)}\,dx\\
&=\frac{1}{2i}\left(\int_0^1 \frac{(1+i)\ln^2 x}{1-(1+i)x}\,dx-\int_0^1 \frac{(1-i)\ln^2 x}{1-(1-i)x}\,dx\right)\\
&=2\times \frac{1}{2i}\left(\text{Li}_3(1+i)-\text{Li}_3(1-i)\right)\\
&=2\times \frac{1}{2i}\left(\text{Li}_3(1+i)-\overline{\text{Li}_3(1+i)}\right)\\
&=2\Im\Big(\text{Li}_3(1+i)\Big)
\end{align}
Since, for $\Im(a)\neq 0$,
\begin{align}\int_0^1 \frac{a\ln^2 x}{1-ax}\,dx=2\text{Li}_{3}(a)\end{align}
NB:
It can be proved easily that the identity used is true for $|a|<1$ using Taylor's expansion and usual definition of $\text{Li}_{3}(a)$ for $|a|<1$.
The two functions are analytic not only for $|a|<1$ and thus, the identity can be extended.