Note that the integral can(not) be simplified as $$ 2\int_0^{1/\sqrt 3} \frac{x^4}{1-x^4} \cos^{-1} \left(\frac{2x}{1+x^2}\right)dx $$ Since $\cos^{-1}$ is not an even function. Let $x=\tan y$ $$ \implies \int_0^{\pi/6}\frac{x^4}{1-x^4} \left(\frac{\pi}{2} -2y\right) \sec^2(y)\ dy $$
So how do I solve the original problem?
Hint:
$$\int_{-a}^a\dfrac{x^4}{1-x^4}\cos^{-1}\dfrac{2x}{1+x^2}\ dx$$
$$=\int_{-a}^0\dfrac{x^4}{1-x^4}\cos^{-1}\dfrac{2x}{1+x^2}\ dx+\int_0^a\dfrac{x^4}{1-x^4}\cos^{-1}\dfrac{2x}{1+x^2}dx=I_1+I_2$$
For the first integral set $x=-y,dx=-dy$
$$I_1=\int_{-a}^0\dfrac{x^4}{1-x^4}\cos^{-1}\dfrac{2x}{1+x^2}\ dx=-\int_a^0\dfrac{y^4}{1-y^4}\cos^{-1}\dfrac{(-2y)}{1+y^2}dy$$
$$=\int_0^a\dfrac{y^4}{1-y^4}\cos^{-1}\dfrac{(-2y)}{1+y^2}dy$$
Using How do I prove that $\arccos(x) + \arccos(-x)=\pi$ when $x \in [-1,1]$?,
$$I_1=\int_0^a\dfrac{y^4}{1-y^4}\left(\pi-\cos^{-1}\dfrac{2y}{1+y^2}\right)dy=\pi\int_0^a\dfrac{y^4}{1-y^4}dy-I_2$$
Hope you can take it from here!
We will integrate by parts to get rid of the arccos. The remaining integral will be trivial by symmetry. Observe that (at least for $0 < x < 1$) $$ \partial_x \cos^{-1} \left( \frac{2x}{1+x^2}\right) = \frac{2(1+x^2) - (2x)(2x)}{1+x^2} \frac{-1}{\sqrt{1 - \left(\frac{2x}{1+x^2} \right)^2}} = \frac{-2}{1+x^2}. $$ Also, $$ \frac{x^4}{1-x^4} = -1 + \frac{1}{2} \left[\frac{1}{1+x^2} + \frac{1}{1-x^2} \right]. $$ Integrating by parts now yields $$ \int_{-\frac{1}{\sqrt{3}}}^{\frac{1}{\sqrt{3}}} dx \frac{x^{4}}{1-x^{4}}\cos^{-1}\left(\frac{2x}{1+x^{2}}\right) = \left( -x + \frac{1}{2} \tan^{-1} x + \frac{1}{2}\tanh^{-1} x \right) \cos^{-1}\left(\frac{2x}{1+x^{2}}\right) \left. \right\lvert _{-\frac{1}{\sqrt{3}}}^{\frac{1}{\sqrt{3}}} - \int_{-\frac{1}{\sqrt{3}}}^{\frac{1}{\sqrt{3}}} dx \left( -x + \frac{1}{2} \tan^{-1} x + \frac{1}{2} \tanh^{-1} x \right) \frac{-2}{1+x^2} \\= \pi \left[\frac{\pi}{12} - \frac{1}{\sqrt 3} + \frac{1}{4} \log \left(\frac{\sqrt 3 + 1}{\sqrt 3 - 1} \right) \right], $$ because the integrand is odd. I fixed a calculation error in my result (a factor 3).
$$I=\int_{-\frac{1}{\sqrt{3}}}^{\frac{1}{\sqrt{3}}}\frac{x^{4}}{1-x^{4}}\cos^{-1}\left(\frac{2x}{1+x^{2}}\right)dx=\int_{-\frac{1}{\sqrt{3}}}^{\frac{1}{\sqrt{3}}}\frac{x^{4}}{1-x^{4}}\cos^{-1}\left(\frac{-2x}{1+x^{2}}\right)dx=\int_{-\frac{1}{\sqrt{3}}}^{\frac{1}{\sqrt{3}}}\frac{x^{4}}{1-x^{4}}\left(\pi-\cos^{-1}\left(\frac{2x}{1+x^{2}}\right)\right)dx$$ $$\Rightarrow I=\frac{\pi}{2}\int_{-1/\sqrt{3}}^{1/\sqrt{3}}\frac{x^4}{1-x^4}\,dx=\frac{\pi}{2}\int_{-1/\sqrt{3}}^{1/\sqrt{3}}\left(-1+\frac{1}{2(1+x^2)}+\frac{1}{2(1-x^2)}\right)\,dx$$ $$\Rightarrow I=-\frac{2}{\sqrt{3}}+\frac{\pi}{6}+\frac{1}{2}\ln\left(\frac{\sqrt{3}+1}{\sqrt{3}-1}\right)$$
