$$ \cos ^2\left(x\right)+\cos ^2\left(2x\right)+\cos ^2\left(3x\right)=\frac{3}{2} $$
How can I solve this one, I mean I get something like this: $-3+\left(-1+2\cos ^2\left(x\right)\right)^22+2\left(-3\cos \left(x\right)+4\cos ^3\left(x\right)\right)^2+2\cos ^2\left(x\right)=0$
This equation seems rather hard to solve from here, any tips or other ways to come to an solution?
Linearise the left-hand side: we start from formula: $$\cos^2u=\frac12(1+\cos 2u)$$ from which we obtain the equation: $$\frac12(3+\cos 2x+\cos 4x+\cos 6x)=\frac32\iff\cos 2x+\cos 4x+\cos 6x=0.$$ Now factorise the left-hand side: \begin{align*} \cos 2x+\cos 4x+\cos 6x&=\cos(4x- 2x)+\cos 4x+\cos(4x+2x)\\ &=2\cos4x\cos2x+\cos 4x=\cos 4x(2\cos2x+1) \end{align*} Thus the equation is equivalent to the system of standard equations: $$\begin{cases} \cos 4x=0\\\cos2x=-\dfrac12\end{cases}$$ Solutions: $$\begin{cases} 4x\equiv \dfrac\pi2&\bmod\pi\\ 2x\equiv \pm\dfrac{2\pi}3&\bmod2\pi \end{cases}\iff \begin{cases} x\equiv \dfrac\pi8&\bmod\dfrac\pi4\\ x\equiv \pm\dfrac\pi3&\bmod\pi \end{cases}$$
Doing it the way you originally started gives us $$-3+\left(-1+2\cos ^2\left(x\right)\right)^22+2\left(-3\cos \left(x\right)+4\cos ^3\left(x\right)\right)^2+2\cos ^2\left(x\right)=0$$ Now let $\cos x = u$ and simplify to get $$32u^6 - 40u^4 + 12u^2 - 1 = 0$$ which is simply a cubic in $u^2$ and can be factorised as $$4\left(\frac{1}{2}+u\right)\left(-\frac{1}{2}+u\right)\left(8u^4-8u^2+1\right)=0$$ giving us $u = \pm \frac{1}{2}$ and we need to solve the remaining quartic(which is a quadratic in $u^2$) to get $$u=-\frac{\sqrt{2-\sqrt{2}}}{2},\:u=\frac{\sqrt{2-\sqrt{2}}}{2},\:u=-\frac{\sqrt{2+\sqrt{2}}}{2},\:u=\frac{\sqrt{2+\sqrt{2}}}{2}$$ as our other solutions.
So we have:
$$\cos \left(x\right)=\frac{1}{2}\implies x=\frac{\pi }{3}+2\pi n,\:x=\frac{5\pi }{3}+2\pi n$$
$$\cos \left(x\right)=-\frac{1}{2}\implies x=\frac{2\pi }{3}+2\pi n,\:x=\frac{4\pi }{3}+2\pi n$$
$$\cos \left(x\right)=-\frac{\sqrt{2+\sqrt{2}}}{2} \implies x=2\pi n-\arccos \left(-\frac{\sqrt{2+\sqrt{2}}}{2}\right), \\ x=2\pi n+\arccos \left(-\frac{\sqrt{2+\sqrt{2}}}{2}\right)$$
$$\cos \left(x\right)=-\frac{\sqrt{2-\sqrt{2}}}{2}\implies x=2\pi n-\arccos \left(-\frac{\sqrt{2-\sqrt{2}}}{2}\right),\\ x=2\pi n+\arccos \left(-\frac{\sqrt{2-\sqrt{2}}}{2}\right)$$
$$\cos \left(x\right)=\frac{\sqrt{2-\sqrt{2}}}{2}\implies x=2\pi n-\arccos \left(\frac{\sqrt{2-\sqrt{2}}}{2}\right),\\ x=2\pi n+\arccos \left(\frac{\sqrt{2-\sqrt{2}}}{2}\right)$$
$$\cos \left(x\right)=\frac{\sqrt{2+\sqrt{2}}}{2}\implies x=2\pi n-\arccos \left(\frac{\sqrt{2+\sqrt{2}}}{2}\right),\\ x=2\pi n+\arccos \left(\frac{\sqrt{2+\sqrt{2}}}{2}\right)$$
I shall use $\sum \cos$ when angles are in arithmetic progression whose usefulness is more apparent if the number of summands is more than three.
If $\sin x=0,x=n\pi\implies\cos2x+\cos4x+\cos6x=1+1+1\ne0$
So, $\cos2x+\cos4x+\cos6x=0\implies\sin x\ne0$
$$\dfrac{2\sin x\cos2x+2\sin x\cos4x+2\sin x\cos6x}{2\sin x}=\dfrac{\sin7x-\sin x}{2\sin x}$$
Now $\sin7x=\sin x\implies7x=m\pi+(-1)^mx$ where $m$ is any integer
If $m$ is even $=2r$(say) $7x=2r\pi+x\iff x=\dfrac{r\pi}3$
If $m$ is odd $=2r+1$(say) $7x=(2r+1)\pi-x\iff x=\dfrac{(2r+1)\pi}8$
Let $\mu = \cos 2x$ for shortness.
Squaring $$ \cos (2x\pm x) = \cos 2x \cos x \mp \sin 2x \sin x $$ we obtain $$ \cos^2 (2x\pm x) = \cos^2 2x \cos^2 x + \sin^2 2x \sin^2 x \mp \text{cancelling terms} = \\ = \mu^2 \frac{1+\mu}{2}+(1-\mu^2)\frac{1-\mu}{2} \mp \text{cancelling terms} = \\ = \mu^3 - \frac{\mu}{2} + \frac{1}{2} \mp \text{cancelling terms} $$ So the original equation becomes $$ 2\mu^3 + \mu^2 - \mu + 1 = \frac{3}{2}\\ 2\mu^3 + \mu^2 - \mu - \frac{1}{2} = 0\\ (2\mu+1)(2\mu^2 - 1) = 0 $$ with solutions $$ \mu_1 = -\frac{1}{2},\quad \mu_{2,3} = \pm \frac{1}{\sqrt{2}} $$
