Problem:
Given the equation $$5x^2 + 5y^2 - 6xy - 8 = 0$$ defining a non-degenerate conic section, find a rotation of the variables, such that the cross term $-6xy$ disappears in the new coordinates $(\bar x, \bar y)$.
Attempts:
Given the standard 2D rotation matrix $\textbf R$, I figured I'd invert it, and use that to find a way to get $\bar x \bar y = 0$.
That's where I'm stuck. I can't seem to find a way to do this, given $\textbf R^{-1} = \begin{bmatrix}\cos\theta & \sin\theta \\ -\sin\theta & \cos\theta\end{bmatrix}$.
Any tips?