Your problem is one of diagonalizing a quadratic form. There are several methods available for doing this. You might for instance compute eigenvalues and eigenvectors of the associated symmetric matrix $$Q = \begin{bmatrix}4&3&5\\3&2&2\\5&2&-6\end{bmatrix}.$$ In this case, we’re pretty lucky because the characteristic polynomial is fairly “nice:” $\lambda^3-66\lambda$, from which the eigenvalues are $0$ and $\pm\sqrt{66}$. This gives you the information you need to classify the equation, which only requires knowing the spectrum of the quadratic form. To proceed onto the canonical form, you’ll need to find a set of corresponding eigenvectors, which doesn’t take too much more work.
In practice, the eigenvalues and eigenvectors aren’t usually this nice, but there are other ways to diagonalize the quadratic form. Another common method is successively completing squares. One possible result of doing this is $$\frac14\left(4\lambda_1+3\lambda_2+5\lambda_3\right)^2-\frac14\left(\lambda_2+7\lambda_3\right)^2.$$ The spectrum of these terms matches the spectrum obtained from the eigenvalues, which we expect.
If you’re not good at completing squares, there’s an entirely mechanical algorithm described here for diagonalizing a quadratic form. Omitting the gory details of the computation, for this problem the algorithm eventually produces the augmented matrix $$\left[\begin{array}{ccc|ccc}4&0&0 & 1&0&0 \\ 0&-4&0 & -3&4&0 \\ 0&0&0 & -64&112&-16 \end{array}\right].$$ From the diagonal of the left-hand side, we get the same spectrum as with the previous two methods. To find the canonical form, we must then invert the right-hand side, which produces the diagonalization $$4\left(\lambda_1+\frac34\lambda_2+\frac54\lambda_3\right)^2-4\left(\frac14\lambda_2+\frac74\lambda_3\right)^2$$ that happens to be the same as the one obtained by completing the squares above.