Added: "Repeated Completing the Square" gives the same outcome this time. The only requirement is Sylvester's Law of Inertia, the details could have been different. You begin with $5x^2 - 8 y^2 - 4 y z + 10 xy.$ We take care of two terms with $5(x+y)^2.$ But now we need $-13 y^2 - 4 y z . $ So we take $-13 (y + \frac{2z}{13})^2.$ At the end, we need to add back $\frac{4}{13} z^2.$
$$ Q^T D Q = H $$
$$\left(
\begin{array}{rrr}
1 & 0 & 0 \\
1 & 1 & 0 \\
0 & \frac{ 2 }{ 13 } & 1 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - 13 & 0 \\
0 & 0 & \frac{ 4 }{ 13 } \\
\end{array}
\right)
\left(
\begin{array}{rrr}
1 & 1 & 0 \\
0 & 1 & \frac{ 2 }{ 13 } \\
0 & 0 & 1 \\
\end{array}
\right)
= \left(
\begin{array}{rrr}
5 & 5 & 0 \\
5 & - 8 & - 2 \\
0 & - 2 & 0 \\
\end{array}
\right)
$$
So, there are two positive and one negative eigenvalue. The actual eigenvalues are approximately -10.0579, 0.2939, 6.7639.
The method, sometimes called "congruence diagonalization"
Algorithm discussed at http://math.stackexchange.com/questions/1388421/reference-for-linear-algebra-books-that-teach-reverse-hermite-method-for-symmetr
https://en.wikipedia.org/wiki/Sylvester%27s_law_of_inertia
$$ H = \left(
\begin{array}{rrr}
5 & 5 & 0 \\
5 & - 8 & - 2 \\
0 & - 2 & 0 \\
\end{array}
\right)
$$
$$ D_0 = H $$
$$ E_j^T D_{j-1} E_j = D_j $$
$$ P_{j-1} E_j = P_j $$
$$ E_j^{-1} Q_{j-1} = Q_j $$
$$ P_j Q_j = Q_j P_j = I $$
$$ P_j^T H P_j = D_j $$
$$ Q_j^T D_j Q_j = H $$
$$ H = \left(
\begin{array}{rrr}
5 & 5 & 0 \\
5 & - 8 & - 2 \\
0 & - 2 & 0 \\
\end{array}
\right)
$$
==============================================
$$ E_{1} = \left(
\begin{array}{rrr}
1 & - 1 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
$$
$$ P_{1} = \left(
\begin{array}{rrr}
1 & - 1 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; Q_{1} = \left(
\begin{array}{rrr}
1 & 1 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; D_{1} = \left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - 13 & - 2 \\
0 & - 2 & 0 \\
\end{array}
\right)
$$
==============================================
$$ E_{2} = \left(
\begin{array}{rrr}
1 & 0 & 0 \\
0 & 1 & - \frac{ 2 }{ 13 } \\
0 & 0 & 1 \\
\end{array}
\right)
$$
$$ P_{2} = \left(
\begin{array}{rrr}
1 & - 1 & \frac{ 2 }{ 13 } \\
0 & 1 & - \frac{ 2 }{ 13 } \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; Q_{2} = \left(
\begin{array}{rrr}
1 & 1 & 0 \\
0 & 1 & \frac{ 2 }{ 13 } \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; D_{2} = \left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - 13 & 0 \\
0 & 0 & \frac{ 4 }{ 13 } \\
\end{array}
\right)
$$
==============================================
$$ P^T H P = D $$
$$\left(
\begin{array}{rrr}
1 & 0 & 0 \\
- 1 & 1 & 0 \\
\frac{ 2 }{ 13 } & - \frac{ 2 }{ 13 } & 1 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
5 & 5 & 0 \\
5 & - 8 & - 2 \\
0 & - 2 & 0 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
1 & - 1 & \frac{ 2 }{ 13 } \\
0 & 1 & - \frac{ 2 }{ 13 } \\
0 & 0 & 1 \\
\end{array}
\right)
= \left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - 13 & 0 \\
0 & 0 & \frac{ 4 }{ 13 } \\
\end{array}
\right)
$$
$$ Q^T D Q = H $$
$$\left(
\begin{array}{rrr}
1 & 0 & 0 \\
1 & 1 & 0 \\
0 & \frac{ 2 }{ 13 } & 1 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - 13 & 0 \\
0 & 0 & \frac{ 4 }{ 13 } \\
\end{array}
\right)
\left(
\begin{array}{rrr}
1 & 1 & 0 \\
0 & 1 & \frac{ 2 }{ 13 } \\
0 & 0 & 1 \\
\end{array}
\right)
= \left(
\begin{array}{rrr}
5 & 5 & 0 \\
5 & - 8 & - 2 \\
0 & - 2 & 0 \\
\end{array}
\right)
$$