Let first illustrate *CarLaTeX comment! If a matrix has declared nodes (ordinary or math) and has a name, than each node in matrix has name determined by:
<name of matrix>-<row index>-<column index>
Fro example, matrix 3 x 3 with ordinary nodes, which contains names of nodes, we can write as:
\documentclass[tikz, margin=3mm]{standalone}
\usetikzlibrary{matrix}
\begin{document}
\begin{tikzpicture}
\matrix (m) [matrix of nodes,
row sep=3pt, column sep = 3pt,
font=\footnotesize
]
{
m-1-1 & m-1-2 & m-1-3\\
m-2-1 & m-2-2 & m-2-3\\
m-3-1 & m-3-2 & m-3-3\\
};
\draw[red] (m-2-1.center) -- (m-3-2.center) -- (m-1-3.center);
\end{tikzpicture}
\end{document}
where red line serve as "proof" for nodes' names.

Knowing that, your problem can be solved also on the following way:
\documentclass[tikz, margin=3mm]{standalone}
\usetikzlibrary{backgrounds, matrix}
\begin{document}
\begin{tikzpicture}
\matrix (m) [matrix of math nodes,
left delimiter = (,
right delimiter = ),
nodes={minimum size=2em}
]
{
1 & 3 & -8\\
2 & 0 & 1 \\
-7 & 9 & 1 \\
};
\begin{pgfonlayer}{background}
\draw[rounded corners, dotted, fill=green!10!white]
(m-1-1.north) -| (m-1-1.south east) -| (m-2-2.south east)
-| (m-3-3.south east) -| (m-1-1.west)
|- (m-1-1.north);
\end{pgfonlayer}
\end{tikzpicture}
\end{document}

(m-1-1)refers to the matrix entry in row 1, column 1 etc. Armed with this information you should be able to fill in the rest of the details yourself! :) – Jun 23 '17 at 21:51m-x-ymeans matrixm, column n.x, row n.y. Hence,m-1-1is the node at the first column and first row of matrixm. – CarLaTeX Jun 23 '17 at 23:02\nodeetc. in the TikZ manual. Nodes, at least, are certainly covered in the introductory tutorials in the first part of the document. – cfr Jun 23 '17 at 23:22