For questions like this a matrix of math nodes is your friend. These are tikz matrix that allow you to draw lines connecting the matrix entries.
Here the output I get:

and here is the code:
\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{matrix}
\begin{document}
\[ A =
\begin{tikzpicture}[baseline=(current bounding box.center),
large/.style={font=\large}]
\matrix (M)[matrix of math nodes, nodes in empty cells,
left delimiter={[}, right delimiter={]},
column sep={1.2em,between origins},
row sep={1.2em,between origins}
]{ 0 & 1 & 0 & & \\
0 & 0 & 1 & & & & & \\
0 & 0 & 0 & & & \\
& & & 2 & & \\
& & & & 2 & 1 & 0 & \\
& & & & 0 & 2 & 1 & \\
& & & & 0 & 0 & 2 & \\
& & & & & & & 0\\
};
\draw(M-3-1.south west)--([xshift=2mm]M-3-4.south east);
\draw(M-4-3.south)--(M-4-4.south east);
\draw(M-7-5.south)--([xshift=2mm]M-7-7.south);
\draw(M-1-3.north east)--(M-4-4.south west);
\draw(M-4-4.north east)--(M-5-5.south west);
\draw(M-5-7.north east)--(M-7-7.south east);
\node[large] at (M-2-7){$0$};
\node[large] at (M-7-2){$0$};
\end{tikzpicture}
\]
\end{document}
Some words of explanation:
- The
(M) after \matrix means that the nodes have labels (M-1-1), (M-1-2) etc. You can change (M) to anything you like.
- The node
(M-3-1.south west) is the south west corner of the entry in row 3 and column 1. Simiarly, there are north, east, south, ...
- I have used a
matrix of math nodes but there is also a matrix of nodes. The difference, as you'd suspect, is that matrix of math nodes puts the matrix entries into math-mode.
- the
nodes in empty cells creates labels for empty cells.
- I didn't put the large
0s into the matrix because this would have warped the row and column sizes
- The lines
column sep={1.2em,between origins} and row sep={1.2em,between origins} makes the distances between the center of the rows and columns the same, which you want if you are drawing vertical and horizontal lines between the matrix entries
- you need to put braces around the delimiters in
left delimiter={[} and right delimiter={]}
- the
baseline=(current bounding box.center) is there to center the matrix in the displayed equation
- the
large/.style={font=\large} sets the font size for the "large" zeros in the matrix (this is the meaning of the large in \node[large]).