The package nicematrix provides tools to add exterior rows and columns to a matrix. It has built-in commands to draw dotted lines (with real rounded dots).
For other styles of rules, it's possible to define letters (in the preamble of the array) and commands (for the horizontal rules) for any
Tikz style of line.
You need at least version 6.6 of nicematrix (2022-02-16).
\documentclass{article}
\usepackage{nicematrix}
\usepackage{mathtools}
\usepackage{tikz}
\begin{document}
\NiceMatrixOptions{code-for-first-row = \scriptstyle,code-for-first-col = \scriptstyle }
$A = \begin{pNiceArray}{c:c}[first-row, first-col, columns-width=auto]
& r & n-r \
r & B & C \
\hdottedline
m-r & D & E
\end{pNiceArray}$
\bigskip
\NiceMatrixOptions
{
custom-line =
{
letter = I ,
command = hdashedline ,
tikz = dashed ,
width = \pgflinewidth
}
}
$A = \begin{pNiceArray}{cIc}[first-row, first-col, columns-width=auto]
& r & n-r \
r & B & C \
\hdashedline
m-r & D & E
\end{pNiceArray}$
\end{document}
You need several compilations (because nicematrix uses PGF/Tikz nodes).

\documentclassand the appropriate packages so that those trying to help don't have to recreate it. – Peter Grill Oct 26 '12 at 04:48