This can be done using the \multicolumn command. The \cline command makes the partial horizontal line. The 1-column \multicolumn in the last row eliminates the last portion of the vertical line. Using \multicolumn for the U^{(k)}_a entry automatically centers it horizontally instead of placing it in the third column.
I also added a bit of space between your rows by setting \arraystretch to 1.6.
\documentclass[a4paper]{article}
\usepackage{amsmath}
\begin{document}
\begin{equation}
U^{(k+1)}_a=
\left(
\renewcommand{\arraystretch}{1.6}
\begin{array}{ccccc|c}
& & & & & \left(U^{(k+1)}_a\right)_{0k} \\
& & & & & \left(U^{(k+1)}_a\right)_{1k} \\
\multicolumn{5}{c|}{U^{(k)}_a} & \vdots \\
& & & & & \vdots \\
& & & & & \vdots \\
\cline{1-5}
a_k & 0 & \dots & 0 & \multicolumn{1}{c}{0} & \left (U^{(k+1)}_a\right)_{kk}
\end{array}
\right).
\end{equation}
\end{document}

I would also probably delete the 4th column and two of the rows of \vdots, and use \cdots instead of \dots, but that's just my opinion. If you add \usepackage{array} you'll get a cleaner join between the horizontal and vertical lines.
U^{(k+1)}_a=
\left(
\renewcommand{\arraystretch}{2}
\begin{array}{cccc|c}
& & & & \left(U^{(k+1)}_a\right)_{0k} \\
\multicolumn{4}{c|}{U^{(k)}_a} & \left(U^{(k+1)}_a\right)_{1k} \\
& & & & \vdots \\
\cline{1-4}
a_k & 0 & \cdots & \multicolumn{1}{c}{0} & \left (U^{(k+1)}_a\right)_{kk}
\end{array}
\right).

\documentclass, includes all relevant\usepackagecommands, ends with\end{document}and compiles without errors, even if it does not produce your desired output. – Sandy G Nov 13 '17 at 05:47