Since you are loading tikz, I assume a tikz solution is fine.
It makes use of two tikz libraries: matrix for the matrix (the syntax is very close to the usual syntax for tabulars, arrays or matrices, with a lot of optional parameters to tweak the final output), and fit to draw rectangles around sets of nodes.
\documentclass[11pt,twoside,openright]{book}
\usepackage{geometry,tikz,amsmath}
\usetikzlibrary{fit,matrix}
\begin{document}
\begin{equation*}
J_{5}=
\tikz[baseline=(M.west)]{%
\node[matrix of math nodes,matrix anchor=west,left delimiter=(,right delimiter=),ampersand replacement=\&] (M) {%
\lambda \& 1 \& 0 \\
0 \& \lambda \& 0 \\
0 \& 0 \& \lambda \\
};
\node[draw,fit=(M-1-1)(M-2-2),inner sep=-1pt] {};
\node[draw,fit=(M-3-3),inner sep=-1pt] {};
}
\end{equation*}
\end{document}

\fboxinstead of a\frame– Steven B. Segletes Apr 26 '13 at 16:10\framewith a\fbox. That will frame a single entry with a border that frame did not provide, but is not directly usable when multi-element blocks of the array are to be framed. – Steven B. Segletes Apr 26 '13 at 16:33arrayenvironment. – T. Verron Apr 26 '13 at 16:36