Brace next to matrix

Question

\begin{equation}\co_\beta(Au_{d+1})=
\begin{bmatrix}\langle u_1,Au_{d+1}\rangle \\ \vdots \\ \langle u_n,Au_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle Au_1,u_{d+1}\rangle \\ \vdots \\ \langle Au_n,u_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle \lambda u_1,u_{d+1}\rangle \\ \vdots \\ \langle \lambda u_d,u_{d+1}\rangle \\ \rule[.5ex]{4.5em}{0.4pt} \\ * \\ \vdots \\ *\end{bmatrix}=
\begin{bmatrix}0 \\ \vdots \\ 0  \\ \rule[.5ex]{1em}{0.4pt} \\ * \\ \vdots \\ *\end{bmatrix}
\begin{tabular}{l}\left.\lefteqn{\phantom{\begin{matrix} 0 \\ \vdots\\ 0 \end{matrix}}}\right\}d\text{ nullen} \\ \left.
\lefteqn{\phantom{\begin{matrix} b_0\\ \ddots\\ b_0\ \end{matrix}}} \right.\end{tabular}\end{equation}

This gives the following, but with some errors. Could someone tell me how to correctly align the brace next to the matrix and make the code more correct?

Thanks

Answer 1

First, I correct some errors. \lefteqn does not make any sense here. And in a math environment, use array instead of table. I also include a solution based on TiKz. This is an MWE, which you are supposed to include in your question.

\documentclass{article}
\usepackage{amsmath}
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing,calc}
% from https://tex.stackexchange.com/questions/402462/tikz-equivalent-of-pstricks-commands-ncbar-and-rnode
\newcommand{\tikznode}[2]{%
\ifmmode%
  \tikz[remember picture,baseline=(#1.base),inner sep=0pt] \node (#1) {$#2$};
\else
  \tikz[remember picture,baseline=(#1.base),inner sep=0pt] \node (#1) {#2};%
\fi}

\DeclareMathOperator{\co}{co}
\begin{document}
Without TiKz
\begin{equation}\co_\beta(Au_{d+1})=
\begin{bmatrix}\langle u_1,Au_{d+1}\rangle \\ \vdots \\ \langle u_n,Au_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle Au_1,u_{d+1}\rangle \\ \vdots \\ \langle Au_n,u_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle \lambda u_1,u_{d+1}\rangle \\ \vdots \\ \langle \lambda u_d,u_{d+1}\rangle \\ \rule[.5ex]{4.5em}{0.4pt} \\ * \\ \vdots \\ *\end{bmatrix}=
\begin{bmatrix}0 \\ \vdots \\ 0  \\ \rule[.5ex]{1em}{0.4pt} \\ * \\ \vdots \\ *\end{bmatrix}
\begin{array}{l}
\left.\vphantom{\begin{matrix} 0 \\ \vdots\\ 0
\end{matrix}}\right\}d\text{ nullen} \\ \left.
\vphantom{\begin{matrix} b_0\\[0.25cm]  \ddots\\ b_0\ \end{matrix}} \right.\end{array}
\end{equation}

With TiKz
% from https://tex.stackexchange.com/questions/40/how-do-i-label-different-rows-or-columns-of-a-matrix-using-braces/1070#1070
\begin{equation}\co_\beta(Au_{d+1})=
\begin{bmatrix}\langle u_1,Au_{d+1}\rangle \\ \vdots \\ \langle u_n,Au_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle Au_1,u_{d+1}\rangle \\ \vdots \\ \langle Au_n,u_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle \lambda u_1,u_{d+1}\rangle \\ \vdots \\ \langle \lambda u_d,u_{d+1}\rangle \\ \rule[.5ex]{4.5em}{0.4pt} \\ * \\ \vdots \\ *\end{bmatrix}=
\begin{bmatrix}\tikznode{x1}{0} \\ \vdots \\ \tikznode{x2}{0}  \\ \rule[.5ex]{1em}{0.4pt} \\ * \\ \vdots \\ *\end{bmatrix}
\tikz[decoration={brace},baseline=(current bounding box.west)]{%
\draw[decorate,transform canvas={yshift=3em},thick] (x1.north) -- (x2.south) node[right=5pt,midway] {$d$ nullen};     }
\end{equation}
\end{document}

Answer 2

Here is a solution with nicematrix (last version: 5.10 of 2021/02/05).

I have replaced the last {bmatrix} by a {bNiceMatrix}.

Then, I have put a brace with the command \SubMatrix available in the \CodeAfter of the {bNiceMatrix}. The . makes a blank left-delimiter for the submatrix, the {1-1}{3-1} specifies that the submatrix should have (1,1) as top-left and (3,1) as bottom-right corners, and the \{ uses a left-brace as the right-delimiter of the submatrix, producing the brace you want. I push the brace a little bit using the options in [right-xshift=0.5em,name=A] and name the object A so we can grab it later.

Then, I have put the label with Tikz on the right of the Tikz node created by nicematrix for the brace.

\documentclass{article}
\usepackage{nicematrix,tikz}
\DeclareMathOperator{\co}{co}
\begin{document}
[\co_\beta(Au_{d+1})=
\begin{bmatrix}\langle u_1,Au_{d+1}\rangle \ \vdots \ \langle u_n,Au_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle Au_1,u_{d+1}\rangle \ \vdots \ \langle Au_n,u_{d+1}\rangle \end{bmatrix}=
\begin{bmatrix}\langle \lambda u_1,u_{d+1}\rangle \ \vdots \ \langle \lambda u_d,u_{d+1}\rangle \ \rule[.5ex]{4.5em}{0.4pt} \ * \ \vdots \ \end{bmatrix}=
\begin{bNiceMatrix}
0 \ \vdots \ 0  \ \rule[.5ex]{1em}{0.4pt} \  \ \vdots \ *
\CodeAfter
\SubMatrix.{1-1}{3-1}}[right-xshift=0.5em,name=A]
\tikz \node [right] at (A-right.east) {$d$ nullen} ;
\end{bNiceMatrix}]
\end{document}

You need several compilations (because nicematrix uses PGF/Tikz nodes).