How to create a simple vertical vector with several vertical curly braces outside the square bracket?

Question

I could not code a vertical vector with several vertical curly braces outside the square bracket of the vector. The best thing I have obtained so far is by using rcases from mathtools package that gives me a vertical vector with the corresponding vertical curly braces inside the square bracket of the vector. My code is as the following:

\documentclass{article}
\usepackage{amsmath,mathtools}
\begin{document}
\begin{equation}
    \mathcal{O}(t) = \left[ \begin{array}{c}
    \begin{rcases} c_1^\top \ \vdots \ c_p^\top \end{rcases}C\ \begin{rcases} c_1^\top A \ \vdots \ c_p^\top A \end{rcases} CA \ \vdots \ \begin{rcases}c_1^\top A^{t-1} \ \vdots \ c_p^\top A^{t-1}\end{rcases} CA^{t-1}
    \end{array}\right].
\end{equation}
\end{document}

that gives me this output:

which is still not accepted. Any hints that can help me to bring out the vertical curly braces would be greatly appreciated.

Answer 1

You can use \vphantom to replicate the vertical space of an object and use that together with a \left....\right\} construction to simulate an \rcases-like environment:

\documentclass{article}
\usepackage{mathtools}
\begin{document}
[
  \mathcal{O}(t) = \left[ \begin{array}{ c }
    \begin{array}{ @{} c @{} }
      c_1^\top \ \vdots \ c_p^\top
    \end{array} \[1.5\normalbaselineskip]
    \begin{array}{ @{} c @{} }
      c_1^\top A \ \vdots \ c_p^\top A
    \end{array} \
    \vdots \
    \begin{array}{ @{} c @{} }
      c_1^\top A^{t - 1} \ \vdots \ c_p^\top A^{t - 1}
    \end{array}
  \end{array}\right]
  \begin{array}{ @{} l @{} }
    \left.\hskip-\nulldelimiterspace\vphantom{\begin{array}{ @{} c @{} }
      c_1^\top \ \vdots \ c_p^\top
    \end{array}}\right} C \[1.5\normalbaselineskip]
    \left.\hskip-\nulldelimiterspace\vphantom{\begin{array}{ @{} c @{} }
      c_1^\top A \ \vdots \ c_p^\top A
    \end{array}}\right} CA \
    \vphantom{\vdots} \
    \left.\hskip-\nulldelimiterspace\vphantom{\begin{array}{ @{} c @{} }
      c_1^\top A^{t - 1} \ \vdots \ c_p^\top A^{t - 1}
    \end{array}}\right} CA^{t - 1}
  \end{array}
]
\end{document}

A slight horizontal correction (\hskip-\nulldelimiterspace) reduces the space between the array and the row indicators.

Answer 2

You can use nicematrix (with a little visual formatting).

\documentclass{article}
\usepackage{amsmath}
\usepackage{nicematrix}
\begin{document}
[
\mathcal{O}(t)=;;
\begin{NiceMatrix}
c_1^T & \Block[l]{3-1}{\ C} \ \vdots \ c_p^T \[1ex]
c_1^TA & \Block[l]{3-1}{\ CA} \ \vdots \ c_p^TA \
\vdots \
c_1^TA^{t-1} & \Block[l]{3-1}{\ CA^{t-1}} \ \vdots \ c_p^TA^{t-1}
\CodeAfter
  \SubMatrix[{1-1}{10-1}]
  \SubMatrix.{1-1}{3-1}}[xshift=3pt]
  \SubMatrix.{4-1}{6-1}}[xshift=3pt]
  \SubMatrix.{8-1}{10-1}}[xshift=3pt]
\end{NiceMatrix}
]
\end{document}