Problems with special arrow shaft in xypic

Question

I'm trying to draw a diagram my teacher drew on the blackboard to summarize divisibility relationships and I just can't get it right. Here is my best attempt:

\documentclass[a4paper]{report}
\usepackage[all]{xy}
\usepackage{amsmath,amsfonts,amssymb,graphicx,color}
\begin{document}
\begin{gather}
|S^G|=|G:\operatorname{Stab}_G(S)|=|G:N_G(S)|. \
\xymatrix{
G \ar@{{}\composite{\hbox{\scalebox{2}{$\underset{\Big{}{\scriptscriptstyle\text{\color{white}A}}$}}}}@<-3ex>[dd]_{p'\quad}\ar@{{}\composite{\hbox{\scalebox{1}[2]{}}}}{}}@<1.5ex>[d]^{p'}\ar@{-}[d] \
N_G(S) \ar@{-}[d] \
S \ar@{-}[d] \
1
}
\end{gather}
\end{document}

Result:

OK, there is an extra line in the code, and I should use a single equation for an isolated matrix. Anyways, the problem is obvious: those long braces just won't properly extend. I need the big brace to extend from close to the G to close to the S. I tried lots of variants, but the brace invariably starts either far from the G or above (!!) the G, and always ends close to the N_G(S). The little brace is also somewhat problematic, but I guess I can leave it as is, and besides when (and if) I fix the big one the little one will probably be easy to fix. So how do I fix that big brace?

Answer 1

If you find time and appetite for changing to tikz-cd, this would look like the following (most stuff taken from @sergej)

% arara: pdflatex

\documentclass{article}
\usepackage{tikz-cd}
\usetikzlibrary{decorations.pathreplacing, positioning}
\usepackage{mathtools}

\begin{document}
\begin{gather*}
|S^G|=|G:\operatorname{Stab}_G(S)|=|G:N_G(S)|\\
\begin{tikzcd}[every arrow/.append style=dash]
G 
\ar[thick, start anchor=center, end anchor=center, xshift=-1.7em, decorate,decoration={brace,amplitude=8pt,mirror}]{dd}[left=8pt]{p}
\ar[thick, start anchor=center, end anchor=center, xshift=1.7em, decorate,decoration={brace,amplitude=4pt}]{d}[right= 4pt]{p}
\ar{d} \\
N_G(S) \ar{d} \\
S \ar{d} \\
1
\end{tikzcd}
\end{gather*}
\end{document}

Answer 2

Is this enough to you?

\[
\xymatrix{
  G      \ar@{-}[d]    \ar@{-} `l[d] `[dd]_{p'} [dd]  \\
  N_G(S) \ar@{-}[d] \\
  S      \ar@{-}[d] \\
  1
}
\qquad
\xymatrix@C=1ex{% 1ex = space between collumns
 & G      \ar@{-}[d]   \ar@{-} `l/0pt[ld] `[dd]_{p'} [dd] \ar@{-} `r[dr] `[d]^{p'} [d] & \\
 & N_G(S) \ar@{-}[d] & \\
 & S      \ar@{-}[d] \\
 & 1
}
\]

Answer 3

Here is a tikz solution. The baces are drawn using the decorations library.

\documentclass[preview]{standalone}
\usepackage{amsmath}

\usepackage{tikz}
\usetikzlibrary{positioning}
\usetikzlibrary{decorations.pathreplacing}

\begin{document}

\begin{gather*}
|S^G|=|G:\operatorname{Stab}_G(S)|=|G:N_G(S)|.
\end{gather*}

\begin{center}
\begin{tikzpicture}[
    % brace style
    bl/.style={decorate,decoration={brace,amplitude=10pt,raise=4pt,mirror},very thick},
    br/.style={decorate,decoration={brace,amplitude=7pt,raise=4pt}}]

    % spacing between the nodes and the braces
    \def\s{13pt}
    \def\t{19pt}

    % nodes
    \node (n1) at (0,0)      {$G$};
    \node [below=of n1] (n2) {$N_G(S)$};
    \node [below=of n2] (n3) {$S$};
    \node [below=of n3] (n4) {$1$};

    % arrows
    \path[->]
        (n1) edge node {} (n2)
        (n2) edge node {} (n3)
        (n3) edge node {} (n4);

    % left brace
    \draw [bl] ([xshift=-\s]n1.center) -- ([xshift=-\s]n3.center)
        node [black,midway,xshift=-\t] {\footnotesize$p'$};

    % right brace
    \draw [br] ([xshift=\s]n1.center) -- ([xshift=\s]n2.center)
        node [black,midway,xshift=\t] {\footnotesize$p'$};

\end{tikzpicture}

\end{center}

\end{document}