I am trying to make a table with different colors alternating per row, and I found that my document was compiling like this:

I've had similar issues with matrices and other large braces, but not others like integrals and sums. I've looked at the solutions on here and here, but I'm not sure that I understand either one, or how it's relevant to my problem.
My questions is whether or not this is an issue with the way I am coloring coloring the rows, and if there was a more conventional way to do so.
My markup is here:
\documentclass{article}
\usepackage[table]{xcolor}
\usepackage{amsmath}
\begin{document}
\begin{table}[h]
\renewcommand{\arraystretch}{1.5}%
\rowcolors{2}{}{gray!10}
\begin{tabular}{p{3cm}|p{5cm}}
\textbf{Problem} & \\
With Color & $n! =
\begin{cases}
1 & \text{if } n = 0 \\
(n-1)! & \text{otherwise}
\end{cases}$ \\
Without Color & $n! =
\begin{cases}
1 & \text{if } n = 0 \\
(n-1)! & \text{otherwise}
\end{cases}$ \\
\end{tabular}
\end{table}
\end{document}


