Lstlisting/Verbatim inside tabularx environment?

Question

I noticed that a \lstlisting inside a tabularx doesn't work:

\documentclass{article}
\usepackage{listings}
\usepackage{tabularx}
\let\code\lstinline
\begin{document}
% It does work
\begin{tabular}{|*3{>{}l|}} \hline
\code!\begin{center}! & \code!\centering! & \code!\begin{raggedleft}! \ \hline
\end{tabular}
% It doesn't work
\begin{tabularx}{\columnwidth}{|*3{>{}X|}} \hline
\code!\begin{center}! & \code!\centering! & \code!\begin{raggedleft}! \ \hline
\end{tabularx}
\end{document}

I get this error:

! Improper alphabetic constant.
<to be read again> 
                   \begin 
l.16 \end{tabularx}

What's wrong and how can I fix it?