If you define
\newcommand{\M}[3]{M(#1 \times #2, #3)}
then the input syntax should be
\M{2}{2}{\mathbb{R}}
and not
\M{2,2,\mathbb{R}}
which indeed reproduces the strange output and, possibly, some error messages.
Example:
\documentclass{article}
\usepackage{amsmath,amssymb}
\newcommand{\RR}{\mathbb{R}} % the reals
\newcommand{\M}[3]{M(#1\times #2,#3)}
\begin{document}
Good syntax \verb|\M{2}{2}{\RR}| yields $A\in\M{2}{2}{\RR}$
Bad syntax \verb|\M{2,2,\RR}| yields $A\in\M{2,2,\RR}$
\end{document}

Can you get the, admittedly easier, syntax to work? Yes.
\documentclass{article}
\usepackage{amsmath,amssymb}
%\usepackage{xparse}% not needed for LaTeX 2020-10-01 or later
\newcommand{\RR}{\mathbb{R}} % the reals
\NewDocumentCommand{\M}{>{\SplitArgument{2}{,}}m}{\MLONG#1}
\NewDocumentCommand{\MLONG}{mmm}{M(#1\times #2,#3)}
\begin{document}
Good new syntax \verb|\M{2,2,\RR}| yields $A\in\M{2,2,\RR}$
\end{document}

Where's the secret? The preprocessor \SplitArgument{2}{,} tells LaTeX to read the argument, which should contain two commas, and pass #1 as
{<a>}{<b>}{<c>}
when the argument is <a>,<b>,<c>. So we can feed the transformed argument to \MLONG that reads three standard arguments.