If you really insist on \Large (but you shouldn't), define your own environment. And it's better to use the “modern” syntax for arrows.
\documentclass{article}
\usepackage{amsmath}
\usepackage{tikz-cd}
\newsavebox{\tikzcdbox}
\newenvironment{Largetikzcd}
{\begin{lrbox}{\tikzcdbox}\Large\begin{tikzcd}}
{\end{tikzcd}\end{lrbox}\usebox{\tikzcdbox}}
\begin{document}
[
\begin{Largetikzcd}
X \arrow[r,"i"] \arrow[dr,swap,"j"] & M \arrow[d,"j'"] \
& N
\end{Largetikzcd}
\qquad
\begin{Largetikzcd}
X \arrow[r,"j"] \arrow[dr,swap,"i"] & N \arrow[d,"i'"] \
& M
\end{Largetikzcd}
]
[
\begin{tikzcd}
X \arrow[r,"i"] \arrow[dr,swap,"j"] & M \arrow[d,"j'"] \
& N
\end{tikzcd}
\qquad
\begin{tikzcd}
X \arrow[r,"j"] \arrow[dr,swap,"i"] & N \arrow[d,"i'"] \
& M
\end{tikzcd}
]
\end{document}

Never use $$ in a LaTeX document environment.
\[...\]instead of$$...$$. Details here. – Sandy G May 19 '23 at 12:45