As egreg noted in the comments:
The \addvspace macro must be issued in vertical mode!
There is no conflict between \NewDocumentCommand and enumerate etc or enumitem. The solution to the error is that \addvspace must follow an explicit \par. The error would occur for \newenvironment too (see the probtrad environment).
I suggest to use \refstepcounter{probctr} instead of \addtocounter. The former allows to use \label and refer to it.
Please note, that my initial version worked (using \vskip 0.15in instead of \par\addvspace{0.15in}) just because \vskip implies \par). It was not really correct then. Joe Cocker sang: With a little help from my friends I took the comment (by permission from egreg) to apply it here.
\documentclass{article}
\usepackage{enumerate}
\usepackage{etoolbox}
%\usepackage[shortlabels]{enumitem} % as an alternative to enumerate
\usepackage{xparse}
\newcounter{probctr}
\NewDocumentEnvironment{prob}{o}
{%
\par
\addvspace{.15in}%
\refstepcounter{probctr}%
\noindent\textbf{Problem \theprobctr}%
\IfNoValueF{#1}{ (#1 points)}%
\par\nopagebreak\smallskip\noindent\ignorespaces%
}{%
\par\addvspace{.15in}% egreg's suggestion
}
\newenvironment{probtrad}[1][]{%
\par
\addvspace{.15in}%
\refstepcounter{probctr}%
\noindent\textbf{Problem \theprobctr}%
\ifblank{#1}{}{ (#1 points)}%
\par\nopagebreak\smallskip\noindent\ignorespaces%
}{%
\par\addvspace{.15in}% egreg's suggestion
}
\begin{document}
\begin{prob}[15]
\begin{enumerate}[(a)]
\item Item 1
\item Item 2
\end{enumerate}
Consider this ...
\end{prob}
\begin{probtrad}[20]
\begin{enumerate}[(a)]
\item Item 3
\item Item 4
\end{enumerate}
Now consider this ...
\end{probtrad}
\end{document}

\par\addvspace{.15in}in the “end” part. – egreg Jun 05 '15 at 20:05\refstepcounter{prbcounter}instead of\addtocounter. You can use labels then – Jun 05 '15 at 20:08\end{prob}. The error is very misleading though. – passerby51 Jun 05 '15 at 20:30\par– Jun 05 '15 at 20:32{1}? Sorry, I am confused! – Jun 05 '15 at 20:37\addvspaceconstantly appears at the end of lists, the LaTeX team has decided to issue\@noitemerrwhen\addvspaceis in unproper place. – egreg Jun 05 '15 at 20:47refstepcounterincrements by 1 by default (it seems). Otherwise, what is the advantage of using it over\addtocounter? – passerby51 Jun 05 '15 at 21:46\addvspaceused in horizontal mode" would have been much more suitable. I have no idea how feasible that would have been. – passerby51 Jun 05 '15 at 21:49\refstepcounterallows to use a label for your problem environment, say\begin{prob} \label{foo}.... and later on, you could sayIn Problem \ref{foo}... for example – Jun 06 '15 at 06:39