Theorems are lists. The headlines are set through the optional argument of \item, and in your class actually in a \box. There won't break.
You can try to fake a suitable look by changing the behaviour for an empty optional argument. But imho it would be better to shorten the title.
\documentclass[prodmode,acmtosn]{acmsmall}
\makeatletter
\def\@opargbegintheorem#1#2#3{%
\trivlist
\item[\hskip 12pt
\hskip \labelsep
{\ifitalicenv{\scshape{#1}}\else{\itshape#1}\fi%
\savebox\@tempboxa{\ifitalicenv{\scshape#3}\else{\itshape#3}\fi}%
\ifdim\wd\@tempboxa>\z@%
\ {\rmfamily #2}\unskip\hskip5pt\relax$($\box\@tempboxa$)$.%
\else \ {\rmfamily #2}\unskip\hskip5pt\relax
\fi\unskip\hskip5pt}]
\ifitalicenv\itshape\else\upshape\fi\hskip-\labelsep}
\makeatother
\begin{document}
\begin{theorem}[blub]
content...
\end{theorem}
\begin{theorem}[] {$($\scshape long long long long long long long long long long long long long long long long long long long long long long long long long long title$)$.\unskip\hskip5pt}
content...
\end{theorem}
\end{document}
