
You would need to play with the values a bit to get it looking nicer, but basically you need to add a \penalty (\linebreak[0] in LaTeX) to add additional break points and then add stretchable glue elsewhere so that the formula can stretch to get a breakpoint at the end of the line.
\documentclass{article}
\DeclareMathSymbol{\oldcomma}{\mathpunct}{letters}{"3B}
\let\olddots\dots
\def\dots{\hspace{0pt plus .7em}\olddots\hspace{0pt plus 7em}}
\mathcode`\,="8000
\begingroup
\makeatletter
\catcode`,\active
\gdef,{%
\oldcomma
\@ifnextchar\dots{%
\penalty0 %
\hspace{0pt plus .2em}%
}{}}
\endgroup
\begin{document}
\noindent X\dotfill X
\noindent aaa \( (B_{r+1,1}, C_{r+1,1}), \dots , (B_{r+1,m_{r+s}}, C_{r+1,m_{r+s}}), \dots , (B_{r+s,1}, C_{r+s,1}), \dots , (B_{r+s,m_{r+s}}, C_{r+s,m_{r+s}}) \) bbb
\end{document}
{ }groups so if you have lots of grouped terms, aome change to the original markup may be necessary, otherwise it should already break at infix operators without needing to be configured. – David Carlisle Oct 27 '12 at 21:06