The problem seems to be very interesting especially when arbitrary number of
levels of brackets is expected. I tried to solve this. Arbitrary number of
levels can be used and the types of brackets for each level can be declared.
The formula enclosed by brackets are typed by \brk{formula} (\brk is
abbreviation for "brackets"). The formula can include another \brk{...}
(nesting at arbitrary number of levels). For example:
\brk{a\brk{b\brk{c\brk{d}e}f}g\brk{h}i\brk{jk\brk{l}pq}r}
this means:
(a(b(c(d)e)f)g(h)i(jk(l)pq)r)
My solution implements two pass algorithm.
The whole parameter of most outer \brk is processed in first pass in order to
decide the number of nested brackets. After first pass is done, each
bracket knows the number of nested levels in its subformula.
It means, for our example:

The second pass of \brk macro prints desired brackets. You have to declare
the shape of these brackets by the \brkDecl macro:
\brkDecl <left0><right0><left1><right1><left2><right2>etc<left-n><right-n><dot><dot>
Examles:
\brkDecl ()[]..
\brk{a\brk{b\brk{c\brk{d}e}f}g\brk{h}i\brk{jk\brk{l}pq}r}

\brkDecl ()[]\{\}\langle\rangle..
$\brk{a\brk{b\brk{c\brk{d}e}f}g\brk{h}i\brk{jk\brk{l}pq}r}$

\brkDecl (){\bigl(}{\bigr)}{\Bigl(}{\Bigr)}{\biggl(}{\biggr)}{\Biggl(}{\Biggr)}..
$\brk{a\brk{b\brk{c\brk{d}e}f}g\brk{h}i\brk{jk\brk{l}pq}r}$

You can use \ensurebalanced macro from here and math-active
character setting in order to user can simply type only rounded brackets:
\def\brkX#1){\ensurebalanced()\brk{#1}}
{\catcode`\(=13 \global\let(=\brkX}
\def\brkActive{\mathcode`\(="8000 }
\brkDecl {\mathopen{\mathchar`(}}{)}{\bigl(}{\bigr)}{\Bigl(}{\Bigr)}{\biggl(}{\biggr)}{\Biggl(}{\Biggr)}..
\brkActive
$(a(b(c(d)e)f)g(h)i(jk(l)pq)r)$

Note, that the <left0> declaration in \brkDecl is somewhat more
complicated here, becuase the character ( is declared as active character in math.
The implementation of the \brk macro follows:
\newcount\brkN \newcount\brkL
\def\brk#1{\def\brkS{0}\global\brkN=0 \let\brk=\brkA
\ifmmode \setbox0=\hbox{$#1$}\else \setbox0=\hbox{#1}\fi
\sxdef{brk.0}{\the\brkL}%
\global\brkN=0 \let\brk=\brkB
\brkE L0#1\brkE R0\let\brk=\brkC
}
\def\brkA#1{{\global\brkL=0 \global\advance\brkN by1
\def\brkS{0}\edef\brkM{\the\brkN}%
#1%
\sxdef{brk.\brkM}{\the\brkL}\global\advance\brkL by1}%
\ifnum\brkS>\brkL \global\brkL=\brkS\relax \else \edef\brkS{\the\brkL}\fi
}
\def\brkB{\global\advance\brkN by1 \expandafter\brkD\expandafter{\the\brkN}}
\let\brkC=\brk
\def\brkD#1#2{\brkE L{#1}#2\brkE R{#1}}
\def\brkE #1#2{\csname brk.#1.%
\ifnum\csname brk.#2\endcsname>\brkF\space \brkF \else \csname brk.#2\endcsname \fi
\endcsname
}
\def\brkDecl{\brkL=0 \brkG}
\def\brkG#1#2{\ifx#1.\advance\brkL by-1 \edef\brkF{\the\brkL}\else
\sdef{brk.L.\the\brkL}{#1}\sdef{brk.R.\the\brkL}{#2}%
\advance\brkL by1 \expandafter\brkG \fi
}
\def\sdef#1{\expandafter\def\csname#1\endcsname}
\def\sxdef#1{\global\expandafter\edef\csname#1\endcsname}
The implementation notes.
In the first pass, the \brk macros inside the parameter are processed like
\brkA. In the second pass, they are processed like \brkB. The normal
meaning of \brk is returned by \brkC.
The first pass is processed inside box0, i. e. the material is fully
processed but not printed. The left brackets are numbered from zero from left to
right (by \brkN counter) and the number of levels is calculated by \brkL
counter. The macro \csname brk.\the\brkN\endcsname includes the number of
sub-brackets for each left bracket after first pass is done.
The macros \csname brk.L.\the\brkL\endcsname include the type of left
bracket for \bkrL level. The right bracket is stored in
\csname brk.R.\the\brkL\endcsname. These macros are initialized by \brkDecl.
\br{…\br{…\br{…}…}…}intelligent parenthesis. – Manuel Oct 08 '14 at 15:19commathpackage could be helpful? – barbara beeton Oct 08 '14 at 16:16\brcommand. But I don't remember… or may be I'm wrong. – Manuel Oct 08 '14 at 16:40