No Warranty nor Liability for any purpose.
\documentclass{article}
% \usepackage{amsmath} % for testing
\newcommand*\autoop{\left(}
\newcommand*\autocp{\right)}
\newcommand*\autoob{\left[}
\newcommand*\autocb{\right]}
\AtBeginDocument {%
\mathcode`( 32768
\mathcode`) 32768
\mathcode`[ 32768
\mathcode`] 32768
\begingroup
\lccode`\~`(
\lowercase{%
\endgroup
\let~\autoop
}\begingroup
\lccode`\~`)
\lowercase{%
\endgroup
\let~\autocp
}\begingroup
\lccode`\~`[
\lowercase{%
\endgroup
\let~\autoob
}\begingroup
\lccode`\~`]
\lowercase{%
\endgroup
\let~\autocb
}}
\delimiterfactor 1001
\makeatletter
% for amsmath "compatibility" (not sophisticated)
% \usepackage{amsmath}
\AtBeginDocument {%
\def\resetMathstrut@{%
\setbox\z@\hbox{\the\textfont\symoperators\char40}%
\ht\Mathstrutbox@\ht\z@ \dp\Mathstrutbox@\dp\z@}%
}%
\makeatother
\begin{document}
\[ [X+(u*[a * (b+c)])]*Y \]
\end{document}
produces (external frame added):

updated (Nov. 2018) for curly braces
as requested per comment:
\documentclass{article}
\newcommand*\autoop{\left(}
\newcommand*\autocp{\right)}
\newcommand*\autoob{\left[}
\newcommand*\autocb{\right]}
\DeclareRobustCommand*\{{\ifmmode \left\lbrace \else \textbraceleft \fi }
\DeclareRobustCommand*\}{\ifmmode \right\rbrace \else \textbraceright \fi }
\AtBeginDocument {%
\mathcode`( 32768
\mathcode`) 32768
\mathcode`[ 32768
\mathcode`] 32768
\begingroup
\lccode`\~`(
\lowercase{%
\endgroup
\let~\autoop
}\begingroup
\lccode`\~`)
\lowercase{%
\endgroup
\let~\autocp
}\begingroup
\lccode`\~`[
\lowercase{%
\endgroup
\let~\autoob
}\begingroup
\lccode`\~`]
\lowercase{%
\endgroup
\let~\autocb
}}
\delimiterfactor 1001
\makeatletter
% for amsmath "compatibility" (not sophisticated)
% \usepackage{amsmath}
\AtBeginDocument {%
\def\resetMathstrut@{%
\setbox\z@\hbox{\the\textfont\symoperators\char40}%
\ht\Mathstrutbox@\ht\z@ \dp\Mathstrutbox@\dp\z@}%
}%
\makeatother
\begin{document}
\[ \{[X+(\{u*[a * (\{b+c\})]\})]\}*Y \]
\end{document}

second request for \| but I think it would be better to use mathtools declarations and input syntax for this.
Here, no nesting of \| and they must be used in pairs.
\documentclass{article}
\makeatletter
% for amsmath "compatibility" (not sophisticated)
\usepackage{amsmath}
\AtBeginDocument {%
\def\resetMathstrut@{%
\setbox\z@\hbox{\the\textfont\symoperators\char40}%
\ht\Mathstrutbox@\ht\z@ \dp\Mathstrutbox@\dp\z@}%
}%
\makeatother
\newcommand*\autoop{\left(}
\newcommand*\autocp{\right)}
\newcommand*\autoob{\left[}
\newcommand*\autocb{\right]}
\DeclareRobustCommand*\{{\ifmmode \left\lbrace \else \textbraceleft \fi }
\DeclareRobustCommand*\}{\ifmmode \right\rbrace \else \textbraceright \fi }
\AtBeginDocument {%
\let\originalbardelimiter\|
\def\myleftbar{\global\let\|\myrightbar\left\originalbardelimiter}%
\def\myrightbar{\global\let\|\myleftbar\right\originalbardelimiter}%
\let\|\myleftbar
\mathcode`( 32768
\mathcode`) 32768
\mathcode`[ 32768
\mathcode`] 32768
\begingroup
\lccode`\~`(
\lowercase{%
\endgroup
\let~\autoop
}\begingroup
\lccode`\~`)
\lowercase{%
\endgroup
\let~\autocp
}\begingroup
\lccode`\~`[
\lowercase{%
\endgroup
\let~\autoob
}\begingroup
\lccode`\~`]
\lowercase{%
\endgroup
\let~\autocb
}}
\delimiterfactor 1001
\begin{document}
\[ \{[X+(\{\|u\|*\| [ a * (\{b+c\})] \|\} )] \}*\|Y\| \]
\end{document}

Finally, (but this is becoming hacky), a trick to allow nesting \| one-level deep. Might be enough for use case in Banach space theory (I hear this is some 20th century old-fashioned math, that long time ago, so that might be enough for elementary human thinking pre-dating AI)(*)
(*) should I explain I don't believe anything about what the highly paid software industry is throwing at us about what AI will be soon?
\documentclass{article}
\makeatletter
% for amsmath "compatibility" (not sophisticated)
\usepackage{amsmath}
\AtBeginDocument {%
\def\resetMathstrut@{%
\setbox\z@\hbox{\the\textfont\symoperators\char40}%
\ht\Mathstrutbox@\ht\z@ \dp\Mathstrutbox@\dp\z@}%
}%
\makeatother
\newcommand*\autoop{\left(}
\newcommand*\autocp{\right)}
\newcommand*\autoob{\left[}
\newcommand*\autocb{\right]}
\DeclareRobustCommand*\{{\ifmmode \left\lbrace \else \textbraceleft \fi }
\DeclareRobustCommand*\}{\ifmmode \right\rbrace \else \textbraceright \fi }
\newif\ifinsidebracedgroup
\AtBeginDocument {%
\let\originalbardelimiter\|
\def\myleftbar{\ifinsidebracedgroup
\right\originalbardelimiter
\global\insidebracedgroupfalse
\else
\left\originalbardelimiter
\fi
\global\let\|\myrightbar}%
\def\myrightbar{\ifnum\currentgrouptype=9
\left\originalbardelimiter
\global\insidebracedgrouptrue
\else
\right\originalbardelimiter
\fi
\global\let\|\myleftbar}%
\let\|\myleftbar
\mathcode`( 32768
\mathcode`) 32768
\mathcode`[ 32768
\mathcode`] 32768
\begingroup
\lccode`\~`(
\lowercase{%
\endgroup
\let~\autoop
}\begingroup
\lccode`\~`)
\lowercase{%
\endgroup
\let~\autocp
}\begingroup
\lccode`\~`[
\lowercase{%
\endgroup
\let~\autoob
}\begingroup
\lccode`\~`]
\lowercase{%
\endgroup
\let~\autocb
}}
\delimiterfactor 1001
\begin{document}
\[ \{[X+(\{\|u\|*\| [ a * (\{{\|b+c\|}\})] \|\} )] \}*\|Y\| \]
\[ \| (X - {\|X\|} e)\|^2 + \| (X + {\|X\|} e)\|^2 = 4\|(X)\|^2 \]
\end{document}

\bigl( \bigr)? – Sigur Jan 26 '16 at 19:12\left(...\right)but usually it is best to choose the size as in Sigur's comment, without the extra left-right markup you can not reliably match brackets, consider a half open interval(1,2]and even with it the automatically determined size is often not ideal. Any package that tries to do this will have to parse the entire expression to match brackets so liable to be incompatible with other code. – David Carlisle Jan 26 '16 at 20:07$(a+(b+c))$the outer parentheses should be bigger. In general this is not true and normal size is good; for more complex formulas or special emphasis one can choose to have outer parentheses bigger, but it's a case based decision. – egreg Jan 26 '16 at 23:43