This appears to work:
Update: see further down a different implementation which does the \left and \right things and also is compatible with a \middle
in-between.
Second Update: mathcodes do not have all the limitations of
catcodes. Based on mathcodes I propose one more approach which
is more powerful than the previous one: it allows (seemingly) very
complicated nesting. I dream there were
textcodes... the point is that mathcodes do not freeze. This is
extremely useful.
Edit: correct restores of the mathcodes after
some additional work. I promise not to edit again...
\documentclass{article}
\makeatletter
\def\p{\afterassignment\p@aux\let\next=}
\def\p@aux{\ifx\next\bgroup\lbrace\bgroup\aftergroup\rbrace\else
\expandafter\next\fi}
\makeatother
\begin{document}
$\p(1)$
$\p[1]$
$\p{1}$
$\p(1|2)$
\end{document}

With automatic \left and \right. Should I also do the middle thing?
Update: added missing ̀% at end lines and thinking about the e-TeX \middle thing which is causing me some problems...
(see further down for the final method: \middle was quite a
challenge)
\documentclass{article}
\usepackage{amsmath}
\makeatletter
\def\p{\afterassignment\p@aux\let\next=}
\def\p@aux{\ifx\next\bgroup
\left\lbrace\bgroup\aftergroup\right\aftergroup\rbrace
\else
\ifx\next(%
\left(\bgroup\catcode`\)=\tw@
\aftergroup\right\aftergroup)%
\else
\ifx\next[%
\left[\bgroup\catcode`\]=\tw@
\aftergroup\right\aftergroup]%
\else\next
\fi\fi
\fi}
\makeatother
\begin{document}\thispagestyle{empty}
$\p(1) \p(\dfrac{1}{2})$
$\p[1] \p[\dfrac{1}{2}]$
$\p{1} \p{\dfrac{1}{2}}$
$\p(1|2) \p(\dfrac{1}{2}|3)$
\end{document}

Giving justice to the
eTeX \middle required a complete change of method. Note that the following works with an arbitrary delimiter after
\middle
\documentclass{article}
\usepackage{amsmath} % for \dfrac
\makeatletter
\def\p{\afterassignment\p@aux\let\next=}
\newtoks\p@toks
\def\@ybrace{\expandafter\left\expandafter\lbrace\the\p@toks\right\rbrace}
%% \def\@yparen{\expandafter\left\expandafter(\the\p@toks\right)\endgroup}
\def\@yparen{\expandafter\endgroup\expandafter\left\expandafter(\the\p@toks\right)}
%% \def\@ybrack{\expandafter\left\expandafter[\the\p@toks\right]\endgroup}
\def\@ybrack{\expandafter\endgroup\expandafter\left\expandafter[\the\p@toks\right]}
\def\p@aux{\ifx\next\bgroup
\def\p@tmp{\p@toks=\bgroup}%
\afterassignment\@ybrace
\else
\ifx\next(%
\begingroup\catcode`\)=\tw@
\def\p@tmp{\p@toks=\bgroup}%
\afterassignment\@yparen
\else
\ifx\next[%
\begingroup\catcode`\]=\tw@
\def\p@tmp{\p@toks=\bgroup}%
\afterassignment\@ybrack
\else
\let\p@tmp\next
\fi\fi\fi
\p@tmp}
\makeatother
\begin{document}\thispagestyle{empty}
$\p(1) \p(\dfrac{1}{2}) \p(\dfrac{1}{2}\middle|3)$
$\p[1] \p[\dfrac{1}{2}] \p[\dfrac{1}{2}\middle|3]$
$\p{1} \p{1\middle|2} \p{\dfrac{1}{2}\middle|3}$
\end{document}

$\p{1} \p{1\middle>2} \p{\dfrac{1}{2}\middle<3}$

And now the method based on mathcodes. Perhaps I will have to think again on the case of the braces, which are treated here as in the previous method. For them only do I use a token list. The token list is because of \middle which makes things difficult when attempting an \aftergroup technique.
It is also because of \middle that I have to explicitely restore the mathcodes of ) and ] rather than trust it to the group which originates in the use of \left and \right. The problem is that \middle closes this group and then opens a second one.
Edit: there was a problem with $ \p( ( A ) )$ and I have edited the macros so that the mathcodes are correctly reset. I could not use the groups created by \left and \right 'cause the \middle.
\documentclass{article}
\makeatletter
\def\p{\afterassignment\p@aux\let\next=}
\newtoks\p@toks
\def\@ybrace{\expandafter\left\expandafter\lbrace\the\p@toks\right\rbrace}
% To get correct nesting I added the \restore@paren and \restore@brack macros
\def\p@aux{\ifx\next\bgroup
\def\p@tmp{\p@toks=\bgroup}%
\afterassignment\@ybrace
\else
\ifx\next(%
\ifnum\mathcode`)="8000
\let\restore@paren\relax
\def\p@tmp{\left(}%
\else
\edef\restore@paren{\mathcode`)=\the\mathcode`)\relax}%
\begingroup
\lccode`\~=`)
\lowercase{%
\endgroup
\def~{\right)\restore@paren}}%
\def\p@tmp{\mathcode`)="8000 \left(}%
\fi
\else
\ifx\next[%
\ifnum\mathcode`]="8000
\let\restore@brack\relax
\def\p@tmp{\left[}%
\else
\edef\restore@brack{\mathcode`]=\the\mathcode`]\relax}%
\begingroup
\lccode`\~=`]
\lowercase{%
\endgroup
\def~{\right]\restore@brack}}%
\def\p@tmp{\mathcode`]="8000 \left[}%
\fi
\else
\let\p@tmp\next
\fi\fi\fi
\p@tmp}
\makeatother
\begin{document}\thispagestyle{empty}
\delimitershortfall=-1pt
\[\p(\frac{1}{2}\p{\p{3^T:4}}\p[\p(A_B^C)]\p(\big]\middle>\big[)\middle\Vert\p[\frac{1}{2}])\]
\[\p{\p(\p[A_B^C\middle< \p{X^Y}])\middle\vert\p[\p(A_B^C\middle> \p(X^Y))]} \]
\end{document}

\[\the\mathcode`) \ \the\mathcode`] \p( \p{ \p[ \p( \p{ \p[ A ]})]})
\the\mathcode`)\ \the\mathcode`] \]

xparsepackage – daleif Jan 30 '13 at 14:28\p[f(2)]– Jan 30 '13 at 14:41\p(f(2))won't work with any "reasonable" definition. – egreg Jan 30 '13 at 14:48\p(f(2))would be\left( f(2) \right). – Jan 30 '13 at 16:18{}groups form part of TeX's parsing rules (and are matched according to nesting depth()are just two unrelated characters TeX has no inbuilt knowledge that one is paired with the other and that is useful for notations such as)a,b(– David Carlisle Jan 30 '13 at 17:04