This is an improvement over John Kormylo's answer -- it works, but has a small quirk: it hard codes the math style.
What does this mean? Consider for example, with that definition:
\[ \mydelim{\frac{x+a}{x+b}} \]
$ \mydelim{\frac{x+a}{x+b}} + \frac{x+a}{x+b} $
the second formula is too large, compared to the one outside the custom delimiter.

One way to fix it is to manually add the \textstyle command:
$ \mydelim{\textstyle \frac{x+a}{x+b}} \frac{x+a}{x+b} $
(in this case you don't need the \displaystyle at all)
Another way is to modify the command definition as follows:
\documentclass{article}
\usepackage{tikz}
\usepackage{amsmath} % ← note the added package
\newcommand{\leftdelim}[2]% #1 = height, #2 = depth
{\tikz[baseline]{\drawcolor=red,arrows={-latex}--(0,#1);}}
\newcommand{\rightdelim}[2]% #1 = height, #2 = depth
{\tikz[baseline]{\drawcolor=red,arrows={latex-}--(0,#1);}}
\newlength{\MyHeight}
\newlength{\MyDepth}
\newsavebox{\MyBox}
\newcommand{\mydelim}[1]% #1 = text to be enclosed
{\text{\savebox{\MyBox}{$#1$}% get size of box
\settoheight{\MyHeight}{\usebox{\MyBox}}%
\settodepth{\MyDepth}{\usebox{\MyBox}}%
\leftdelim{\MyHeight}{\MyDepth}%
,\usebox{\MyBox},%
\rightdelim{\MyHeight}{\MyDepth}}}
\begin{document}
[ \mydelim{\frac{x+a}{x+b}} ]
$ \mydelim{\frac{x+a}{x+b}} + \frac{x+a}{x+b} $
\end{document}
which gives the desired result.

An explanation of the \text command may be found at rotating - Rotated math with correct font size - TeX - LaTeX Stack Exchange.
\leftand\rightbut I'm not sure if it is a good idea... this and this answer might help you. – Pouya Jun 09 '14 at 13:17\leftor\rightonly if scalable versions of the symbol are available as part of the font. – egreg Jun 09 '14 at 13:20