Instead of using \left( and \right) to auto-size the outermost set of parentheses, you could use \biggl( and \biggr):

(Aside: See the postings Why the control sequences \bigl, \biggl, \bigr or \biggr, as I can always use \big or \bigg? and Difference between \big[ and \bigl[ for a discussion of the differences between \Bigg( and \Biggl(.)
I would recommend, actually, using \Biggl[ (while adding \, immediately afterwards) and \Biggr] to provide a bit more -- but not too much... -- visual prominence as well as variety to the outer fences:

The default LaTeX/amsmath style for underbraces frankly looks quite boring to me. If you have access to the mtpro2 package, you could use its macro \undercbrace to generate a curly underbrace. Cautions: (i) The font used by this package is Times Roman, which may or may not be to your liking. (ii) The mtpro2 package isn't free of charge; however, its "lite" subset, which is all that's needed to use the \undercbrace macro, can be obtained without (financial) charge.

\documentclass[preview,border=12pt,varwidth]{standalone}
\usepackage{amsmath}
\usepackage[lite]{mtpro2}
\begin{document}
\[
12\Biggl[\,\undercbrace{y^2 +2y\left(-\frac{2}{3}\right) +\left(-\frac{2}{3}\right)^2}_{\left[y+\left(-\frac{2}{3}\right)\right]^2}{} -\left(-\frac{2}{3}\right)^2 +\frac{4}{5}\Biggr] = 0
\]
\end{document}
Addendum: The immediately preceding example is intended mainly to demonstrate the shape of the "curly" underbrace. Thus, the only changes, relative to the code used in the second example, were in loading the mtpro2 package and using \undercbrace instead of \underbrace.
If you're serious about using the mtpro2 package, I would recommend making a few additional changes to the code. Chief among them are these: (i) load the mleftright package and execute \mleftright in the preamble to eliminate the extra horizontal whitespace that's otherwise inserted by \left and \right; (ii) don't insert a thinspace after \Biggl[; and (iii) insert a negative thinspace (\!) after the undercbrace material.

\documentclass[preview,border=12pt,varwidth]{standalone}
\usepackage{amsmath}
\usepackage{mleftright}\mleftright
\usepackage[lite]{mtpro2}
\begin{document}
\[
12\Biggl[ \undercbrace{y^2 +2y\left(-\frac{2}{3}\right)
+\left(-\frac{2}{3}\right)^2}_{
\left[y+\left(-\frac{2}{3}\right)\right]^2} \! {}
-\left(-\frac{2}{3}\right)^2 +\frac{4}{5}\Biggr] = 0
\]
\end{document}
\big-like delimiters for the outer (all) brackets instead. – Werner Nov 10 '13 at 07:32\smashand\vphantom): Changing the size of the braces from the cases environment, ignoring vertical space due to overbraces in cases environment and Underbraces in Matrix Divided in Blocks (ignore the TikZ solutions). Though, in this case using\big…is of course easier. – Qrrbrbirlbel Nov 10 '13 at 07:40