Nested Mathematical Functions

Question

This question has bugged me for a while. I want to write a nested function, f(g(x)). If I naively enter it:

\begin{align}
    f(g(x))=y
\end{align}

The problem is that each and every parenthesis is the same size as all of the others. Shouldn't the outer parentheses be larger? But if I do this:

\begin{align}
    f\left(g(x)\right)=y
\end{align}

Now it looks even less like the functions are nested. What's the proper way to do this?

Answer 1

There are enlarged grouping symbols available as \big, \Big, \bigg, and \Bigg. A following l or r indicates left or right (which helps allocate the proper surround spacing), and then follow it up with the grouping symbol itself.

Obviously the \displaystyle example [p(q(f(g(h(x)))))] in my MWE is to make an exaggerated point. I am not actually suggesting it for the case given (see answer SUPPLEMENT for alternative).

\documentclass{article}
\begin{document}
$f\bigl(g(x)\bigr)$ 
\[
p\Biggl(q\biggl(f\Bigl(g\bigl(h(x)\bigr)\Bigr)\biggr)\Biggr)
\]
\end{document}

---

In a comment, Gonzalo suggests using the mathtools package to declare delimiter pairs, which in his MWE, he calls \Comp. Then, using \Comp[<size>]{...} will place the specified size delimiters around the embraced quantity.

\documentclass{article} 
\usepackage{mathtools} 
\DeclarePairedDelimiter\Comp{(}{)} 
\begin{document} 
$f\Comp[\big]{g(x)}$ 
\[ 
p\Comp[\Bigg]{ q\Comp[\bigg]{ f\Comp[\Big]{g\Comp[\big]{h(x} } } } 
\] 
\end{document}

---

SUPPLEMENT:

If one truly needed a more finely graded scaling of adjacent delimiters, my scalerel package could be used in that regard. Here, I add 0.4pt height above and below each "unit" to govern the size of the next delimiter.

\documentclass{article} 
\usepackage{scalerel,stackengine}
\stackMath
\ignoremathstyle
\newcommand\Comp[1]{\scaleleftright{(}{\addstackgap[.4pt]{#1}}{)}}
\begin{document} 
$f\Comp{g(x)}$ 
\[ 
p\Comp{ q\Comp{ f\Comp{g\Comp{h(x)} } } } 
\] 
\end{document}