For comparison I am posting a ConTeXt solution.
Such a macro already exists, albeit for quotes. One of the good things about ConTeXt is that a feature is never defined in a one-off basis. For example, instead of defining a macro for quotations that changes the quote symbol depending on the level of nesting, ConTeXt defines a generic delimitedtext mechanism and quotation is a special case of delimited text. The desired parenthesis macro is also a delimited text where the left and right symbol change depending on the level of nesting. So, use delimited text to define this macro as follows:
\definedelimitedtext[parenthesis] [location=text]
\setupdelimitedtext [parenthesis:1] [left={(}, right={)}]
\setupdelimitedtext [parenthesis:2] [left={[}, right={]}]
\setupdelimitedtext [parenthesis:3] [left={\{},right={\}}]
This can be used as
\starttext
\parenthesis{My outer layer \parenthesis{my inner layer \parenthesis{my innermost layer}}}
\stoptext
which gives
