I don't see how the proposed syntax would be clearer or easier than
\petito_{x\to\infty}(\log x)
or, if you want to specify in an easier way the size of the parentheses,
\petito_{x\to\infty}{\log x}
with maybe
\petito_{x\to\infty}[big]{\log x}
for larger parentheses or
\petito_{x\to\infty}[*]{\log x}
for autosizing ones.
You can realize the latter syntax quite easily with ltcmd (formerly xparse).
\documentclass{article}
\usepackage{amsmath}
\NewDocumentCommand{\petito}{e{}O{}m}{%
\genericlandau{o}{#1}{#2}{#3}%
}
\NewDocumentCommand{\grando}{e{}O{}m}{%
\genericlandau{\mathit{O}}{#1}{#2}{#3}%
}
\ExplSyntaxOn
\NewDocumentCommand{\genericlandau}{mmmm}
{
\operatorname*{#1}
\IfValueT{#2}{\sb{#2}}% the subscript
\jowe_landau_arg:nn { #3 } { #4 }
}
\cs_new_protected:Nn \jowe_landau_arg:nn
{
\str_case:nnF { #1 }
{
{}{\left}
{}{}
}
{ \use:c { #1l } } (
#2
\str_case:nnF { #1 }
{
{}{\right}
{}{}
}
{ \use:c { #1r } } )
}
\ExplSyntaxOff
\begin{document}
\begin{gather}
\petito{\log x} \
\petito[big]{\log x} \
\petito[Big]{\log x} \
\petito[*]{\frac{\sin x}{x}} \
\petito_{x\to\infty}{\log x} \
\petito_{x\to\infty}[big]{\log x} \
\petito_{x\to\infty}[Big]{\log x} \
\petito_{x\to\infty}[*]{\frac{\sin x}{x}} \
\grando{\log x} \
\grando[big]{\log x} \
\grando[Big]{\log x} \
\grando[*]{\frac{\sin x}{x}} \
\grando_{x\to\infty}{\log x} \
\grando_{x\to\infty}[big]{\log x} \
\grando_{x\to\infty}[Big]{\log x} \
\grando_{x\to\infty}[*]{\frac{\sin x}{x}}
\end{gather}
\end{document}

expl3,latex3andxparse? – Jasper Habicht Mar 08 '23 at 15:17latex3as here: https://tex.stackexchange.com/questions/87407/iterating-through-comma-separated-arguments – jowe_19 Mar 08 '23 at 15:20