A naive implementation of nestable parentheses is easily written
\documentclass[british]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{babel}
\usepackage{csquotes}
\makeatletter
\newcount\kart@flu@parenlevel
\newcommand*{\kart@flu@paren@toodeep}{%
\rule[0.25ex]{1.25ex}{1.25ex}%
}
\newcommand*{\kart@flu@paren@open}{%
\advance\kart@flu@parenlevel\@ne
\ifnum\kart@flu@parenlevel=\@ne
(%
\else
\ifnum\kart@flu@parenlevel=\tw@
[%
\else
\ifnum\kart@flu@parenlevel=\thr@@
\{%
\else
\PackageWarning{myparens}{%
Too deeply nested parentheses}%
\kart@flu@paren@toodeep
\fi
\fi
\fi
}
\newcommand*{\kart@flu@paren@close}{%
\ifnum\kart@flu@parenlevel=\@ne
)%
\else
\ifnum\kart@flu@parenlevel=\tw@
]%
\else
\ifnum\kart@flu@parenlevel=\thr@@
\}%
\else
\PackageWarning{myparens}{%
Too deeply nested parentheses}%
\kart@flu@paren@toodeep
\fi
\fi
\fi
\advance\kart@flu@parenlevel\m@ne
}
\newcommand*{\kartparen}[1]{%
\kart@flu@paren@open#1\kart@flu@paren@close}
\newcommand*{\kartparenopen}{\kart@flu@paren@open}
\newcommand*{\kartparenclose}{\kart@flu@paren@close}
\makeatother
\begin{document}
\kartparen{Lorem \kartparen{ipsum} dolor}
\kartparen{Lorem \kartparen{ipsum \kartparen{dolor} sit} amet}
\kartparen{Lorem \kartparen{ipsum \kartparen{dolor \kartparen{sit} amet} concectur} donec}
\kartparen{Lorem\footnote{\kartparen{ipsum}} dolor}
\end{document}
![(Lorem [ipsum] dolor)//(Lorem [ipsum {dolor} sit] amet)//(Lorem [ipsum {dolor sit amet} concectur] donec)//(Lorem dolor)](../../images/b3cd8a6b9e7bb55a538765be992ac6f9.webp)
With a bit more work this command could be changed so that brackets nest infinitely deep (the bracket type would then be determined by the modulo operation).
The hard part is to get it to interact with all other kinds of parentheses that might be added from other commands and sources beyond your manual control. There is no general solution to that problem and each command you want to work with probably needs to be patched manually.
Since you explicitly mention citations, biblatex has the commands \mkbibparens and \mkbibrackets as well as a few lower-level commands for this job. The commands are available outside of biblatex as well and interact as expected.
\documentclass[british]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{babel}
\usepackage{csquotes}
\usepackage[style=authoryear, backend=biber]{biblatex}
\addbibresource{biblatex-examples.bib}
\begin{document}
\mkbibparens{Lorem \mkbibparens{ipsum} dolor}
\mkbibparens{Lorem \mkbibparens{ipsum \mkbibparens{dolor} sit} amet}
\mkbibparens{Lorem\footnote{\mkbibparens{ipsum}} dolor}
\mkbibparens{Lorem \parencite{sigfridsson}}
\parencite{sigfridsson}
\mkbibbrackets{Lorem \mkbibparens{ipsum} dolor}
\mkbibbrackets{Lorem \mkbibbrackets{ipsum} dolor}
\end{document}
![(Lorem [ipsum] dolor)//(Lorem [ipsum (dolor) sit] amet)//(Lorem dolor)//(Lorem [Sigfridsson and Ryde 1998])//(Sigfridsson and Ryde 1998)//[Lorem (ipsum) dolor]//[Lorem (ipsum) dolor]](../../images/9aa8f7c66353280312a704eea3228263.webp)
biblatex.biblatexusually typesets its brackets and parentheses with\mkbibparensand\mkbibbracketsand those command have support for different forms at different nesting levels if they are used all the way:\mkbibparens{Lorem \parencite{sigfridsson}}produces "(Lorem [Sigfridsson und Ryde 1998])" in theauthoryearstyle while\parencite{sigfridsson}produces just "(Sigfridsson und Ryde 1998)". But not all packages and commands will have something like this.\ref, for example, would need to be modified. – moewe Jan 08 '19 at 16:08\myparensthat nests properly with itself. But when you want it to interact with other commands, (\cite,\ref) you'll probably have to patch all those that add some kind of parentheses. And then it becomes extremely important which commands you want to work with and how they are defined (i.e. which packages you use). – moewe Jan 08 '19 at 16:15It's been a long day, sorry for the lacklustre responses and lack of pro-activity.
Thank you.
– Kartöfluvofan Jan 08 '19 at 16:21\mkbibparensonly does()and[]and alternates between the two. – moewe Jan 08 '19 at 16:24