Regarding the answer of Steven B. Segletes I expanded the question looking for a nested sorting function. This should be able to sort the macros of type X given in \sortedNormal as well as the macros of type Y given in \sortedSuperscript within those first macros of type X.
Writing this answer I realize, that this might be too much, especially when the macros of type X should be separated by comma. Which means I have to write to different sortedLists anyway, since macros of type Y should not be separated by comma.
So if there is a solution I appreciate the help, but maybe this gets out of hand.
\documentclass{article}
\usepackage[utf8]{inputenc}
\newcommand{\sortedSuperscript}[1]{\edef\tmp{#1}\alphabubblesort{\tmp}\textsuperscript{\sortedlist}}
\newcommand{\sortedNormal}[1]{\edef\tmp{#1}\alphabubblesort{\tmp}\sortedlist}
\newcommand{\x}[1]{\sortedSuperscript{#1}}
%% macros of type X
\newcommand{\listA}{ColorlistA\x{\red\blu}}
\newcommand{\listB}{ColorlistB\x{\gre\blu}}
\newcommand{\listC}{ColorlistC\x{\red\gre}}
%% macros of type Y
\newcommand{\red}{R}
\newcommand{\blu}{B}
\newcommand{\gre}{G}
\parindent0pt
\newcommand\alphabubblesort[1]{\def\sortedlist{}\expandafter\sortlist#1\relax\relax}
\def\sortlist#1#2#3\relax{%
\let\next\relax
\ifx\relax#2\relax%
\edef\sortedlist{\sortedlist#1}%
\else
\ifnum`#1<`#2\relax%
\edef\sortedlist{\sortedlist#1}%
\def\next{\sortlist#2#3\relax}%
\else%
\ifnum`#1>`#2\relax%
\let\tmp\sortedlist%
\def\sortedlist{}%
\def\next{\expandafter\sortlist\tmp#2#1#3\relax}%
\fi%
\fi%
\fi%
\next
}
\begin{document}
\section{expected result hard coded}
ColorlistA\textsuperscript{BR}, ColorlistB\textsuperscript{BG}, ColorlistC\textsuperscript{RG}
\section{expected result with macros}
\sortedNormal{\listC\listA\listB}
\end{document}
\xto sort its argument (and then display the result as a superscript), where the argument always consists of some combination of\red,\greand\blu? – Nicola Talbot May 19 '18 at 11:57