Your code, after the action of \expandafter, leaves
\SETCOL\AAA\else\BBB\fi
and the argument to \SETCOL is \AAA, not the first letter.
Let's try with
\expandafter\expandafter\expandafter\SETCOL\expandafter\ifx\expandafter\TST\expandafter\AAA\AAA\else\BBB\fi
After the first \expandafter acts, you get
\expandafter\SETCOL\ifx\TST\AAA aaa\else\BBB\fi
and now you get
\SETCOL aaa\else\BBB\fi
In either case the tokens from \else to \fi disappear. However, if the test returns false, you have the same problem: when \TST is not \ifx equal to \AAA, the \expandafter nightmare would leave
\SETCOL\BBB\fi
You'd need to reach also \BBB if you want that \SETCOL sees the first token of the expansion of \BBB. Better use a different strategy.
\documentclass{article}
\usepackage[usenames]{color}
\begin{document}
\def\AAA{aaa}
\def\BBB{bbb}
\let\TST\AAA
\def\SETCOL #1{\expandafter\SETCOLAUX#1}
\def\SETCOLAUX #1{\textcolor{red}{#1}}
\expandafter\SETCOL\ifx\TST\AAA\AAA\else\BBB\fi
\let\TST\BBB
\expandafter\SETCOL\ifx\TST\AAA\AAA\else\BBB\fi
\end{document}
