When the breaklines=true option is given, a code that's executed after doing \lst@literate (which assigns the special meaning to the defined characters) is
\lst@ifbreaklines \lst@Def {`)}{\lst@breakProcessOther )}\fi
and this expands to
\lccode `\~=#1\lowercase {\def ~}{\lst@breakProcessOther )}
which makes the active ) expand to
\lst@breakProcessOther )
where ) is not active. All this happens when \lsthk@SelectCharTable is executed.
If, just for trying, I apply a patch with etoolbox
\makeatletter
\patchcmd{\lsthk@SelectCharTable}{`)}{`]}{}{}
\makeatother
then the right parenthesis will be printed red, but a similar problem will appear with the ] in case, say,
literate=
{[}{{\textcolor{red}{[}}}{1}
{]}{{\textcolor{red}{]}}}{1},
is specified (the right bracket will become a black right parenthesis, which I expected).
I believe this has to do with precautions listings makes in order to correctly treat parenthesized expressions. And I suspect that not much can be done about this.
If I try to patch the definition of \lsthk@SelectCharTable removing that conditional part, then the right parenthesis comes out red. But I don't know if other problems will arise.