Original answer
The line
\cmpdinit[init-sub = true]{1,2,3{a,b}}
initates three labels: 1, 2 and 3{a,b}. Every further label used in the text will still be created (and write a warning in the log that it hasn't been initiated). Thus the numbering goes:
Here are two compounds: \cmpd{1} and \cmpd{2}.% numbers 1 and 2
\cmpdreset% new numbers will start at 1 again
Here are two transition states: \cmpdTS{TS1.a} and \cmpdTS{TS2}.
% 1a and 2
Which of these two transition states is involved in transformation of \cmpd{1} to \cmpd{2}?
% 1 and 2 again
What about \cmpd{3.a}? It is formed from \cmpd{2} via \cmpdTS{TS3.a}
% `3' was never declared but gives `3' as this is the next number to be declared,
%`TS3' consequently gives `4`
So what you want is this:
\cmpdinit[init-sub = true]{1,2,3.{a,b}}
which initiates labels 1, 2 and 3 and the sublabels 3.a and 3.b.
You could have seen this easier by using the option init-strict which will cause errors if an uninitiated label is used. My suggestion would be to do something like the following, i.e. declare every label in the preamble together with init-strict:
\documentclass[b5paper,12pt]{article}
\usepackage{chemnum}
\cmpdsetup{init-strict,init-sub}% error if unititialized label is used
% regular labels
\cmpdinit{1,2,3.a}
% transition states
\newcommand*{\cmpdTS}[1]{\cmpd[cmpd-prefix=\textbf{TS},cmpd-space={}]{#1}}
\cmpdreset
\cmpdinit{TS1.a,TS2,TS3.a}
\begin{document}
Here are two compounds: \cmpd{1} and \cmpd{2}.
Here are two transition states: \cmpdTS{TS1.a} and \cmpdTS{TS2}.
Which of these two transition states is involved in transformation of \cmpd{1} to \cmpd{2}?
What about \cmpd{3.a}? It is formed from \cmpd{2} via \cmpdTS{TS3.a}.
\end{document}
Edit 2014/03/14
Since 2014/03/12 version 1.0 of chemnum is available. This this new release comes with a change in many ways I'd like to give the solution for the new version as well:
\documentclass[b5paper,12pt]{article}
\usepackage{chemnum}[2014/03/12]
\setchemnum{init,log=verbose}
% regular labels
\initcmpd{1,2,3.a}
% transition states
\resetcmpd
\initcmpd[pre-label-code=\textbf{TS}]{TS1.a,TS2,TS3.a}
\begin{document}
Here are two compounds: \cmpd{1} and \cmpd{2}.
Here are two transition states: \cmpd{TS1.a} and \cmpd{TS2}.
Which of these two transition states is involved in transformation of \cmpd{1}
to \cmpd{2}?
What about \cmpd{3.a}? It is formed from \cmpd{2} via \cmpd{TS3.a}.
\end{document}

\cmpdinit. Change it to\cmpdinit[init-sub = true]{1,2,3.{a,b}}. – cgnieder Sep 17 '12 at 20:03