classicthesis is printing the heading as lower case small caps already. So you (a) need to switch off the \scshape, easiest done with \textnormal, (b) need to reduce the size of the letter, \scriptsize is one choice, other sizes could be acheived with \fontsize{size}{baselineskip}\selectfont. For the subscripts an extra \textnormal seems fine. You should also provide an optonial argument to \section so these formatting commands don't apply to the table of contents.
EDIT If you also have running heads, then you have change \sectionmark to include the formatting there.

running head

\documentclass{scrbook}
\usepackage[nochapters,pdfspacing]{classicthesis}
\newcommand{\hdlower}[1]{\textnormal{\scriptsize #1}}
\newcommand{\hdsub}[1]{\textsubscript{\textnormal{#1}}}
\usepackage{lipsum} %dummy text for illustration
\begin{document}
\tableofcontents
\bigbreak
\hrule
\bigbreak\noindent
Document Body:
\section[Fe\textsubscript{oct2} terminated
Fe\textsubscript{3}O\textsubscript{4} nanocluster model]%
{F\hdlower{e}\hdsub{oct2} terminated
F\hdlower{e}\hdsub{3}O\hdsub{4} nanocluster model}
\sectionmark{F\hdlower{e}\hdsub{oct2} terminated
F\hdlower{e}\hdsub{3}O\hdsub{4} nanocluster model}
\lipsum[1-10]
\end{document}
The above code provides two commands \hdlower and \hdsub to enact the above suggestions.
\chemsetup{chemformula/format=\normalfont\fontfamily{ppl}\selectfont}so that chemical formulae get lining numbers instead of oldstyle ones. – cgnieder Mar 08 '13 at 12:32\chemsetup{chemformula/format=\normalfont\fontfamily{MinionPro}\selectfont}instead ? – user26939 Mar 08 '13 at 12:47MinionPropackage installed but according to the documentation something like\chemsetup{chemformula/format=\normalfont\figureversion{lining,proportional}}should work. You might also need to change the redefinition of\spacedlowsmallcapsto\renewcommand\spacedlowsmallcaps[1]{\textssc{\MakeTextLowercase{#1}}}with Minion Pro. – cgnieder Mar 08 '13 at 12:59\textsubscript{}instead. But that's another issue. – user26939 Mar 08 '13 at 14:06\textsubscript{}to replace\ch{...}didn't work here. Is that expected ? – user26939 Mar 08 '13 at 14:31chemmacros(and some other packages, too) relies on up to date versions of some programming packages. – cgnieder Mar 08 '13 at 14:33