Using the MWE
\documentclass[fontsize=8pt, a4paper, DIV12]{scrbook}
\usepackage{amsmath}
\begin{document}
\[ e^t = x^{(5)} - 2x^{(3)} + \dot{x}
= \sum_{k = 0}^{5} \binom{5}{k} p^{(k)}(t) e^t
-2\sum_{k = 0}^{3} \binom{3}{k} p^{(k)}(t) e^t
+ \sum_{k = 0}^{1} \binom{1}{k} p^{(k)}(t) e^t
\]
\end{document}
I get the output

with strange looking binomial coefficients. Root cause is the font size.
Is there any way to get the usual look without changing font size?

\tbinominstead of\binom? – Werner Oct 31 '14 at 20:21\smalland\large-- and also withamsartusing the same three sizes, i see that withscrbook, the parentheses, except with\large, the parentheses fail to "cover" the contents. usingamsart, the parentheses are consistently tall enough to encompass both elements fully, and the result is much less objectionable. i also tried it with the (not well known} 9pt option ofamsart, and in that test, the parens on the\smallinstance are undersized. so i think it's attributable to the size, as already suggested. – barbara beeton Oct 31 '14 at 20:55pmatrix), I might as well keep it. – arney Oct 31 '14 at 21:32