The big brace you get is built from smaller pieces and that's the minimum size as the following minimal document shows:
\documentclass{article}
\usepackage[bitstream-charter]{mathdesign}
\begin{document}
\def\test#1{$\left\{\rule{0pt}{#1pt}(#1\textrm{pt})\right.$ }
\noindent\test{6}
\test{8}
\test{10}
\test{12}
\test{14}
\test{16}
\test{18}
\test{20}
\test{22}
\test{24}
\test{26}
\test{28}
\test{30}
\end{document}

Up to the 18pt case the brace is a unique glyph. Beyond that point TeX must build the brace from smaller pieces and in the case of the Charter math fonts provided by mathdesign this has an unfortunate gap.
There's a fundamental difference in behaviour between tufte-book and article: the former sets the normal baselineskip to 14pt instead of the 12pt of the article class. So, with tufte-book, the array that contains the two cases is
\vbox(28.55986+5.04002)x83.62628
while with article it is
\vbox(24.47987+4.32002)x80.47983
So it's a bit wider both horizontally and vertically. Indeed, if you try
\[
a(1,j)=\begin{cases}
r^{j-1} & ;j\le n\\[4.8pt]
f\cdot r^{j-(n+1)} & ;j>n
\end{cases}
\]
with the article class, you'll see the same behavior as with tufte-book.
\delimiterfactor=912. That seems to be the largest value that solves the problem. Tufte-LaTeX must set it to something larger. – orome Jul 08 '12 at 22:21