Instead of using the newfloat package, you can create the example floating object in a fashion similar to the one used for the tufte classes; this will give you the possibility to automatically inherit the desired formatting. In the following example I defined the example floating object from scratch, using the "tufte way"; the code follows the tufte definitions for figure and table, which can be found in the file tufte-common.def: I also made the necessary provisions for a possible \listofexamples:
\documentclass{tufte-book}
\usepackage{framed, color}
\usepackage{lipsum}
\newcounter{example}[chapter]
\newcommand\examplename{Example}
\newcommand\listexamplename{List of Examples}
\makeatletter
\newcommand\listofexamples{%
\ifthenelse{\equal{\@tufte@class}{book}}%
{\chapter*{\listexamplename}}%
{\section*{\listexamplename}}%
% \begin{fullwidth}%
\@starttoc{loe}%
% \end{fullwidth}%
}
\renewcommand\theexample
{\ifnum \c@chapter>\z@ \thechapter.\fi \@arabic\c@example}
\def\fps@example{tbp}
\def\ftype@example{1}
\def\ext@example{loe}
\def\fnum@example{\examplename\nobreakspace\theexample}
\newenvironment{example}[1][htbp]
{\begin{@tufte@float}[#1]{example}{}}
{\end{@tufte@float}}
\let\l@example\l@figure
\makeatother
\begin{document}
\begin{example}
\caption{Currently, this caption is in the right margin similar to the table caption below.}
\definecolor{shadecolor}{rgb}{1,0.8,0.3}
\begin{shaded}
\lipsum[2]
\end{shaded}
\end{example}
\begin{table}
\caption{This is how I want the new float environment caption to behave.}
\centering
\begin{tabular}{llr}
Column 1 & Column 2 & Column 3 \\
Value A1 & Value A2 & Value A3 \\
\end{tabular}
\end{table}
\end{document}

If you want all your example environments to be with a colored background, you can include the coloring environment directly in the definition of example. The following example illustrates this, but using the mdframed environment (from the mdframed package) instead of shaded (from the framed package); this gives a better vertical alignment between the object and the caption (compare with the example above):
\documentclass{tufte-book}
\usepackage{xcolor}
\usepackage{mdframed}
\usepackage{lipsum}
\definecolor{shadecolor}{rgb}{1,0.8,0.3}
\newcounter{example}[chapter]
\newcommand\examplename{Example}
\newcommand\listexamplename{List of Examples}
\makeatletter
\newcommand\listofexamples{%
\ifthenelse{\equal{\@tufte@class}{book}}%
{\chapter*{\listexamplename}}%
{\section*{\listexamplename}}%
% \begin{fullwidth}%
\@starttoc{loe}%
% \end{fullwidth}%
}
\renewcommand\theexample
{\ifnum \c@chapter>\z@ \thechapter.\fi \@arabic\c@example}
\def\fps@example{tbp}
\def\ftype@example{1}
\def\ext@example{loe}
\def\fnum@example{\examplename\nobreakspace\theexample}
\newenvironment{example}[1][htbp]
{\begin{@tufte@float}[#1]{example}{}
\begin{mdframed}[backgroundcolor=shadecolor,hidealllines=true]}
{\end{mdframed}\end{@tufte@float}}
\makeatother
\begin{document}
\begin{example}
\caption{Currently, this caption is in the right margin similar to the table caption below.}
\lipsum[2]
\end{example}
\begin{table}
\caption{This is how I want the new float environment caption to behave.}
\centering
\begin{tabular}{llr}
Column 1 & Column 2 & Column 3 \\
Value A1 & Value A2 & Value A3 \\
\end{tabular}
\end{table}
\end{document}

newfloatv1.1-91 (will be uploaded to CTAN today) – May 11 '13 at 12:27