You get two spurious spaces: one at the start of the line, which is more evident, but also tighter vertical space above the display than below, because you're using the above skip pertaining to \small, but the below skip pertaining to \normalsize.
You can emulate the standard behavior:
\documentclass{article}
\begin{document}
Lorem Ipsum is simply dummy text of the printing and typesetting
industry. Lorem Ipsum has been the industry's standard dummy text ever
since the 1500s, when an unknown printer took a galley of type and
scrambled it to make a type specimen book. It has survived not only five
{\par\penalty\predisplaypenalty\small\[
A / B + C = D
\]\par}\penalty\postdisplaypenalty\noindent
centuries, but also the leap into electronic typesetting, remaining
essentially unchanged. It was popularised in the 1960s with the release
of Letraset sheets containing Lorem Ipsum passages, and more recently
with desktop publishing software like Aldus PageMaker including versions
of Lorem Ipsum.
\end{document}
You could define a vardisplaymath environment:
\makeatletter
\newenvironment{vardisplaymath}[1][\small]
{\par\penalty\predisplaypenalty\begingroup#1\begin{displaymath}}
{\end{displaymath}\par\endgroup\penalty\postdisplaypenalty
\@endparenv}
\makeatletter
Full example:
\documentclass{article}
\makeatletter
\newenvironment{vardisplaymath}[1][\small]
{\par\penalty\predisplaypenalty\begingroup#1\begin{displaymath}}
{\end{displaymath}\par\endgroup\penalty\postdisplaypenalty
\@endparenv}
\makeatletter
\begin{document}
Lorem Ipsum is simply dummy text of the printing and typesetting
industry. Lorem Ipsum has been the industry's standard dummy text ever
since the 1500s, when an unknown printer took a galley of type and
scrambled it to make a type specimen book. It has survived not only five
\begin{vardisplaymath}
A / B + C = D
\end{vardisplaymath}
centuries, but also the leap into electronic typesetting, remaining
essentially unchanged. It was popularised in the 1960s with the release
of Letraset sheets containing Lorem Ipsum passages, and more recently
with desktop publishing software like Aldus PageMaker including versions
of Lorem Ipsum.
\begin{vardisplaymath}[\footnotesize]
A / B + C = D
\end{vardisplaymath}
centuries, but also the leap into electronic typesetting, remaining
essentially unchanged. It was popularised in the 1960s with the release
of Letraset sheets containing Lorem Ipsum passages, and more recently
with desktop publishing software like Aldus PageMaker including versions
of Lorem Ipsum.
\end{document}
This could be adjusted to always get the parameters from \normalsize.
Thanks to Tobi for suggesting an improvement.
A simpler approach that doesn't require plunging into details such as \penalty\postdisplaypenalty (needed in order to avoid page breaks after the display) and keeping the above and below skips from \normalsize is
\newsavebox{\vardisplaymathbox}
\newenvironment{vardisplaymath}[1][\small]
{\begin{displaymath}\begin{lrbox}{\vardisplaymathbox}
#1$\displaystyle}
{$\end{lrbox}\usebox{\vardisplaymathbox}\end{displaymath}%
\ignorespacesafterend}
The result with this definition is
%(comment) character immediately after\]}. – Mico Feb 24 '16 at 11:34\smallinside the\[...\]also seems to fix it.) – Roly Feb 24 '16 at 11:38\]}forms a control symbol plus}, which ends a group. Thus, the following whitespace character is not discarded. Get rid of it either by providing a comment character or by providing a suitable control word, say,\noindent. – Mico Feb 24 '16 at 11:43\smallinside\[produces a warning that shouldn't be disregarded. – egreg Feb 24 '16 at 11:46\smallit is just the space after}(try with just the braces and no font change.) – David Carlisle Feb 24 '16 at 11:49