\documentclass{beamer}
\setbeamercolor{block title}{fg=white,bg=red}
\setbeamercolor{block body}{fg=black,bg=gray}
\setbeamercolor{background canvas}{bg=yellow}
\addtobeamertemplate{frametitle}{\vspace*{-1.0ex}}{\vspace*{-1.0ex}}
\setbeamersize{text margin left=3mm}
\setbeamersize{text margin right=3mm}
\begin{document}
\begin{frame}[t]
\frametitle{Frame Title}
\color{blue}\vrule width\textwidth height\textwidth
\end{frame}
\begin{frame}[t]
\frametitle{Frame Title}
Text
\end{frame}
\begin{frame}[t]
\frametitle{Frame Title}
\color{blue}\vrule width\textwidth height\textwidth
\end{frame}
\begin{frame}[t]
\frametitle{Frame Title}
\begin{block}{Block Title}
Block Text.
\end{block}
\end{frame}
\end{document}
I want to have a constant vertical spacing between the frame title and the top line of the "arbitrary" content (text, block, ...). The reason is, that I do not like the "dancing" when I go to the next slide.

blockenvironment has some additional top spacing. And your colour '\vrule' is of course too large vertically – Mar 02 '17 at 23:54