Align beamer slides with frame breaks

Question

As addressed in First slide of references in beamer starts one line further down, a bibliography across several slides is not vertically aligned from the first slide to the next. In that case, it seems to be sufficient to remove an apparent topsep introduced by the bibliography (see moewe's answer).

But when I reduce the font size of the bibliography, the top of the list on the first slide is once again not aligned with the following slides:

\documentclass{beamer}
\usepackage{biblatex}
\addbibresource{biblatex-examples.bib}
\renewcommand{\bibfont}{\normalfont\small}

\begin{document}

\begin{frame}
\nocite{*}
\end{frame}

\begin{frame}[t, allowframebreaks]
    \frametitle{References}
    \vspace{-\topsep}
    \printbibliography
\end{frame}

\end{document}

In this MWE, the misalignment is quite small, but in my actual document, the misalignment is much bigger. What causes this behavior, and how can I undo it?

moewe suggests that this is the same problem as what occurs when an item list is broken across several slides:

\documentclass{beamer}
\begin{document}
\begin{frame}[t, allowframebreaks]
\begin{itemize}
    \item A
    \item B
    \item C
    \item D
    \item E
    \item F
    \item G
    \item H
    \item I
    \item J
    \item K
    \item L
    \item M
    \item N
    \item O
    \item P
    \item Q
    \item R
    \item S
    \item T
    \item U
    \item V
    \item W
    \item X
    \item Y
\end{itemize}
\end{frame}
\end{document}