Combination of subequations and align environments leaves a lot of white space (sometimes)

Question

The fragment of my document:

\documentclass[a4paper, 12pt]

\usepackage{fancyhdr}
\fancyhf{}
    \renewcommand\headrulewidth{0.5pt}
    \renewcommand\footrulewidth{0.5pt}
    \pagestyle{fancy}
    \fancyhead[LO,RE]{\small\leftmark}
    \fancyhead[LE,RO]{\small\thepage}
    \fancyfoot[LE,RO]{xxx}
    \fancyfoot[LO,RE]{xxx}
    \renewcommand*\chapterpagestyle{fancy}

\newcommand{\diff}{\mathop{}\!\mathrm{d}}
\newcommand{\mbeq}{\overset{!}{=}}

This is known as the \emph{Born rule} or Born's probabilistic interpretation \citep{BornRule}. If two additional observables $ \{ B, C \} $ are necessary to form a CSCO (see \ref{CSCO}), the projector onto the degenerate eigenspace associated with $ a' $ is given by:
\begin{subequations}
\label{EigenSpaceProjector}
\begin{align}
\hat{\Lambda}_{a'} & = \smashoperator{\sum_{%
  \substack{i, j \mid b_{i} \in \mathscr{B},\\ 
            \hfill c_{j} \in \mathscr{C}\phantom{,}}}} 
  \,\ket{a', b_{i}, c_{j}} \bra{a', b_{i}, c_{j}} \rightarrow \ket{a'} \bra{a'} \label{EigenSpaceProjectorDiscr} \\
\text{and} \quad \hat{\Lambda}_{a'} & = \int\limits_{\mathscr{B}} \! \diff b \int\limits_{\mathscr{C}} \! \diff c \, \ket{a', b, c} \bra{a', b, c} \rightarrow \ket{a'} \bra{a'} \, , \label{EigenSpaceProjectorCont}
\end{align}
\end{subequations}
with the simpler form included if the eigenvalue $ a' $ is non-degenerate. The probability of the measurement returning any value at all has to equal unity in order for the probability interpretation \eqref{BornRule} to be viable:
\begin{subequations}
\label{NormalizedKet}
\begin{align}
\sum_{a' \in \mathscr{A}} \braket{\hat{\Lambda}_{a'}}_{\ket{\psi}} = \braket{\smash[b]{\sum_{a' \in \mathscr{A}}} \hat{\Lambda}_{a'}}_{\ket{\psi}} = \braket{\hat{\mathbb{I}}}_{\ket{\psi}} = \bra{\psi} \hat{\mathbb{I}} \ket{\psi} = \braket{\psi \vert \psi} & \mbeq 1 \, , \label{NormalizedKetDiscr} \\
\int\limits_{\mathscr{A}} \! \diff a' \, \braket{\hat{\Lambda}_{a'}}_{\ket{\psi}} = \braket{\smash[b]{\int\limits_{\mathscr{A}}} \! \diff a' \, \hat{\Lambda}_{a'}}_{\ket{\psi}} = \braket{\hat{\mathbb{I}}}_{\ket{\psi}} = \bra{\psi} \hat{\mathbb{I}} \ket{\psi} = \braket{\psi \vert \psi} & \mbeq 1 \, . \label{NormalizedKetCont}
\end{align}
\end{subequations}
The (sesqui-)linearity of the inner product \eqref{SesquiLinInnerProd} was used to establish the first equality and the completeness of the set of a self-adjoint operator's eigenkets \eqref{ComplSetKets} for the second. Kets that meet condition \eqref{NormalizedKet} are normalized (to unity).

gives the output:

However, this only happens sometimes. Over the course of me changing something unrelated, equation (1.6) went back to normal behaviour. I've never noticed it for (1.7), but my guess is, it's the same situation there: If some changes on the preceding section changes it to be three lines higher (as a random example) on the page, it's displayed fine again.

Why is that? And how can I fix it?

Due to the mysterious nature of the problem, I haven't put this into a working example because I myself cannot reproduce the error on purpose.

EDIT: The text under 1.7 is followed by a new section. Might be important ..

EDIT II:added preamble in response to Mico's comments

EDIT III: Expansion of the preamble and added a second screenshot of the output after the inclusion of \raggedbottom:

Answer 1

According to your comments, there's a sectioning header followed by a long-ish align environment at the top of the next page. Moreover, you seem to be unsure as to whether it's a good idea to allow page breaks in align and gather environments (by making scope of \allowdisplaybreaks either global or local.

If you're ok with a page break being inserted before that sectioning header, you can afford being a bit more profligate with the spacing in and around the four subequations. E.g., you could (a) replace \text{and}\quad in the first subequation with \intertext{and} and (b) insert an \intertext{and} directive between the two parts of the second subequation. Then, insert a \clearpage instruction after the end of the final paragraph and keep your fingers crossed that the "hole" at the bottom of the page isn't too prominent. (However, if the "hole" is indeed rather prominent, scratch the \intertext idea and go back to experimenting with \allowdisplaybreaks.)

\documentclass[a4paper,12pt]{scrreprt}
\usepackage[english]{babel}
\usepackage{mathtools,mathrsfs,amssymb,braket,natbib}
\newcommand{\diff}{\mathop{}\!\mathrm{d}}
\newcommand{\mbeq}{\overset{!}{=}}
\newcommand\braketI{\braket{\mkern1mu\hat{\mathbb{I}

\begin{document}
\setcounter{chapter}{1}  % just for this example
\setcounter{equation}{5}

This is known as the \emph{Born rule} or Born's probabilistic interpretation \citep{BornRule}. If two additional observables $ \{ B, C \} $ are necessary to form a CSCO (see \ref{CSCO}), the projector onto the degenerate eigenspace associated with $a'$ is given by:
\begin{subequations}
\label{EigenSpaceProjector}
\begin{align}
\hat{\Lambda}_{a'} & = \smashoperator{%
  \sum_{\substack{i, j \mid b_{i} \in \mathscr{B},\\ 
              \hfill c_{j} \in \mathscr{C}\phantom{,}}}} 
  \,\ket{a', b_{i}, c_{j}} \bra{a', b_{i}, c_{j}} \rightarrow \ket{a'}\bra{a'} \label{EigenSpaceProjectorDiscr} \\
\intertext{and} 
\hat{\Lambda}_{a'} & = \int\limits_{\mathscr{B}} \! \diff b \int\limits_{\mathscr{C}} \! \diff c \, \ket{a', b, c} \bra{a', b, c} \rightarrow \ket{a'} \bra{a'} \, , \label{EigenSpaceProjectorCont}
\end{align}
\end{subequations}
with the simpler form included if the eigenvalue $a'$ is non-degenerate. The probability of the measurement returning any value at all has to equal unity in order for the probability interpretation \eqref{BornRule} to be viable:
\begin{subequations}
\label{NormalizedKet}
\begin{align}
\sum_{a' \in \mathscr{A}} \braket{\hat{\Lambda}_{a'}}_{\ket{\psi}} 
&= \Bigl\langle \smashoperator[r]{\sum_{a' \in \mathscr{A}}} \hat{\Lambda}_{a'}\!\Bigr\rangle_{\!\ket{\psi}} 
= \braketI_{\ket{\psi}} 
= \bra{\psi} \hat{\mathbb{I}} \ket{\psi} 
= \braket{\psi \vert \psi} \mbeq 1  \label{NormalizedKetDiscr} \\
\intertext{and}
\int\limits_{\mathscr{A}} \! \diff a' \, \braket{\hat{\Lambda}_{a'}}_{\ket{\psi}} 
&= \Bigl\langle \int\limits_{\mathscr{A}} \! \diff a' \, \hat{\Lambda}_{a'}\!\Bigr\rangle_{\!\ket{\psi}} 
= \braketI_{\ket{\psi}} 
= \bra{\psi} \hat{\mathbb{I}} \ket{\psi} 
= \braket{\psi \vert \psi}  \mbeq 1 \, . \label{NormalizedKetCont}
\end{align}
\end{subequations}

The (sesqui-)linearity of the inner product \eqref{SesquiLinInnerProd} was used to establish the first equality and the completeness of the set of a self-adjoint operator's eigenkets \eqref{ComplSetKets} for the second. Kets that meet condition \eqref{NormalizedKet} are normalized (to unity).

\clearpage % by all means force a page break here
\end{document}