Understanding "stretchy" negative vertical space above framed chapter titles

Question

This question is somewhat related to my previous question overfull hbox on chapters with line above/below chapter title using KOMA-script, as (parts of) the issue occurs with various/all methods of adding the top and bottom lines (not only mdframed which I found to be the easiest to control).

Given the following minimal example to showcase the issue

\documentclass[
    parskip=full,
    open=any,
]{scrreprt}
\usepackage{mdframed}
\usepackage{lipsum}

\RedeclareSectionCommand[
    beforeskip=0pt,
    afterskip=0pt,
]{chapter}

\newmdenv[
    skipabove=16pt, % why is this necessary?
    leftline=false,
    rightline=false,
    innerleftmargin=0.5mm,
    innerrightmargin=0.5mm,
    innertopmargin=4pt,
    innerbottommargin=1pt,
    linewidth=.6pt,
]{chapterframe}

\makeatletter
\renewcommand{\chapterlinesformat}[3]{%
\begin{chapterframe}%
    \raggedchapter%
    \@hangfrom{\vphantom(#2}{#3}%
\end{chapterframe}%
\nointerlineskip}

\widowpenalty10000
\clubpenalty10000
\begin{document}
\flushbottom
\chapter{Foo}
\lipsum[1-4]
\chapter{Bar}
\lipsum[1-4]

% just the right amount of text to cause paragraph spacing to stretch
Fusce mauris. Vestibulum luctus nibh at lectus. Sed
bibendum, nulla a faucibus semper, leo velit ultricies
tellus, ac venenatis arcu wisi vel nisl. Vestibulum
diam. Aliquampellentesque, augue quis sagittis posuere,
turpis lacus congue quam, in hendrerit risus eros eget
felis.
\end{document}

I get the following two issues:

1. I need to define a skip so that the upper line of the title does not go outside the text box. This would seem to indicate, that the beforeskip in \RedeclareSectionCommand does not result in a zero skip, but a negative skip.
2. When paragraph spacing is stretched to prevent orphans/widows, the position of the chapter title relative to the page changes, as apparently the negative space is not only present but "stretchy".

Therefore my question is the following: Why is there a negative space and how can I remove it or at the very least prevent it from stretching so that the chapter title remains at a fixed position?

After some experimenting, the following appears to solve the issue, but I'd still like to know the cause (neater solutions are welcome too!):

\makeatletter
\renewcommand{\chapterlinesformat}[3]{%
\makebox[.65\linewidth][l]{%
\begin{minipage}{\linewidth}%
\vspace{1.6pt}
\begin{chapterframe}%
    \raggedchapter%
    \@hangfrom{\vphantom(#2}{#3}%
\end{chapterframe}%
\end{minipage}%
}}

The \makebox is needed to avoid the original overfull hbox issue, and the minipage appears to solve the spacing issue. There is still a minor negative, but apparently constant, vspace of 1.6pt. This may or may not be the sum of the line width and the inner bottom margin.

Answer 1

looking at the showbox output the code is compensating for \parskip but only its natural length so you end up with (just) the stretch component before the heading. I didn't trace exactly where but both headings come at the same point if you locally use a fixed parskip:

\documentclass[
    parskip=full,
    open=any,
]{scrreprt}
\usepackage{mdframed}
\usepackage{lipsum}

\RedeclareSectionCommand[
    beforeskip=0pt,
    afterskip=0pt,
]{chapter}

\newmdenv[
    skipabove=16pt, % why is this necessary?
    leftline=false,
    rightline=false,
    innerleftmargin=0.5mm,
    innerrightmargin=0.5mm,
    innertopmargin=4pt,
    innerbottommargin=1pt,
    linewidth=.6pt,
]{chapterframe}

\makeatletter

\showoutput
\showboxdepth=1
\renewcommand{\chapterlinesformat}[3]{%
\begin{chapterframe}%
    \raggedchapter%
    \@hangfrom{\vphantom(#2}{#3}%
\end{chapterframe}%
\nointerlineskip}

\widowpenalty10000
\clubpenalty10000
\begin{document}
\flushbottom
{\parskip=1\parskip\chapter{Foo}}

\lipsum[1-4]

{\parskip=1\parskip\chapter{Bar}}
\lipsum[1-4]

% just the right amount of text to cause paragraph spacing to stretch
Fusce mauris. Vestibulum luctus nibh at lectus. Sed
bibendum, nulla a faucibus semper, leo velit ultricies
tellus, ac venenatis arcu wisi vel nisl. Vestibulum
diam. Aliquampellentesque, augue quis sagittis posuere,
turpis lacus congue quam, in hendrerit risus eros eget
felis.
\end{document}

The original produces

Underfull \vbox (badness 1783) has occurred while \output is active
\vbox(595.80026+0.0)x418.25555, glue set 2.61316
.\write-{}
.\write1{\@writefile{toc}{\protect \contentsline {chapter}{\protect \numberline
 \ETC.}
.\write1{\@writefile{lof}{\protect \addvspace {10\p@ }}}
.\write1{\@writefile{lot}{\protect \addvspace {10\p@ }}}
.\glue(\topskip) 11.0
.\vbox(0.0+0.0)x0.0, glue set 13.6fil []
.\glue -13.6 plus -1.36008
.\glue -13.6

that plus -1.360008 is never compensated, and as the page is underfull any stretch component of any amount can stretch to fill the space.