Interesting puzzle, but not impossible to solve.
It is true that there is no vertical tolerance because pages are cut off the scroll one by one, but we can determine the badness of each page made this way. And what you are asking for is that the badness is not exeeding 100.
LaTeX doesn't offer hooks into that part of the output routine, but here is a little implementation that implements your rule:
\documentclass{article}
\usepackage{kantlipsum}
\flushbottom
\setlength\textheight{8\baselineskip}
%\showoutput
\AddToHook{cmd/@makecol/before}{%
\setbox0\vbox to\ht255{\unvcopy255}%
\typeout{==> Badness of this page body: \the\badness}%
\ifnum \badness > 100
\typeout{===> ... setting ragged}%
\setbox255\vbox{\unvbox255 \vfill}%
\else
\typeout{===> ... ok}%
\fi
}
\begin{document}
\kant[1][1-4]
\kant[2][1]
\kant[3][1]
\kant[4][1]
\kant[5][1]
\kant[6][1]
\end{document}
The badness is calculated as 100 times the cube of the the ratio by which the glue has to stretch or shrink, so 100 means it has stretched less or equal to the amount it is allowed to stretch.
However, you will find that this is way too conservative, you should probably allow for stretching twice or even more (2.5/3?) than that, so a value of 800 or 1600 or ... is more appropriate.
Actually, rereading your rule: you want to apply the stretch to the maximum and then fill, so it gets a little more complicated. In that case you would need to also measure the natural height of 255 without stretch, and then use that to figure out how much space is missing that you need to fill and the badness value to figure out how much glue you need to add at the bottom to get to a badness of 100 (instead of simply using \vfillas I did above). Excerise for the reader :-)
\toleranceworks in making paragraphs because the entire paragraph is read into memory. To the contrary, TeX only considers one page at a time and doesn't have any notion of “chunk tolerance” for producing them. – egreg Mar 31 '23 at 18:01