How good a juggler is TeX?

Question

I recently made an adaptation of pgfpages which allows one to gather in several typeset pages and then lay them out in some layout over several physical pages (the original pgfpages only allows one physical page). Thinking about other possible extensions, I started wondering just how many typeset pages TeX can hold in its memory at a time. In the comments to a Village's self-answer on How to sort the order of pages so that they can be printed in a folded book?, there's a hint that TeX can hold a large number of pages before actually shipping any out. And when I think about what little I know about a PDF document, it would appear that TeX has to process the entire document before it can actually write a single page to the output (I believe that the PDF has to say at the start how long it is).

PGFPages works by storing the pages in boxes, so this might be a bit different to how TeX stores actual pages before writing them to the file. If this is different, I'm more interested in how many boxes TeX can hold at a single time.

I realise that this also might be system-dependent, in which case I'd like to get some idea of how it is dependent, and what the "stock TeX" can do as I'd rather not modify the memory capabilities just for this juggling ability.

So how many boxes can TeX juggle before it runs the risk of dropping them?

Answer 1

Within limits you can adjust the size of allocated memory in texmf.cnf but to show the defaults using an unmodified default TL2012 (later versions should have similar values)

$ pdftex "\def\n{100}\input box.tex"
This is pdfTeX, Version 3.1415926-2.4-1.40.13 (TeX Live 2012)
 restricted \write18 enabled.
entering extended mode
(./box.tex )
No pages of output.
Transcript written on box.log.

so that works, but increase by a factor of 10 and

$ pdftex "\def\n{1000}\input box.tex"
This is pdfTeX, Version 3.1415926-2.4-1.40.13 (TeX Live 2012)
 restricted \write18 enabled.
entering extended mode
(./box.tex
! TeX capacity exceeded, sorry [main memory size=3000000].
\body ->\hbox {}
                \advance \a 1 \ifnum \a < 1000 
\next ->\body 
              \let \next \iterate \else \let \next \relax \fi \next 
\kbox ...{}\advance \a 1 \ifnum \a < 1000 \repeat 
                                                  }
\body ->\hbox {\kbox 
                     }\advance \a 1 \ifnum \a < \n 
\next ->\body 
              \let \next \iterate \else \let \next \relax \fi \next 
l.17 \repeat
            }
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on box.log.

The test file looks like:

\newcount\a

\def\kbox{{\a=0
\loop
\hbox{}%
\advance\a 1
\ifnum\a< 1000
\repeat}}


\setbox0\hbox{%
\a=0
\loop
\hbox{\kbox}%
\advance\a 1
\ifnum\a < \n
\repeat}

\bye

---

On the other hand, if you switch engines, it takes a while but:

$ time luatex "\def\n{10000}\input box.tex"
This is LuaTeX, Version beta-0.70.2-2012062812 (TeX Live 2012)
 restricted \write18 enabled.
(./box.tex )
No pages of output.
Transcript written on box.log.

real    0m17.010s
user    0m15.381s
sys     0m1.575s

But if you push your luck too far:

$ time luatex "\def\n{100000}\input box.tex"
This is LuaTeX, Version beta-0.70.2-2012062812 (TeX Live 2012)
 restricted \write18 enabled.
(./box.tex
! TeX capacity exceeded, sorry [node memory size=197283195].
\body ->\hbox {}
                \advance \a 1 \ifnum \a < 1000 
\next ->\body 
              \let \next \iterate \else \let \next \relax \fi \next 
\kbox ...{}\advance \a 1 \ifnum \a < 1000 \repeat 
                                                  }
\body ->\hbox {\kbox 
                     }\advance \a 1 \ifnum \a < \n 
\next ->\body 
              \let \next \iterate \else \let \next \relax \fi \next 
l.17 \repeat
            }
!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on box.log.

real    0m56.810s
user    0m33.539s
sys     0m4.055s

Answer 2

When I was developing the booklet package, which puts pages into boxes and after a certain number of pages have been processed then outputs all the boxes in one go, I noticed that the number of boxes that TeX could handle depended on the content of the pages. The more complex the boxed pages the fewer boxes before memory overflow. I never quantified the relationship. So, following on from David's answer, mine is: it all depends.