0

I am using \nocite{*} to create a typeset bibliography of the complete contents of all my .bib files. Under Ubuntu 18 (TexLive 2017), this worked perfectly and gave a 200 page document. Now Ubuntu 20 (TexLive 2019), I get "TeX capacity exceeded" when the document size goes over about 150 pages. The exact error message is:

Overfull \hbox (5.53261pt too wide) in paragraph at lines 182--182
[][]\T1/zgmx/m/n/10 Laur, Wolf-gang (1954), `Sli-esthorp, Schleswig, Hedeby und
 Had-deby. Die Na-men von Schleswig-
[125]
Runaway argument?
Lennard:Lawshall|Lennard:satellite|Lennard_1959|Lepelley:Normandiedia\ETC.
! TeX capacity exceeded, sorry [main memory size=5000000].
\blx@segm@0@0 ...nshire|Padel:Parochia|Padel:chang
                                                  e|commodity|JEPNS-39-Padel...
l.182 ...ibliography[heading=bibliography-heading]

!  ==> Fatal error occurred, no output PDF file produced!

The files below give a MWE. The python script generates a dummy .bib file with random entries.

make_bibtex_file.py:

# python3 make_bibtex_file.py >| foo.bib && texclean && latexmk -pdf bibliography_all

from random import seed,random,randrange,choices,choice


book='''@book{%s,
  author=   {%s},
  title=    {%s},
  publisher={%s},
  address=  {%s},
  year=     {%d},
}
'''


article='''@article{%s,
  author= {%s},
  title=  {%s},
  journal={%s},
  volume= {%d},
  pages=  {%d--%d},
  year=   {%d},
}
'''


def go(na,nb):
  alphabet='abcdefghijklmonpqrstuvwxyz'
  for i in range(na):
    k=''.join(choices(alphabet,k=10)).title()
    a=choice(alphabet).upper()+'. '+(('',choice(alphabet).upper()+'. ')[random()<0.7])+''.join(choices(alphabet,k=8)).title()
    t=''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()
    j=''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()
    v=randrange(1,200)
    p=''.join(choices(alphabet,k=randrange(5,12))).title()
    l=''.join(choices(alphabet,k=randrange(6,12))).title()
    p0=randrange(1,200)
    p1=p0+randrange(1,40)
    y=randrange(1900,2023)
    print(article%(k,a,t,j,v,p0,p1,y))
  for i in range(nb):
    k=''.join(choices(alphabet,k=10)).title()
    a=choice(alphabet).upper()+'. '+(('',choice(alphabet).upper()+'. ')[random()<0.7])+''.join(choices(alphabet,k=8)).title()
    t=''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()+' '+''.join(choices(alphabet,k=randrange(3,12))).title()
    p=''.join(choices(alphabet,k=randrange(5,12))).title()
    l=''.join(choices(alphabet,k=randrange(6,12))).title()
    y=randrange(1900,2023)
    print(book%(k,a,t,p,l,y))


if name=='main':
  seed(1)
  go(na=3000,nb=3000) # 3000,3000 ok => 123 pages
  #go(na=4000,nb=3000) # not ok
  #go(na=3500,nb=3000) # not ok
  #go(na=3400,nb=3000) # not ok
  #go(na=3300,nb=3000) # not ok
  #go(na=4000,nb=4000) # not ok => (./bibliography_all.bbl Runaway text?  Szurrycplj|Dpopvymmif|Khzggpogfj|Iazmfsnmey|Wpczdiuelp|Yyktvieuhz|Mku\ETC.  ! TeX capacity exceeded, sorry [main memory size=5000000].  \blx@dlist@entry@0@nyt/global//global/global ...kj ggsjyc|Elfwxhwise|Lhznuyxn...  l.206098     \endentry

bibliography_all.tex:

\documentclass[10pt,a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage[german,american,english]{babel}
\usepackage[babel]{csquotes}
\usepackage{time}
\usepackage{garamondx}
\usepackage[lmargin=60mm,rmargin=15mm,tmargin=20mm,bmargin=20mm]{geometry}
\usepackage{ragged2e}
\usepackage{showkeys}
\usepackage[style=authoryear,sorting=nyt,hyperref,dashed=true,backend=biber,bibencoding=utf8,texencoding=ascii]{biblatex}

\bibliography{foo}


\usepackage[colorlinks,linkcolor=blue,anchorcolor=blue,citecolor=blue,urlcolor=blue,pdfauthor={Keith Briggs},pdfmenubar=false,pdffitwindow=true,pdfwindowui=false,pdftitle={A big bibliography},plainpages=false]{hyperref}


\title{A big bibliography}
\author{Keith Briggs}
\def\Today{\number\year\space
 \ifcase\month\or January\or February\or March\or April\or May\or June\or
   July\or August\or September\or October\or November\or December\fi
     \space\number\day
}
\date{Last revised \Today\ \now}


\begin{document}
\renewcommand{\baselinestretch}{1.1}
\maketitle
\nocite{*}
\printbibliography
\end{document}
Phelype Oleinik
  • 70,814
KeithB
  • 451
  • 1
    I will not run some script to generate a bib-file. Either copy the problematic entries in your question, or put a bib somewhere for download. – Ulrike Fischer Mar 11 '22 at 11:38
  • I have put a file which fails at https://keithbriggs.info/foo.bib. However, because there is no specific entry which fails (it just depends on the size), you need to be able to generate .bib files of different sizes. Hence the python script. – KeithB Mar 11 '22 at 11:53
  • it would help if you showed the full error message from the log, which will show what capacity is exceeded and what the current limit is – David Carlisle Mar 11 '22 at 12:11
  • I already showed this in the python script: Runaway text? Szurrycplj|Dpopvymmif|Khzggpogfj|Iazmfsnmey|Wpczdiuelp|Yyktvieuhz|Mku\ETC. ! TeX capacity exceeded, sorry [main memory size=5000000]. \blx@dlist@entry@0@nyt/global//global/global ...kj ggsjyc|Elfwxhwise|Lhznuyxn... l.206098 – KeithB Mar 11 '22 at 12:16
  • @DavidCarlisle the bib is simply too large. With lualatex it compiles. (main memory fails, but even with 8000000 it doesn't compile). – Ulrike Fischer Mar 11 '22 at 12:19
  • But that was my point. Why did it work with TeXLive 2017 (using pdflatex), but not now? – KeithB Mar 11 '22 at 12:25
  • if you showed the final part of the log of the tl2017 run tex would show how close it got to reaching its limits, either the 2017 run used less memory or the limit was higher... (the error text isn't really "in the python script" (which I haven't run and is inconveniently scrolled out of view even in the form it is there), can you add the working and non working log fragments as code blocks in the question, as you see log fragments don't really work in comments as line endings are dropped – David Carlisle Mar 11 '22 at 12:43
  • Thanks for explaining; however, I cannot show the tl2017 log any more as I have now upgraded all my machines to tl2019. For the time being I can cope by using lualatex. On my real .bib files it gives a 207-page bibliography without problems. I don't expect them to get too much larger. I will add the tl2019 error message to my original posting. – KeithB Mar 11 '22 at 13:03
  • biblatex/biber creates more data now than in texlive2017, so it is not really suprising that it fails now earlier on a very large bib. – Ulrike Fischer Mar 11 '22 at 13:42

1 Answers1

2

Just to get a baseline I ran the following documents with Overleaf's TeX Live 2017 and TeX Live 2019.

Document 1

\documentclass{article}
\usepackage[utf8]{inputenc}

\begin{document}
Lorem
\end{document}

Document 2

\documentclass{article}
\usepackage[utf8]{inputenc}

\usepackage[style=authoryear]{biblatex}
\addbibresource{biblatex-examples.bib}


\begin{document}
\section{Introduction}
Lorem
\autocite{sigfridsson}
\nocite{*}
\printbibliography
\end{document}

The following table shows how many words of memory out of 5000000 TeX used in each document with the given TeX Live versions.

Document 2017 2019 difference
1 56297 62012 ~10%
2 555443 594316 ~7%

I'm afraid it is how it is: New features sometimes come with more complex or new implementations that need more memory.

You can try to compile your document with LuaTeX, which has dynamic memory allocation and is usually much better with memory-heavy documents. (There is also How to expand TeX's "main memory size"? (pgfplots memory overload) suggesting a few methods to increase TeX's memory, but there is still a hard-coded limit and Ulrike Fischer's comment suggests that is still not enough).

moewe
  • 175,683