Why does biber increase compilation time of pdflatex runs dramatically (factor 2.8!!)?

Question

Since I am using biber, the compilation time for my document has increased dramatically. It's a dissertation with <300 pages and a full latexmk compilation suddenly needed 26 minutes which is not acceptable IMHO.

I already read, that biber might be slower than bibtex (see https://tex.stackexchange.com/a/53302/4009), but can it really make such a difference?

I already described the problem here (https://tex.stackexchange.com/a/230749/4009 ) and narrowed the problem down to biber which seems to have a large impact on compiling time.

I use the following biblatex setup:

\usepackage[natbib=true, style=numeric-comp, backend=biber, 
 defernumbers,useprefix,firstinits=true, maxnames=99, maxcitenames=3]{biblatex}

My question

Is biber really that much slower than bibtex
Can I do anything to still use biber but decrease compiling time?
It seems that it is not the biber run itself, that makes compilation slow, but using backend=biber slows down pdflatex a lot
can I use other options (kind of a draft mode) to speed up biber and only use the slow compilation for the final versions?

Unfortunately, I've switched all my input files (.tex, .bib) to UTF-8 encoding, as I head problems with exotic characters and it seems to me there is no easy way back to the good old latin1 times. And it seems, that UTF8 requires the biber backend.

Test setup

my LaTeX document, limited to one long chapter by \includeonly,
98 pages with many many references, many of them in large tables.
large .bib file (5.6 MB) with >3000 references, 350 of them cited in the document
I'm compiling from OS X terminal to measure the duration of the compilation:
latexmk -C; time pdflatex main.tex

Test results

with biber:

biber cache removed once with rm -rf `biber --cache`
latexmk -C; time pdflatex Diss.tex
Output written on Diss.pdf (83 pages, 2130562 bytes).
real 2m17.499s
user 1m47.626s
sys 0m1.030s
biber Diss
real 0m27.257s
user 0m20.568s
sys 0m1.247s
pdflatex Diss.tex Output written on Diss.pdf (95 pages, 2453607 bytes).
real 4m7.708s
user 3m27.864s
sys 0m1.029s
again pdflatex Diss.tex Output written on Diss.pdf (95 pages, 2452370 bytes). real 4m33.769s user 3m36.491s sys 0m1.252s

with `bibtex8`

remove all aux and bbl files etc.
latexmk -C; time pdflatex Diss.tex
Output written on Diss.pdf (82 pages, 2130383 bytes).
real 0m48.470s
user 0m19.161s
sys 0m0.522s
time bibtex Diss
(There were 200 error messages) sure :-(
real 0m1.258s
user 0m0.850s
sys 0m0.061s
time pdflatex Diss.tex
Output written on Diss.pdf (95 pages, 2452225 bytes).
real 2m26.433s
user 1m44.921s
sys 0m0.932s
second pdflatex Diss.tex
Output written on Diss.pdf (95 pages, 2450936 bytes).
real 2m27.148s
user 1m51.958s
sys 0m0.953s

My `.biber.conf`

path: ~/.biber.conf

<config>
  <sourcemap>
    <maps datatype="bibtex" map_overwrite="1">
      <map>
        <map_step map_field_set="Annote" map_null="1"/>
        <map_step map_field_set="Abstract" map_null="1"/>
        <map_step map_field_set="Keywords" map_null="1"/>
        <map_step map_field_set="comment" map_null="1"/>
      </map>
     </maps>
   </sourcemap>
</config>

Is this ok?
Are the field names case sensitive?
is something missing here?

Test 2 with example document

I set up an example document using the biblatex-example.bib file included in TeXLive 2014. For details please see my answer below , as it seems to crash the question, if I insert that much text here. The sample document compiles 80% faster with bibtex! Maybe you want to try it on your machine and see how long it takes?

Mhh, obviously Biber takes longer to compile as you already hinted at. It should not make your pdflatex run significantly longer though (sure, if you use BibTeX you cannot use all of Biber's features and we load biblatex1.sty and not biblatex2.sty). But I doubt that we can actually compare pdflatex's compile times here since due to the errors you don't get the same output (probably a lot of entries are skipped - less for pdflatex to write out). — moewe, Mar 01 '15 at 16:33
@MaxNoe: as I wrote in my question, it is not the biber run itself but the pdflatex runs which take much longer with biber! — MostlyHarmless, Mar 01 '15 at 16:33
@Moewe: hm, unfortunately I have no copy of the PDF created with biber, but the bibtex version has 291 references in the bibliography, which is not that bad. Which info do you need to see if the results can be compared or not? — MostlyHarmless, Mar 01 '15 at 16:36
You might be able to speed up Biber (not sure if you are really interested in that) by not having all the references in the .bib file, but only the ones you actually cite. 5.6MB is quite hefty. - Though Biber only really processes entries if they are cited, so the performance gain might be quite small. — moewe, Mar 01 '15 at 16:36
Where does the 26 minutes come from? The times you show don't seem to support that.... — cfr, Mar 01 '15 at 16:37
@cfr: the test above is done with only 1 chapter of 98 pages. The 26 minutes were the shocking result of one latexmk run of the complete document (nearly 300 pages) — MostlyHarmless, Mar 01 '15 at 16:38
If you run the sequence pdflatex -> biber -> pdflatex manually, how long does that take? Note that you do not always need a third pdflatex run with biblatex/biber. — cfr, Mar 01 '15 at 16:39
I would just compare the bibliography output by Biber vs the one after a BibTeX run. If there is roughly the same number of entries in a very similar format, it should be fine. I would really like to see an example of this live (I realise that is probably a bit too much to ask for) since I would not have expected the Biber "goodies" to hold up compilation significantly (though there is unarguably more for LaTeX to process). — moewe, Mar 01 '15 at 16:41
@cfr: what do you mean by "run manually"? Am I not doing that already by running the commands in terminal in my test setup? — MostlyHarmless, Mar 01 '15 at 16:43
I would not be running a complete compilation sequence every time. In general, you don't need all your references, cross-references etc. resolved each time you compile a draft. I'd just use a simple pdflatex run in most cases, only doing the additional stuff when required. — cfr, Mar 01 '15 at 16:44
@cfr: I already do that, but >3 minutes is still a long time if I could have the same result in 40 seconds. I also use auctex and often compile only the current region to check if I did not add new mistakes with my edit, but that still felt painfully slow suddenly, so I tried to find the root cause. — MostlyHarmless, Mar 01 '15 at 16:46
@moewe: as you can imagine I do hesitate so give the complete "works" outside. :-( Maybe we can still find a way to find out what's making things so slow in my case, cause I think that's still not normal. How can I extract only the cited references from my .bib file? — MostlyHarmless, Mar 01 '15 at 16:47
biber --output_format=bibtex file should give you a .bib file with only the cited entries. If you have time you could try and replicate your problem with biblatex-examples.bib (comes with biblatex and is automatically found by LaTeX) and some blind text, so you don't have to upload sensitive things. — moewe, Mar 01 '15 at 16:52
On my pc using pdflatex in "biber-mode" slows down the compilation by a factor around 2. I don't find this surprising. You only have to compare the bbl-files created by biber and bibtex: biber provides much more information and pdflatex has to handle it. — Ulrike Fischer, Mar 01 '15 at 17:16
@UlrikeFischer: ok, but how do you explain that the first run of pdflatex takes so much longer? the .bbl file is created only after the first biber run which is afterwards, afaik and during its first run, pdflatex does not have to handle the informative bbl log file. But still it's a good hint - I discovered that biber prints all the abstracts and keywords in the .bbl file!! This could be a hint, because pdflatex has to process all that each time... but it still does not explain the duration of the first pdflatex run. — MostlyHarmless, Mar 02 '15 at 07:10
@PLK: I hope it's ok, that I address you directly, as you seem to be the expert for biber. Could you have a look at my question and give me a hint how to reduce my compiling time? — MostlyHarmless, Mar 02 '15 at 07:16
I get a factor of around 20% in the first compilation when I simply cite everything from biblatex-examples.bib. The main difference between biber and bibtex here is that in "biber mode" pdflatex write a 2000-lines-.bcf, but only some lines in the aux and a small bib when in bibtex mode. Btw: .bbl is not a log-file, it is your bib prepared for pdflatex. — Ulrike Fischer, Mar 02 '15 at 09:18
@Ulrike: could you share with me your example .tex document? — MostlyHarmless, Mar 02 '15 at 09:22
Have you tried removing options from biblatex to find the most time-consuming one? There is a draft option for biblatex to speed up compilation. — Jonas Nyrup, Mar 02 '15 at 09:39
@JonasNyrup: in a first attempt I removed all options except the backend, but that did not speed up biber (see my answer below) — MostlyHarmless, Mar 03 '15 at 08:19
Have you considered asking the maintainers of Biber why this might be happening? — shadowtalker, Apr 23 '15 at 02:44
@ssdecontrol: I thought I had done that, but it seems I did not directly. In this answer here (http://tex.stackexchange.com/a/53302/4009 ) the maintainer states that there are good reasons, why biber is slower than bibtex , but I'll ask him about this question here. However it would help me a lot if other people could repeat my "benchmark" on their hardware to see how much time it takes on a recent computer. — MostlyHarmless, Apr 23 '15 at 19:30
With biblatex-example.bib, a compilation with Biber is only about 60% slower than with BibTeX, not the orders of magnitude that were being discussed. — ShreevatsaR, Jan 03 '20 at 23:08

MostlyHarmless · Answer 1 · 2015-03-03T08:12:48.717

General Test setup for performance comparison

I've set up a test file which uses the biblatex-examples.bib References included in TeXLive 2014. It cites each reference 10 times (not very elegant, but it works).

Results:

with `biber` backend

If I cite each source only 1 time (instead of 10), it takes 29 seconds for the whole script to run with biber (3x pdflatex, 1x biber).
If I cite each source 10 times, it takes already 1min36sec for the script (with biber).

with `bibtex` backend

If I change to backend=bibtex (still each source cited 10 times), the whole script needs less then 18 seconds - this is 80% less than with biber!!

further tests:

using 10 times \nocite{*} instead of citing each source separately reduces the time with biber to 21 sec!
removing all options from biblatex (except backend=biber) and again citing each source 10 times separately does not decrease compiling time (1min40sec)

Compiler scripts:

run: `time compile-bibtex.sh`

#!/bin/bash

rm -rf `biber -cache`

time pdflatex -interaction=batchmode BiberPerformance.tex

time bibtex Biberperformance

time pdflatex -interaction=batchmode BiberPerformance.tex
time pdflatex -interaction=batchmode BiberPerformance.tex

run: `time compile-biber.sh`

#!/bin/bash

rm -rf `biber -cache`

time pdflatex -interaction=batchmode BiberPerformance.tex

time biber Biberperformance

time pdflatex -interaction=batchmode BiberPerformance.tex
time pdflatex -interaction=batchmode BiberPerformance.tex

LaTeX source code `BiberPerformance.tex`

## \documentclass{scrbook}

%%%% {PACKAGES} %%%%
\usepackage[UKenglish, ngerman]{babel} %Sprachpakete Brit. Englisch, Deutsch Neue Rechtschreibung
\usepackage[utf8]{inputenc}
\usepackage[TS1, T1]{fontenc} %Kodierungen für Text 1 und Text Symbols
\usepackage{aecompl}
\usepackage{textcomp} %für Mü-Symbol
\usepackage{blindtext}

%Pakete für Zusatzfunktionen
\usepackage{hyperref} %Automatisches Verlinken der PDF-Datei


%Weitere Pakete
\usepackage[natbib=true, style=numeric-comp, backend=bibtex, defernumbers, useprefix,firstinits=true, maxnames=99, maxcitenames=3]{biblatex}  % für bessere Literaturverzeichnisse
\renewcommand{\bibfont}{\normalfont\small}
\renewcommand\multicitedelim{\addsemicolon\space}
\addbibresource{biblatex-examples.bib}



%In Biblatex URL ignorieren, wenn DOI existiert (Quelle: http://tex.stackexchange.com/a/5779/4009) 
\DeclareFieldFormat{url}{%
  \iffieldundef{doi}{%
    \mkbibacro{URL}\addcolon\space\url{#1}%
  }{%
  }%
}

\DeclareFieldFormat{urldate}{%
  \iffieldundef{doi}{%
    \mkbibparens{\bibstring{urlseen}\space#1}%
  }{%
  }%
}


\usepackage{csquotes} % Empfohlen bei Verwendung von biblatex



\begin{document}

%\nocite{*}


\chapter{Test}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}


\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}

\cite{westfahl:space}
\cite{set}
\cite{stdmodel}
\cite{aksin}
\cite{angenendt}
\cite{baez/article}
\cite{bertram}
\cite{gillies}
\cite{glashow}
\cite{herrmann}
\cite{kastenholz}
\cite{murray}
\cite{reese}
\cite{shore}
\cite{sigfridsson}
\cite{spiegelberg}
\cite{springer}
\cite{weinberg}
\cite{yoon}
\cite{aristotle:anima}
\cite{aristotle:physics}
\cite{aristotle:poetics}
\cite{aristotle:rhetoric}
\cite{augustine}
\cite{averroes/bland}
\cite{averroes/hannes}
\cite{averroes/hercz}
\cite{cicero}
\cite{coleridge}
\cite{companion}
\cite{cotton}
\cite{gerhardt}
\cite{gonzalez}
\cite{hammond}
\cite{iliad}
\cite{knuth:ct}
\cite{knuth:ct:a}
\cite{knuth:ct:b}
\cite{knuth:ct:c}
\cite{knuth:ct:d}
\cite{knuth:ct:e}
\cite{knuth:ct:related}
\cite{malinowski}
\cite{maron}
\cite{massa}
\cite{nietzsche:ksa}
\cite{nietzsche:ksa1}
\cite{nussbaum}
\cite{piccato}
\cite{vangennep}
\cite{vangennep:related}
\cite{vazques-de-parga}
\cite{worman}
\cite{britannica}
\cite{gaonkar}
\cite{gaonkar:in}
\cite{jaffe}
\cite{westfahl:frontier}
\cite{kant:kpv}
\cite{kant:ku}
\cite{nietzsche:historie}
\cite{brandt}
\cite{hyman}
\cite{pines}
\cite{moraux}
\cite{salam}
\cite{cms}
\cite{baez/online}
\cite{ctan}
\cite{itzhaki}
\cite{markey}
\cite{wassenberg}
\cite{almendro}
\cite{kowalik}
\cite{laufenberg}
\cite{sorace}
\cite{chiu}
\cite{padhye}
\cite{geer}
\cite{loh}
\cite{jcg}
\cite{sarfraz}
\cite{moore:related}
\cite{kullback:reprint}
\cite{kullback:related}
\cite{matuz:doody}
\cite{doody}
\cite{moore}
\cite{vangennep:trans}
\cite{vizedom:related}
\cite{kullback}
\cite{wilde}


 \printbibliography


\end{document}

You do realise that your MWE would have been slightly shorter had you used \nocite{*} instead of citing each item manually? — moewe, Mar 03 '15 at 07:55
yes, I do, but I wanted to make pdflatex print each single citation and not only [1-90]. I'm not sure, if that makes a difference in compiling time. — MostlyHarmless, Mar 03 '15 at 07:57
Aha! For exactly that purpose the new citeall packages was created (it is very new, so I'm not sure if it is available right now.) Plus I'm not sure whether printing the citations vs. not printing them makes a difference (but then I've been wrong before about things like this.) — moewe, Mar 03 '15 at 08:03
@moewe: nocite is a great performance booster for biber - I should use that in my dissertation and remove all the cite commands ;-) — MostlyHarmless, Mar 03 '15 at 08:18
It is rather inevitable - biber is not comparable to bibtex any more. Its functional set is a large superset of bibtex.It's also not in C, which bibtex is. I may look into adding options to disable parts of the internals whe.n drafting or when certain things are not needed. — PLK, Apr 28 '15 at 13:59
One thing to note is that if you are using the unpacked perl install, make sure you are not using Unicode::Collate version 1.18+ at the moment because that removed XS code and this makes it a lot slower. The new release enforces version 1.17 for unpacked installs. — PLK, Apr 28 '15 at 14:07
@PLK: thanks a lot! Could you explain your last point and tell me, what to do or not to do? — MostlyHarmless, Apr 28 '15 at 14:16
I made a mistake there - I meant Unicode::Normalize, not Unicode::Collate. So, if you are using the raw perl install, then do this: perl -MUnicode::Normalize -E 'say $Unicode::Normalize::VERSION;' and make sure it does not return anthing greater than 1.17 — PLK, May 01 '15 at 17:13
@PLK: _"and make sure it does not return anything greater than 1.17", well it does: It returns 1.18. So what to do now? — Daniel, Mar 13 '16 at 19:24
This is no longer an issue as Unicode::Normalize >=1.22 has XSUB back in. So just upgrade to the latest Unicode::Normalize and you'll be alright. — PLK, Mar 13 '16 at 19:47
Finally... What should we do to get a faster Biber? Can we try to compile anything with tools such as pp? — skan, Jun 26 '18 at 16:23

score 3 · Answer 2 · answered Mar 13 '16 at 19:49

If not using a binary biber release, that is if you are downloading biber perl source and installing yourself, make sure you have Unicode::Normalize >= 1.22 as this reinstated the XSUB version and is dramatically faster. When you do perl Build.PL, you will be warned if you don't have the latest Unicode::Normalize and as usual, Build installdeps will install the latest version along with all other dependencies.

Why does biber increase compilation time of pdflatex runs dramatically (factor 2.8!!)?

My question

Test setup

Test results

with biber:

with `bibtex8`

My `.biber.conf`

Test 2 with example document

2 Answers2

General Test setup for performance comparison

Results:

with `biber` backend

with `bibtex` backend

further tests:

Compiler scripts:

run: `time compile-bibtex.sh`

run: `time compile-biber.sh`

LaTeX source code `BiberPerformance.tex`

Linked

Why does biber increase compilation time of pdflatex runs dramatically (factor 2.8!!)?

My question

Test setup

Test results

with biber:

with bibtex8

My .biber.conf

Test 2 with example document

2 Answers2

General Test setup for performance comparison

Results:

with biber backend

with bibtex backend

further tests:

Compiler scripts:

run: time compile-bibtex.sh

run: time compile-biber.sh

LaTeX source code BiberPerformance.tex

Linked

with `bibtex8`

My `.biber.conf`

with `biber` backend

with `bibtex` backend

run: `time compile-bibtex.sh`

run: `time compile-biber.sh`

LaTeX source code `BiberPerformance.tex`