How to include crossrefs in bib files generated by biber --output-format=bibtex

Question

The Question

How can I ensure that cross-referenced entries are included in a .bib file generated by biber for a document in case the document does not cite the cross-referenced entries themselves but only entries referring to them? To be clear, I don't want the cross-referenced entries to appear separately in the list of references but I need them in the.bib file so that complete data is available for the entries I do cite.

Background

Consider the following document which uses a regular database of bibliography entries (included here as orig.bib):

\documentclass{article}
\usepackage[backend=biber]{biblatex}
  \bibliography{orig}
\usepackage{filecontents}
\begin{filecontents}{orig.bib}
@bookinbook{author-bk1,
  crossref = {author-bigbk},
  title = {Originally a Separate Book},
  pages = {345--789}}

@suppbook{author-essays-intro,
  crossref = {author-essays},
  title = {Introduction},
  pages = {1--8}}

@incollection{author-essay,
  crossref = {editor-collection},
  title = {My Essay},
  pages = {8-34},
  author = {Author, Bit Busy}}

@book{author-bigbk,
  author = {Author, Prolific},
  title = {Collected Works},
  year = 1453,
  publisher = {Printers 'R Us},
  address = {Milky Way}}

@book{author-essays,
  title = {Essays},
  author = {Author, Busy},
  year = 1578,
  publisher = {We Print 4U},
  address = {Alpha Centauri}}

@book{editor-collection,
  editor = {Editor, Fair},
  booktitle = {Collection of People's Thoughts},
  year = 1679,
  publisher = {Great Collections in Print},
  address = {Earth}}
\end{filecontents}

\begin{document}

\autocites{author-bk1,author-essays-intro,author-essay}

\printbibliography

\end{document}

As expected, this produces the following:

Complete reference list

The Promise

Now suppose that I want to generate a .bib file containing just those entries needed for this particular document. (Obviously this would be pointless with this example since orig.bib contains nothing else, but that's not typically the case, of course.)

Turning to biber's manual, I find this:

3.1.1 The output_format option

Biber is able to output ... BibTeX .bib files. .bib output is possible in tool mode, when you are converting an entire datasource file independently of any particular document (see section 3.12). It is also useful when you want, instead of a .bbl, a new .bib file containing only the cited entries from a document so that you can, for example, send a minimally complete package for typesetting to someone. To do this, you would, after the first LaTeX run, call Biber like this:
biber --output_format=bibtex test.bcf 
This would result in a new .bib file called test_biber.bib containing all cited entries in test.tex, in citation order, formatted according to the various ouput_* options.

So I try this:

biber --output_format=bibtex prawf3.bcf

The Problem

My attempt creates prawf3_biber.bib:

@BOOKINBOOK{author-bk1,
  CROSSREF = {author-bigbk},
  PAGES    = {345--789},
  TITLE    = {Originally a Separate Book},
}

@SUPPBOOK{author-essays-intro,
  CROSSREF = {author-essays},
  PAGES    = {1--8},
  TITLE    = {Introduction},
}

@INCOLLECTION{author-essay,
  AUTHOR   = {Author, Bit Busy},
  CROSSREF = {editor-collection},
  PAGES    = {8-34},
  TITLE    = {My Essay},
}

which is obviously not what you would hope for since using this .bib file will clearly not yield the required results. That is:

\documentclass{article}
\usepackage[backend=biber]{biblatex}
  \bibliography{prawf3_biber}

\begin{document}

\autocites{author-bk1,author-essays-intro,author-essay}

\printbibliography

\end{document}

will encounter problems when biber is run:

INFO - Found 3 citekeys in bib section 0
INFO - Processing section 0
INFO - Looking for bibtex format file 'prawf3_biber.bib' for section 0
INFO - Found BibTeX data source 'prawf3_biber.bib'
WARN - I didn't find a database entry for crossref 'author-bigbk' in entry 'author-bk1' - ignoring (section 0)
WARN - I didn't find a database entry for crossref 'author-essays' in entry 'author-essays-intro' - ignoring (section 0)
WARN - I didn't find a database entry for crossref 'editor-collection' in entry 'author-essay' - ignoring (section 0)

which is obviously only to be expected given that the cross-referenced entries are not there.

biber's manual does discuss using crossref data when in tool mode but that seems to apply only when converting an entire datasource. So if I wanted to apply that to orig.bib, I could do that and then I could extract the (now complete) entries into prawf3_biber.bib using the above command. [At least, I assume I could - I haven't tested.]

The Question Revisited

But how can I ensure the entries in the generated .bib file are completed with data from the cross-referenced entries without converting the original .bib file and without citing the cross-referenced entries in the document?

@jon Thanks. That's pretty much what I did except without automation. I temporarily added the crossref entries as \nocite in the document, re-latexed etc. with the original .bib datasources and then extracted a complete .bib with biber --output-format=bibtex. It isn't a very satisfactory approach, though. It is surprising biber can't do this. It just seems so powerful and this seems pretty basic. Also, the documentation doesn't even hint at this deficiency which seems somewhat remiss. Moreover, biber can do this when transforming datasources so I thought there must be a way.... — cfr, Mar 07 '14 at 22:15
@jon Finally figured it out - see my answer below (if interested). I have no idea if anybody else will find this useful or not but since I asked... — cfr, Mar 07 '14 at 22:41

score 13 · Accepted Answer · answered Mar 07 '14 at 22:38

I finally figured out how to do this. In the documentation, it looks as if using the --output_format in the way I was is distinct from invoking tool mode. So the way crossref data is treated there doesn't seem relevant to the case of concern here. However,

biber --help

is rather more helpful. In particular, it states that --output_format=bibtex only makes sense in tool mode. So, since that option is clearly intended for use in this sort of case, as the documentation suggests, it must make sense to use tool mode here, too. Moreover, the command help is also a whole lot simpler to understand for relatively simple uses. Whereas the documentation on tool mode goes into great detail about configuring the right kinds of processing for different datasource transformations, the command help clarifies the basic options better.

In particular, it turns out that there is an option to do just what is wanted here:

biber --output_format=bibtex --output_resolve prawf3.bcf

What this does is not include the cross-referenced entries in the resulting .bib. What it does instead is complete the entries which use them with data from the cross-referenced entries. The resulting prawf3_biber.bib:

@BOOKINBOOK{author-bk1,
  AUTHOR     = {Author, Prolific},
  BOOKAUTHOR = {Author, Prolific},
  BOOKTITLE  = {Collected Works},
  LOCATION   = {Milky Way},
  PAGES      = {345--789},
  PUBLISHER  = {Printers 'R Us},
  TITLE      = {Originally a Separate Book},
  YEAR       = {1453},
}

@SUPPBOOK{author-essays-intro,
  AUTHOR     = {Author, Busy},
  BOOKAUTHOR = {Author, Busy},
  BOOKTITLE  = {Essays},
  LOCATION   = {Alpha Centauri},
  PAGES      = {1--8},
  PUBLISHER  = {We Print 4U},
  TITLE      = {Introduction},
  YEAR       = {1578},
}

@INCOLLECTION{author-essay,
  AUTHOR    = {Author, Bit Busy},
  BOOKTITLE = {Collection of People's Thoughts},
  EDITOR    = {Editor, Fair},
  LOCATION  = {Earth},
  PAGES     = {8-34},
  PUBLISHER = {Great Collections in Print},
  TITLE     = {My Essay},
  YEAR      = {1679},
}

which is actually better for many purposes than including the cross-referenced entries separately.

+1. So, it's really just a kind of RTFM! (Annoying that I have used biber --help before and just never noticed, or, worse, thought to look this time.) Anyway, I guess it's good to have both a complete-the-references version since that would be much harder to do 'by hand' than my add-the-crossrefs-as-is workaround. — jon, Mar 07 '14 at 23:08
@jon Interesting. It never occurred to me before to try biber --help. I don't know why I did it. I just kept looking in the manual. I guess it is really a kind of DRTM - RTFH! That said, I'm sure this information must be somewhere in the manual. I think that really the example use of biber to generate a document-specific .bib file should include --output_resolve or, at the very least, mention it. After all, the use of crossref is common and you won't get a 'minimally complete package' without the option. I don't know generally but I do think the manual is pretty poor on this point. — cfr, Mar 07 '14 at 23:27
@jon I only saw it in relation to tool mode which the description of generating the doc-specific .bib made me think was distinct. Did you find it somewhere else? (The manual is quite long, there are a lot of manuals, life is quite short...) — cfr, Mar 08 '14 at 02:07
@jon Ah, right. I see. I do think the documentation could be improved... I agree that sometimes a solution which keeps the entries separate would be useful although generally I think using the data to complete the cited entries will often work at least as well, sometimes better. It is more concise for sending with a file if the point is just to enable typesetting, for example. — cfr, Mar 08 '14 at 03:24

jon · Answer 2 · 2014-03-08T07:45:48.597

If at all possible, cfr's answer is to be preferred for obvious reasons (esp. convenience).

However, if one needs to avoid how Biber 'resolves' crossref-ed entries and keep all entries separate -- e.g., perhaps because you cite several articles from one collection of essays and you therefore need still to have each essay merely point to the book in the bibliography rather than each provide the full bibliographical reference -- then you could do something like this:

latex      basefile.tex 
biber -m 1 basefile.bcf     # `--mincrossref | -m 1` produces a .bbl with all the references
bbl2nocite basefile tmpfile # see below
latex      basefile.tex
biber --output_format=bibtex basefile.bcf # produces `basefile_biber.bib`

In order for this to work, you need to \input what you are extracting from the .bbl somehow. I did this with a mindless bbl2nocite script, which is essentially just wrapping up grep and sed:

#!/bin/bash
#
# Find and convert all lines in a .bbl file from
# \entry{<key>}{<entrytype>}{}
# to
# \nocite{<key>}
#
# usage: bbl2nocite inputfile outputfile
INFILE=$1
OUTFILE=$2

grep '\\entry' $INFILE.bbl | sed 's/\\entry/\\nocite/g' | sed 's/}{.*/}/' > $OUTFILE.tex

The other modification you need to make is in the .tex: you need to \input what I called tmpfile.tex above. Using cfr's example file, all you need to add is:

\input{tmpfile}

right before the \printbibliography; or you could do instead:

\InputIfFileExists{tmpfile.tex}{\input{tmpfile}}{\relax}

And, finally, you get the new bibliography file, which ends _biber.bib, which should contain, in this case, six entries.

One last thing to note is that you might need to comment out the \Input... line and run through a sequence of latex, biber, and latex again to get back to the original appearance of the bibliography. In cfr's file above, e.g., this technique will lead to having all six items showing up in the bibliography, which may not be wanted.

Thanks. I wasn't aware of the -m 1 option. That seems useful! — cfr, Mar 08 '14 at 16:05