Merge two BibTeX files

Question

I was constructing a BibTeX file for my Thesis. At the middle of that I had copied that bib file to another document I had to produce, I had to add some new citations to the copy of the original.

Now I have 2 bib files that have a lot in common but differ in some citations. Is there a way to merge them? So that I have one single file with all the citations from both files but no duplicates?

Answer 1

bibtool -s bibliography1.bib bibliography2.bib will merge two bib files, keeping duplicate entries. bibtool -s -d bibliography1.bib bibliography2.bib will merge two bib files, commenting out one of the duplicated entries (not sure which one). For more info, see documentation.

Bibtool is on CTAN. Not to be confused with bibtools which is also on CTAN and probably also has the capacity to do this sort of thing...

Bibtool can also be found in Ubuntu repositories. (I wasn't able to compile the one from CTAN)

Answer 2

JabRef is a bibliography manager that has tools for merging bib-files. Or rather, for importing one file into another: File --> Import --> Import into current library. It checks for duplicates on import as far as I can see, but there is a separate tool for checking for duplicates as well: Quality --> Find duplicates.

Answer 3

I think there is no easy way to do this. Though it is possible to write a perl script alike to use regex to do this. But the problem is

1. Sometimes two entries are duplicates though they are not exactly the same, for example different capitalization in titles or extra {and }. So we have to workaround this by define some threshold for difference.

2. Sometimes two entries from the same author of the same year with a little difference in the title are indeed two papers. This make the threshold in (1) hard to define.

So I suggest you can only do this by hand. A relative easier way is do it with a BibTeX editor, for example BibDesk on Mac OS X. Import both files into it, and it shall warn you when two entries have the same citekey. So for those entries with same citekeys, which are almost surely duplicates, you can clean them up easily. If I remember correctly BibDesk and other editors do have this functionality. After this, sort all entries by title, and spot further duplicates by your eyes.

Answer 4

Just to re-ensure Torbjorn's point, JabRef checks for duplicates when using File -->Import into current database.

Entries marked with "D" are duplicates found by JabRef 2.10. You can then use "Deselect All Duplicate" button to get rid of the duplicates.

Thanks

Answer 5

Obviously you should not delete the original database files until you are certain you no longer need them. Add a couple of extra years after you are really sure, just to be on the safer side of certain.

Caveat emptor ...

Biber can do this. In tool mode, it removes entries with duplicate keys by default.

Note that this assumes that it is not necessary to merge entries, but only to ensure that exactly one copy of each entry with a unique bibkey ends up in the merged .bib file.

For example if you have something like

@article{bernard-travels,
  author = {Dickens, Bernard R.},
  year = 2003,
  pages = {3--4},
  volume = 4,
  journal = {Travelling}}

@book{bertha-stargazing,
  author = {Smithson, Bertha},
  year = 1987,
  publisher = {Great Discoveries},
  address = {Mars},
  title = {Stargazing for Gazelles}}

in one file and

@article{liz-tricks,
  author = {Davies, Elizabeth H.},
  year = 2010,
  pages = {31--43},
  volume = 56,
  journal = {Magicians' Monthly}}

@book{bertha-stargazing,
  author = {Smithson, Bertha},
  year = 1987,
  publisher = {Great Discoveries},
  address = {Mars},
  title = {Stargazing for Gazelles}}

then all will be well. But if the second file contains, for example,

@article{bernard-travels,
  author = {Dickens, Bernard R.},
  year = 2007,
  pages = {23--41},
  volume = 12,
  journal = {Travelling Again}}

@book{bertha-stargazing,
  author = {Smithson, Bertha},
  year = 1987,
  isbn = {1234567890123},
  address = {Mars},
  title = {Stargazing for Gazelles}}

then the result will probably not be what you want.

However, assuming unique bibkeys for unique entries and no need to merge individually partial entries, begin by combining the .bib files into one large .bib.

Suppose your large .bib is called big.bib. Then

biber --tool --output-format bibtex --output-file merge.bib big.bib 

will produce a combined .bib file, merge.bib, with entries sorted and de-duplicated.

biber --help

provides further details of the options available.

Answer 6

To remove the duplicates in BibTex, use the bibtex-tidy, a web app to format your BibTex source. Click the link to open it. https://varhowto.com/how-to-remove-duplicates-in-bibtex-bib-file-latex/

Please find the source of bibtex-tidy at https://github.com/FlamingTempura/bibtex-tidy . According to the current description there you can use it

• in your browser
• with a command line interface (CLI)
• as a pre-commit-hook
• as a Javasrcipt or Typescript API

Example from the trial link https://flamingtempura.github.io/bibtex-tidy/ :

Before:

After:

Answer 7

Jabref is better at detecting the duplicates than BibDesk. But, since BibDesk has AppleScript support, some people have written tools for cleaning duplicates. I recently discovered this cool tool by Andreas Fischlin's tools: http://se-server.ethz.ch/staff/af/bibdesk/. The script that says Clean Duplicates can be modified. I modified the CiteKey to Title; and, able to find more than 400 duplicates that both BibDesk and Jabref missed. You can find the modified version here: https://www.dropbox.com/s/naegllc0yaril4q/Cleanup%20Duplicates.scpt?dl=0

But, be careful. It is dangerous. It can delete non-duplicate entries if they have the same Title by chance.