Automatically resolving duplicate cite keys in .bib database

Question

My current workflow is

1. edit CiteULike database on-line
2. wget http://www.citeulike.org/bibtex/user/MYUSER/tag/MYTAG?key_type=4
3. biber biblio
4. pdflatex biblio

where biblio.tex is a file for a bibliography/reference list, generated with a \nocite{*}.

The problem is that the CiteULike algorithm for generating the AuthorYearTitle key (key_type=4 in the wget URL above) may generate collisions, i. e. two different entries with the same key.

While this is for sure a CiteULike bug, I would like to patch this problem by inserting another step between wget and biber that rewrites the .bib database resolving duplicate keys, inserting a trailing 'a', 'b', ... letter after the duplicated AuthorYearTitle key. (Edit: I would like to maintain an AuthorYearTitle key in case the bibliography needs further editing.)

I would like a solution where no user intervention is required, so that steps 2., 3., 4., ... can be run in a script.

I was not able to find an existing solution to this (simple?) problem, and before reinventing the wheel I would like to ask for advice on this forum.

Note

By omitting the key_type=4 parameters, one gets unique numeric keys (citeulike:123456), so a possible strategy is to use a more robust key generator and forget about the ones generated by CiteULike.

Answer 1

Since you are using \nocite{*}, you do not need the bib tags to represent anything meaningful. So an easy solution would very simply replace all tags in the .bib file by some random character sequence (or append one, see below). For me, this worked:

sed -i -e 's~^@\(.\+\){.\+,$~(echo @\1{; cat /dev/urandom | tr -dc "a-zA-Z0-9" | fold -w 8 | head -1; echo ,) | paste -sd ""~e' biblio.bib

What does this do?

sed 

well, call sed

-i 

do inline-replacement

-e 

use extended regular expressions (for capture-group and replacement)

'

begin sed command

s

substitute

~

substitution start delimiter

^@\(.\+\){.\+,$

Search for start of line, @, a group of non-whitespace followed by {, another group of non-whitespace, ,, end of line; e.g., @article{AuthorYear,

~

substitution middle delimiter

(echo @\1{; cat /dev/urandom | tr -dc "a-zA-Z0-9" | fold -w 8 | head -1; echo ,) | paste -sd ""

Replace by

1. echo @, the first non-whitespace group captured earlier, {
2. echo one random tag (cat through head)
3. echo ,

(After removing newlines in between with paste)

~

substitution end delimiter

e

call the replacement as a shell command and use output as replacement text

'

end sed command

biblio.bib

filename

Update 1 If you want to keep the original tags, just capture them and append the random tag:

sed -i -e 's~^@\(.\+\){\(.\+\),$~(echo @\1{\2_; cat /dev/urandom | tr -dc "a-zA-Z0-9" | fold -w 8 | head -1; echo ,) | paste -sd ""~e' biblio.bib

Update 2 Instead of replacing by a random tag, this one add a, b, ... to the tags that were seen before; so the first occurrence of "AuthorYear" remains unchanged, the second becomes "AuthorYeara", and so forth.

echo > cache
sed -i -e 's~^@\(.\+\){\(.\+\),$~(echo @\1{\2; awk "BEGIN{printf \\\"%c\\\", 96+`grep \2 cache | wc -l`}" | sed "s/\\\`//"; echo \2 >> cache; echo ,) | paste -sd ""~e' biblio.bib

Can be changed to make the second "b", of course:

echo > cache
sed -i -e 's~^@\(.\+\){\(.\+\),$~(echo @\1{\2; awk "BEGIN{printf \\\"%c\\\", 97+`grep \2 cache | wc -l`}" | sed "s/a//"; echo \2 >> cache; echo ,) | paste -sd ""~e' biblio.bib