Best script for automatically copying references from a centralised bib file based on aux file

Question

In a previous question I asked about best practices when managing multiple bibtex files.

One strategy involves having a single centralised bibtex file. Then, citations in a particular article are imported based on an aux file.

While it is possible to do the importing manually using JabRef, I would like to be able to automate the process as part of a makefile.

My question:

• Which of the various scripts that are available would be best for this purpose?
• Which of them work on Windows?
• Are there are any other general tips related to automating the extraction of citations for a particular article from a centralised bibtex database?

The ones that I've seen mentioned:

• bibtools: particularly aux2bib and subset
• bibexport
• bibtool
• bibextract

Update (7/12/2010): The reason that I want to do this is that I want to upload all files to a self-contained repository. I'd also like others to be able to re-use components of the repository (e.g., the bib file). For example someone might want to get a copy of all the references in my paper and add them to their bibtex database.

Answer 1

(This is an answer)

Here's a script that I used to use for managing BibTeX .bib files before I learnt how to Do Things Properly! The only bit it doesn't do is extract the references from the .aux file. But that wouldn't be very hard to do. The vast majority of this script is about setting up the configuration. The actual meat of it is in the last loop which slurps through a .bib file and looks for the appropriate entries. The reason that that loop is so short is because someone else has already written a Text::BibTeX perl module, whereupon the rest is window dressing. If this is the sort of thing that you are after, then it would be really easy to adapt it to (a) search through the .aux file to find the references and (b) not bother with all the other things that I used it to do (such as automatically download references from the arXiv).

#! /usr/bin/perl -w

use strict;
#use Getopt::Long qw(:config auto_help bundling);
use Getopt::Long qw(:config bundling);
use Pod::Usage;
use Text::BibTeX;
use LWP::MediaTypes qw(guess_media_type);

# This is the only variables that should need customising

my $conffile = "$ENV{HOME}/.refsrc";

###
# Nothing below here should need customisation
###

my (
    $bibbase,
    $hostname,
    $docbase,
    $reffile,
    $lynx,
    $arxivdir,
    $ext,
    $authors,
    $refs,
    $reduced,
    $silent,
    $titles,
    $append,
    $view,
    $show,
    $help,
    $man,
    $gpl,
    $bibfile,
    $entry,
    $output
    );

my @reffiles;
my %mime;
my %values;
my @shrt;
my %tests;

@shrt = ("title","author");

%tests = (
    "authors" => [0, sub {
    my ($e,$b) = @_;
    my @a;
    if ($$e->exists('author')) {
        @a = $$e->split('author');
    } elsif ($$e->exists('editor')) {
        @a = $$e->split('editor');
    } else {
        return 0;
    }
    for (my $j = 0; $j <= $#$b; $j++) {
        for (my $i = 0; $i <= $#a; $i++) {
        return 1 if $a[$i] =~ /$$b[$j]/i;
        }
    }
    return 0;
    }],
    "refs" => [0, sub {
    my ($e,$r) = @_;
    for (my $i = 0; $i <= $#$r; $i++) {
        return 1 if ($$e->key =~ /^$$r[$i]$/);
    }
    return 0;
    }],
    "titles" => [0, sub {
    my ($e,$t) = @_;
    for (my $i = 0; $i <= $#$t; $i++) {
        return 1 if ($$e->get('title') =~ /$$t[$i]/i);
    }
    return 0;
    }]
    );

# Default display routine
$output = sub {
    my ($e) = @_;
    if ($append) {
    $$e->set_key($append . $$e->key);
    }
    $$e->print();
};

GetOptions (
# List of things to look for in the author/editor entries
    "a|authors=s@" => sub {
    my ($a, $b) = @_;
    push @{$values{"authors"}}, split(",",$b);
    $tests{"authors"}[0] = 1;
    },
# List of things to look for in the reference entries, array?
    "e|refs=s@" =>  sub {
    my ($a, $b) = @_;
    push @{$values{"refs"}}, split(",",$b);
    $tests{"refs"}[0] = 1;
    },
# Whether to print the full entry or a basic summary
    "r|reduced" => sub {
    $output = sub {
        my ($e) = @_;
        print $append if ($append);
        print $$e->key . "\n";
        for (my $i = 0; $i <= $#shrt; $i++) {
        $$e->exists("$shrt[$i]") and print $$e->get("$shrt[$i]") and print "\n";
        }
    }
    },
# Whether to produce any output, or just an exit code
    "s|silent" => sub {
    $output = sub {
        exit 1;
    }
    },
# List of things to look for in the title entries
    "t|title=s@" => sub {
    my ($a, $b) = @_;
    push @{$values{"titles"}}, split(",",$b);
    $tests{"titles"}[0] = 1;
    },
# Modify references by appending this string
    "u|append=s" => \$append,
# Try to display an appropriate file
    "v|view" => sub {
    $output = sub {
        my ($e) = @_;
        my ($mi,$ex);
# if the key has three digits, then it's highly likely to be an arXiv ref
        my $u = ($entry->key =~ /\d\d\d/ ? "" : chr(95));
        my @p = glob("$docbase/*/" . $entry->key . "$u*");
        if (!@p) {
        print "Couldn't find a file for: " . $entry->key . "\n";
        }
        for (my $i = 0; $i <= $#p; $i++) {
        $mi = guess_media_type($p[$i]);
        if (exists($mime{$mi})) {
            ($ex = $mime{$mi}) =~ s/%s/$p[$i]/;
            if (!fork) {
            exec $ex
                or die "Couldn't execute $ex";
            }

        } else {
            print "Couldn't find a program to view: " . $entry->key . " file: $p[$i] mimetype: $mi\n";
        }
        }
    }
    },
# Try to locate an appropriate file
    "x|locate" => sub {
    $output = sub {
        my ($e) = @_;
# if the key has three digits, then it's highly likely to be an arXiv ref
        my $u = ($entry->key =~ /\d\d\d/ ? "" : chr(95));
        my @p = glob("$docbase/*/" . $entry->key . "$u*");
        if (!@p) {
        print "Couldn't find a file for: " . $entry->key . "\n";
        }
        for (my $i = 0; $i <= $#p; $i++) {
        print $p[$i];
        }
    }
    },
    "h|?|help" => \$help,
    "m|man" => \$man,
    "gpl" => \$gpl
    ) or pod2usage(2);

pod2usage(-exitval => 1, -verbose => 0) if $help;
pod2usage(-verbose => 2 ) if $man;
exec 'perldoc perlgpl' if $gpl;

if (!-e $conffile) {
    print STDERR "$conffile does not exist.\n";
    exit 1;
}

open (CONF, $conffile)
    or die "Couldn't open $conffile for reading.\n";

while (<CONF>) {
# Set configuration variables; currently only bibbase and docbase
    if (/=/) {
    /(\w*)\s*=\s*(.*?)\s*$/;
    my $s = "\$$1 = \"$2\"";
    eval($s);
    $@ and die $@;
    next;
    }

# Stuff of the form 'name.bib' is a bibtex file to parse
    if (/(\w*\.bib)/) {
    push @reffiles, $1;
    next;
    }

# Stuff of the form 'mimetype; application' tells us how to view
# certain files

    if (/^\s*([a-z]*\/[a-z-]*);\s*(.*)/) {
    $mime{$1} = $2;
    }
}

die "$bibbase is not a directory.\n" unless -d $bibbase;
die "$docbase is not a directory.\n" unless -d $docbase;


# Loop over the bibtex reference files looking for the entries

for (my $i = 0; $i <= $#reffiles; $i++) {
    if (! -f "$bibbase/$reffiles[$i]" ) {
    print STDERR "$reffiles[$i] is not a regular file.\n";
    next;
    }

    $bibfile = new Text::BibTeX::File "$bibbase/$reffiles[$i]";

  ENTRY: while ($entry = new Text::BibTeX::Entry $bibfile) {
      if (!$entry->parse_ok) {
      print STDERR "Error in input, skipping entry\n";
      next;
      }

      next unless ($entry->metatype eq BTE_REGULAR);

      foreach my $t (keys %tests) {
      next unless $tests{$t}[0];
      if (&{$tests{$t}[1]}(\$entry,$values{$t})) {
          &$output(\$entry);
          next ENTRY;
      }
      }

  }

    $bibfile->close;
}

Configuration file example:

bibbase = /home/astacey/texmf/bibtex/bib
docbase = /home/astacey/docs/MathsPapers
hostname = front.math.ucdavis.edu
reffile = /home/astacey/texmf/bibtex/bib/arxiv.bib
lynx = /bin/lynx
arxivdir = /home/astacey/docs/MathsPapers/arXiv/
ext = .pdf


arxiv arxiv.bib
article articles.bib
book books.bib
other misc.bib

application/pdf; /usr/bin/xpdf %s
application/postscript; /usr/bin/gv %s
application/x-dvi; /usr/bin/xdvi %s

book          -> book
article       -> article
booklet       -> book
conference    -> book
inbook        -> article
incollection  -> article
inproceedings -> article
manual        -> book
mastersthesis -> other
misc          -> other
phdthesis     -> other
proceedings   -> book
techreport    -> article
unpublished   -> other

Answer 2

You can do this at a UNIX shell, with some usually true assumptions about the structure of .aux and .bib files, using mostly awk. This is less robust than a maintained program, but it is more flexible if you have any skill with shell programming, and probably less work for a couple of hundred Bibtex items than evaluating a complex piece of software. First, you can get the cited keys from the .aux file using the shell function, which accepts the filenames of multiple .aux files as arguments:

getbibkeys () { awk -F{ '$1=="\\bibcite" { print substr($2,0,length($2)-1)}' "$@"; }

You can then fetch from bibfiles, with the first argument the Bibtex key and the following arguments the .bib files to look in:

fetchbibitem () { 
    key="$1"; shift; 
    awk -v key="$key" 'BEGIN {RS="@"} $1~".*{" key "," {print "@" $0}' "$@";
}

A simple read-while loop using the above gives the desired functionality:

getbibkeys $AUXFILE | while read -r key; do fetchbibitem "$key" $BIBFILE; done

(I've tested the two shell functions, but not the read-while loop).

Be clear: these will break on some uncoventional .bib files, even though they are accepted by Bibtex.

Answer 3

(This is not an answer: it is an extended comment, but the comment box got too restrictive.)

I'm probably being a bit dense, but I don't see the point in what you are trying to do. Let me describe what I see, so that you can say what's wrong in this picture.

1. There is a central .bib file containing just about every reference you (or your coauthors) have ever even thought of.
2. You (and your various coauthors) have a variety of documents that refer to stuff in this central .bib file.
3. You want a program that looks in the .aux file for a list of references, then goes to this .bib file, extracts the relevant entries, and then puts them in a local .bib file ready for insertion in to the document.

Is that right?

If so, let me change two letters in the above. Let me change the last .bib to .bbl. So now it reads:

1. There is a central .bib file containing just about every reference you (or your coauthors) have ever even thought of.
2. You (and your various coauthors) have a variety of documents that refer to stuff in this central .bib file.
3. You want a program that looks in the .aux file for a list of references, then goes to this .bib file, extracts the relevant entries, and then puts them in a local .bbl file ready for insertion in to the document.

Correct me if I'm wrong, but isn't that exactly what BibTeX does?

What do you gain by having the local .bib file? The only thing that I can see is that it allows you to change the bibliography style without needing a trip to the central repository again. But how often do you do that? Is it so often that recreating the .bbl file is so painfully time consuming? Surely, you do that once when you submit it to a journal (or if you're like me, once each time you submit it to a journal).

If you want to be able to make changes to the local file, then I don't see that that's so much more difficult with a .bbl file than a .bib file; and in both cases you have problems with your changes surviving a recreation of the file so I can't see that that is the issue.

Answer 4

Not sure why exactly this would be necessary but here's a working python solution. It will get screwed up if you have anything on your entries' first lines other than @<type>{key, and on their last lines something other than } but you shouldn't be doing that anyway.

Save to exportcitations.py, chmod +x exportcitations.py and then just run ./exportcitations.py doc.aux bigbibtexfile.bib > actualcitedpapers.bib

Like I said, you probably don't need this (you really just want to copy doc.bbl into your tex file. But if you want it, here it is:

#!/usr/bin/python
from sys import argv 

try: 
    citems=[i[9:].strip().split("}")[0] for i in open(argv[1]).xreadlines() if i[:9]==r"\bibcite{"]
    printme=False
    for row in open(argv[2]).xreadlines():
        if row[0]==r"@":
            if row.strip().split("{")[1].strip(',') in citems:
                printme=True
        if printme:
            print(row.rstrip())
        if row.strip() == "}":
            printme=False
except:
    print("You need to input a valid aux file for arg1 and bib file for arg2.\n")

Answer 5

There is also the bib2bib program that comes with bibtex2html: http://www.lri.fr/~filliatr/bibtex2html/

A windows installer is up for download.