11

I need to automatically remove certain fields - e.g. abstract, review, group, and file - from .bib files that contain not only bibtex entries but also (newer) biblatex entries such as @Thesis. This is the same as asked and answered in this question, but for files including biblatex entries.

This is how an example entry for a @Thesis in the .bib file might look like:

 @Thesis{Author_18_TheThesis,
  author      = {Mr Author},
  title       = {The Thesis},
  type        = {Doctoral Dissertation},
  institution = {Department of Documents, University of Stackexchange},
  year        = {2018},
  abstract    = {This is the abstract.},
  file        = {:author/Author_18_TheThesis.pdf:PDF},
  review      = {This is the review.},
  groups      = {publications},
}

bibtool, which is the accepted answer is the referred question, does not yet seem to support such entries and skips them with a warning:

@Thesis{Author_18_TheThesis,
_^
*** BibTool ERROR:  (line 123 in ./yourbibliography.bib): Unknown entry type

*** BibTool WARNING: Skipping to next '@'

How would you automatically remove such fields from .bib files containing biblatex entries? (I would prefer solutions that run on a Linux machine).

geekoverdose
  • 303
  • 1
    Welcome to TeX.SX. You question will probably receive more attention if you include some sample bibentries for people to try things out. – gusbrs Feb 12 '18 at 19:47

4 Answers4

14

Andrew Swann's answer using bibtool originally linked in the OP does work, provided the resource biblatex is given (ht to moewe).

So, for a file remove-fields.rsc:

preserve.keys = On
preserve.key.case = On
resource{biblatex}
delete.field = { abstract }
delete.field = { review }
delete.field = { groups }
delete.field = { file }

The command:

bibtool -r remove-fields ./references.bib -o new.bib

will result in:

@Thesis{      Author_18_TheThesis,
  Author    = {Mr Author},
  Title     = {The Thesis},
  Type      = {Doctoral Dissertation},
  Institution   = {Department of Documents, University of Stackexchange},
  Year      = {2018},
  ispreprintpublic={test}
}
gusbrs
  • 13,740
  • This works just fine. Short note: I also needed to prevent bibtool from changing keys and preventing it from changing their casing, which is possible with --preserve.key.case=on --preserve.keys=on. – geekoverdose Feb 13 '18 at 08:54
  • @geekoverdose, I added those. – gusbrs Feb 13 '18 at 10:24
11

Notice: By default, biber silently drops fields which are unknown to the datamodel. So, if you happen to use non-standard fields, see update below.

You can use biber's tool mode with an appropriate sourcemap.

In biber's tool mode it operates on your datasource, so you should run if on command line as, e.g.:

biber --tool --configfile=biber-tool.conf <mybibfile>.bib

(Of course, <> are there just for you to substitute with the adequate file name).

biber-tool.conf specifies what you want biber to do with your file. In your case, you want to delete certain fields from your entries, so a sourcemap is the adequate tool for that. The contents of biber-tool.conf would than be (with some other options relevant to the control of output appearence):

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <output_fieldcase>lower</output_fieldcase>
  <output_indent>2</output_indent>
  <output_align>true</output_align>
  <sourcemap>
    <maps datatype="bibtex" map_overwrite="1">
      <map map_overwrite="1">
        <map_step map_field_set="abstract" map_null="1"/>
        <map_step map_field_set="review" map_null="1"/>
        <map_step map_field_set="groups" map_null="1"/>
        <map_step map_field_set="file" map_null="1"/>
      </map>
    </maps>
  </sourcemap>
</config>

With this setup, the command above biber will output a new file <mybibfile>_bibertool.bib having removed the specified fields.

The result for your entry would be:

@thesis{Author_18_TheThesis,
  author      = {Author, Mr},
  institution = {Department of Documents, University of Stackexchange},
  date        = {2018},
  title       = {The Thesis},
  type        = {Doctoral Dissertation},
}

Update: By default, biber silently drops fields which are unknown to the datamodel. So, if you have any of those in your datasource, or if you are unsure and wants to be warned about any ignored fields, use the option --validate-datamodel:

biber --tool --validate-datamodel <mybibfile>.bib

For your entry, that would give you the following warnings:

WARN - Datamodel: Entry 'Author_18_TheThesis' (references.bib): Field 'groups' invalid in data model - ignoring
WARN - Datamodel: Entry 'Author_18_TheThesis' (references.bib): Field 'ispreprintpublic' invalid in data model - ignoring

Now, if the dropping of these fields is not wanted and you must keep them, you have to extend biber's data model to include them, which can be done in your custom biber-tool.conf, by adding of your non-standard field(s) within the <fields>...</fields> group. In your case (assuming here these are "literal" type fields):

<field fieldtype="field" datatype="literal">ispreprintpublic</field>
<field fieldtype="field" datatype="literal">groups</field>

And, within the group <entryfields><entrytype>thesis</entrytype>...<\entryfields> add:

<field>ispreprintpublic</field>
<field>groups</field>

The resulting custom biber-tool.conf is then:

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <output_fieldcase>lower</output_fieldcase>
  <output_indent>2</output_indent>
  <output_align>true</output_align>
  <sourcemap>
    <maps datatype="bibtex" map_overwrite="1">
      <map map_overwrite="1">
        <map_step map_field_set="abstract" map_null="1"/>
        <map_step map_field_set="review" map_null="1"/>
        <map_step map_field_set="groups" map_null="1"/>
        <map_step map_field_set="file" map_null="1"/>
      </map>
    </maps>
  </sourcemap>
  <datamodel>
    <fields>
      <field fieldtype="field" datatype="literal">ispreprintpublic</field>
      <field fieldtype="field" datatype="literal">groups</field>
    </fields>
    <entryfields>
      <entrytype>thesis</entrytype>
      <field>ispreprintpublic</field>
      <field>groups</field>
    </entryfields>
  </datamodel>
</config>

With it, for this input:

@Thesis{Author_18_TheThesis,
  author      = {Mr Author},
  title       = {The Thesis},
  type        = {Doctoral Dissertation},
  institution = {Department of Documents, University of Stackexchange},
  year        = {2018},
  abstract    = {This is the abstract.},
  file        = {:author/Author_18_TheThesis.pdf:PDF},
  review      = {This is the review.},
  groups      = {publications},
  ispreprintpublic = {test},
}

The output is:

@thesis{Author_18_TheThesis,
  author           = {Author, Mr},
  institution      = {Department of Documents, University of Stackexchange},
  date             = {2018},
  ispreprintpublic = {test},
  title            = {The Thesis},
  type             = {Doctoral Dissertation},
}

This is not specially straightforward. But, to quote a comment from PLK on the matter: "The benefits of having a datamodel in tool mode outweigh this sort of problem."

gusbrs
  • 13,740
  • Seems to work like a charm. Give me some minutes to check it out on the real .bib files I got here - I'll accept your answer once I see the result. – geekoverdose Feb 12 '18 at 20:11
  • Yes, please proceed with care and do some checking, for biber is performing more operations there than simply remove the fields. And even though I know how this works and believe it is safe, I don't have much experience in using this tool in real case scenarios of my own. – gusbrs Feb 12 '18 at 20:14
  • @geekoverdose, I added some of the available options that control the appearance of the output. To be used as required (and if required), of course. – gusbrs Feb 12 '18 at 20:43
  • This approach works well, but it also has a side effect I did not expect: it seems to strip entries of all non-standard-fields, e.g. a ispreprintpublic field would be gone afterwards. I can either accept your answer now and ask a new question for this (my preferred way), or if you happen to know how to prevent removal of non-standard-fields you could also just include it in the answer. Thx! – geekoverdose Feb 12 '18 at 20:53
  • @geekoverdose, that's indeed unfortunate, and I don't know how to fix it. Perhaps someone else can. You can leave this question open and add the requirement of non-standard fields, or open a new question, as you think it is best. I do think that the silent vanishing of some fields, even if non-standard is a significant problem with this answer anyway. – gusbrs Feb 12 '18 at 21:15
  • 1
    AFAIK Biber's tool mode only outputs fields defined in the data model. If you use any fields Biber doesn't know it is silently dropped. I didn't find an explicit passage about this in the documentation I have to admit, so you might want to drop PLK a line about this. You can also request that unknown fields be included in the output .bib, but I'm not sure if that is possible. https://github.com/plk/biber/issues – moewe Feb 12 '18 at 21:48
  • @moewe thx for the pointer. For now I've opened a new question at tex to see if anyone knows a way or workaround, but if nothing pops up I might do that. – geekoverdose Feb 12 '18 at 21:53
  • @moewe, I see. Thanks for the response! I do think this is worth reporting, if nothing else a clear statement in the documentation would be worth it. – gusbrs Feb 12 '18 at 21:53
  • @geekoverdose, see update. It is not specially convenient, but should be enough, as far as I can see. – gusbrs Feb 12 '18 at 23:06
5

Another option is the bib2bib tool, which provides pretty flexible and reliable ways to filter/extract/expand bibtex entries. This (little known) utility is part of the bibtex2html tool suite. (Note: you have to look for the PDF documentation, the HTML documentation does not discuss bib2bib.)

For instance, to remove certain fields from a biblatex.bib file and save the output to bibtex.bib: 

bib2bib --remove abstract --remove file --remove review -ob bibtex.bib biblatex.bib

It is also possible to specify filter and sorting options, rename fields (--rename <old> <new>) and so on.

Daniel
  • 37,517
1

You can just do it manually using a text editor, like Sublime. Activate the Regex function (option+command+R on Mac) and look for:

abstract = {.*},

and substitute it with nothing.

This removes anything between abstract = { and },

You can apply this to other fields.

Carlo Bianchi
  • 53
  • Well, that depends on the editor used. The OP did not mention his used editor. Could be a solution or not (extra installing an editor for this I would not do) ... – Mensch Aug 08 '19 at 19:36
  • 2 problems with this: a) I need to automate the task, hence would need to programmatically use the API of whichever text editor you would suggest. b) If I understand correctly then your solution would not work with nested brackets, such as review={Some \emph{nasty}, nasty text.}, or related special cases. Those are present in my data. – geekoverdose Aug 22 '19 at 13:04