Prevent `biber --tool` from removing non-standard fields in .bib files

Question

This question was about a side effect encountered in this answer. One way to avoid this side effect is to not use biber but bibtool -biblatex as stated in this answer.

I'm using biber --tool to automatically remove fields like abstract, review, groups, and file from biblatex entries in .bib files. A side effect of this is that all non-standard-fields¹ seem to be removed as well, which I want to prevent. Here's an minimal example:

This is the entry in the .bib file mybib.bib:

@Thesis{Author_18_TheThesis,
 author           = {Mr Author},
 title            = {The Thesis},
 type             = {Doctoral Dissertation},
 institution      = {Department of Documents, University of Stackexchange},
 year             = {2018},
 abstract         = {This is the abstract.},
 file             = {:author/Author_18_TheThesis.pdf:PDF},
 review           = {This is the review.},
 groups           = {publications},
 ispreprintpublic = {true},
}

And this is the clean-bibfiles.conf configartion file for biber --tool:

<?xml version="1.0" encoding="UTF-8"?>
<config>
    <output_fieldcase>lower</output_fieldcase>
    <output_indent>2</output_indent>
    <output_align>true</output_align>
    <sourcemap>
        <maps datatype="bibtex" map_overwrite="1">
            <map map_overwrite="1">
                <map_step map_field_set="abstract" map_null="1"/>
                <map_step map_field_set="review" map_null="1"/>
                <map_step map_field_set="groups" map_null="1"/>
                <map_step map_field_set="file" map_null="1"/>
            </map>
        </maps>
    </sourcemap>
</config>

Running biber --tool --configfile=clean-bibfiles.conf mybib.bib will produce a mybib_bibertool.bib file that contains this biblatex entry:

@thesis{Author_18_TheThesis,
  author      = {Author, Mr},
  institution = {Department of Documents, University of Stackexchange},
  date        = {2018},
  title       = {The Thesis},
  type        = {Doctoral Dissertation},
}

The non-standard-field ispreprintpublic has been stripped from the entry. To be precise, the groups and review fields would also be stripped even if they would not be in the sourcemap, as both are also non-standard fields or are "borrowed" from the article type, respectively. How to prevent the automatic stripping of such fields when using biber --tool?

¹See e.g. "2.1.1 Regular Types" and "4.2.4.1 Generic Fields" in the biblatex documentation.

I don't think this is possible at the moment without adding these fields to the data model. All unknown fields are dropped. If you need a different behaviour, start a feature request: https://github.com/plk/biber/issues — moewe, Feb 12 '18 at 21:49
My solution would be to try BibTool, but of course your other question shows you tried it and that is why you ended up with Biber in the first place. I don't use BibTool, but the manual shows a possibility to define new entry types. And BibTool even has a biblatex mode that already knows @thesis. Run BibTool with bibtool -r biblatex <file.bib> — moewe, Feb 12 '18 at 22:03
I've opened the request: https://github.com/plk/biber/issues/211 — gusbrs, Feb 12 '18 at 22:13
Ha found, a bit of the docs that describe this. Its in the changes section at the beginning and not in the section describing the tool mode: "Fields which are not defined in the data model described in the default biber-tool.conf are ignored and are neither read nor output. If custom fields are required, they should be defined in the data model by using a custom tool mode config file." This confirms my suspicion that it is indeed not possible without a data model definition. — moewe, Feb 12 '18 at 22:15
Oh, there it was then. I sort of tried adding the field with a datamodel, but then it only accepts the fields that are there stated, and not these plus the default data model. Which would then require that the full data model is specified in biber-tool.conf. — gusbrs, Feb 12 '18 at 22:22
You might give bib2bib a try, as described in the linked original question. — Daniel, Feb 12 '18 at 23:17
Perhaps it is worth to link this answer here too: https://tex.stackexchange.com/a/415013/105447. For the record, even though the question and another answer to it are linked in the OP. @PaulGaborit, I wish I had gotten your advice earlier! :) Thanks! — gusbrs, Feb 13 '18 at 16:30
Seems legit. I think one of you could make a full answer out of the comment above and the details in the first answer to the other question - it might help others running into the same question. — geekoverdose, Feb 13 '18 at 17:01
I guess a self answer would be fair enough here. It is good that the info is concentrated there. But not to leave this one unattended for, with some short outline and the link, might be a good idea as well. — gusbrs, Feb 13 '18 at 18:03
@PaulGaborit solution is correct.The benefits of having a datamodel in tool mode outweigh this sort of problem. You should add unknown fields to the datamodel by copying the default tool-mode config file. Also, you can force biber to report on skipping things unknown to the data model by using the --validate-datamodel option. — PLK, Feb 13 '18 at 20:47

gusbrs · Accepted Answer · 2024-01-09T19:07:00.403

Indeed, biber does not handle fields which are not part of the datamodel, so if your data source has non-standard fields or entrytypes, you have to provide biber with a data model which includes them.

But, before proceeding with this task, it is a good idea to know what is actually missing in the default datamodel that you need. You can do that with the --validate-datamodel option:

biber --tool --validate-datamodel mybib.bib

On a .bib file containing that entry of yours, that gives you the following warnings:

WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'review' invalid in data model - ignoring
WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'groups' invalid in data model - ignoring
WARN - Datamodel: Entry 'Author_18_TheThesis' (mybib.bib): Field 'ispreprintpublic' invalid in data model - ignoring

Which lets you know precisely that you have to include the fields review, groups and ispreprintpublic to the datamodel.

(Of course, adding review and groups to the datamodel to then drop them in the sourcemap would not be the "usual". I'm just doing it here for the sake of generality and procedure.)

To set your custom biber-tool.conf file (you called it clean-bibfiles.conf, let's keep it that way, but in general there are other operations which could be achieved through this procedure) you will have to adjust the datamodel to include the line(s) of your non-standard field(s) within the <fields>...</fields> group. In your case (assuming here these are "literal" type fields):

<field fieldtype="field" datatype="literal">review</field>
<field fieldtype="field" datatype="literal">groups</field>
<field fieldtype="field" datatype="literal">ispreprintpublic</field>

And, within the group <entryfields><entrytype>thesis</entrytype>...</entryfields> add:

<field>review</field>
<field>groups</field>
<field>ispreprintpublic</field>

The resulting clean-bibfiles.conf should then be:

<?xml version="1.0" encoding="UTF-8"?>
<config>
  <output_fieldcase>title</output_fieldcase>
  <output_indent>2</output_indent>
  <output_align>true</output_align>
  <sourcemap>
    <maps datatype="bibtex" map_overwrite="1">
      <map map_overwrite="1">
        <map_step map_field_set="abstract" map_null="1"/>
        <map_step map_field_set="review" map_null="1"/>
        <map_step map_field_set="groups" map_null="1"/>
        <map_step map_field_set="file" map_null="1"/>
      </map>
    </maps>
  </sourcemap>
  <datamodel>
    <fields>
      <field fieldtype="field" datatype="literal">review</field>
      <field fieldtype="field" datatype="literal">groups</field>
      <field fieldtype="field" datatype="literal">ispreprintpublic</field>
    </fields>
    <entryfields>
      <entrytype>thesis</entrytype>
      <field>review</field>
      <field>groups</field>
      <field>ispreprintpublic</field>
    </entryfields>
  </datamodel>
</config>

With this configuration file, the command:

biber --tool --validate-datamodel --configfile=clean-bibfiles.conf mybib.bib

will output mybib_bibertool.bib with the entry as desired:

@Thesis{Author_18_TheThesis,
  Author           = {Author, Mr},
  Institution      = {Department of Documents, University of Stackexchange},
  Date             = {2018},
  Ispreprintpublic = {test},
  Title            = {The Thesis},
  Type             = {Doctoral Dissertation},
}

To check the default datamodel and configuration, you can find the location for the default biber-tool.conf with:

biber --tool-config

This may not be specially straightforward. But, to quote PLK's comment above: "The benefits of having a datamodel in tool mode outweigh this sort of problem."

Prevent `biber --tool` from removing non-standard fields in .bib files

1 Answers1

Linked