Biblatex-biber bug

Question

I am victim of the known biblatex-biber bug described here which leads to mass spurious missing citations, resolved only by deletion of the biber cache (dozens of files). Previously it was occasional but now it is driving me up the wall - I have to delete the cache on every single compile across multiple projects.

Anyone know the status of this bug and the purpose of the cache before I resort to abandoning biber completely.

Biber needs to cache things as it's a self-contained Perl program, at least in the standard distribution. However, once the code is extracted into the cache all should be well. If this is happening every time you compile then something else is up. We'll need more detail of what exactly you are doing. — Joseph Wright, Nov 01 '14 at 06:59
OK many thanks -- will work on this and keep multiple serial copies of whole cache. But based on my experiences (and what others seem to have experienced) it's not simply a matter of extracted biber code in cache but something to do with runtime code either in cache (or stored elsewhere but deleting cache somehow re-sets). For example on one occasion cite complied perfectly except for two never "found". They became "found" after cache deletion and recompile. So cache must know something more than simply its own code? — Aubrey Blumsohn, Nov 01 '14 at 10:28
This is very strange. The "cache" is just where biber unpacks itself to on the first run after install of a new version. It stores no data about the actual work it does there. It does store some data in system temp dirs which depend on the OS. It could conceivably be that the cache location where it unpacks to is monitored and cleaned up by some other process which breaks it but it's unlikely. We'd need to know the version, the OS and the output of biber --cache too. — PLK, Nov 02 '14 at 21:06
Apologies for getting back so slowly on this. I have explored the problem in detail, and this bug appears very clearly due to runtime data that is stored in the cache. Latex runs clearly does change the cache. — Aubrey Blumsohn, Nov 29 '14 at 22:46
You should 'ping' the person you hope to reach by doing @(unless it's the person who posted the question or answer). I think you also need to provide the output ofbiber --cache`.... — jon, Nov 29 '14 at 23:20
See my "comment" posted in answer section. I think the main issue is that the biber cache does not appear to be operating simply as a program-file unpacking destination and is clearly changed between runs in contrast to what might be thought above. Should I run biber --cache on a nonworking cache or a working one, or both. Anyway I will add that to my "answer" below. — Aubrey Blumsohn, Nov 29 '14 at 23:27
I ran biber --cache and that gives no information other than the location of the cache (see below) — Aubrey Blumsohn, Nov 29 '14 at 23:33
jon explained how to ping @PLK in an earlier comment. On GNU/Linux, the cache does not always change when biber is run. I compared 4 snapshots of the cache (snap, run biber, snap, biber, snap, biber, snap). The first 2 differed. The final 3 were identical. The difference consisted in the addition of a single .so to the second cache. I changed the bib resource and citations in one case without the cache changing. The addition of anything seems odd if biber merely unpacks itself when the version is new, but nothing like you are reporting on Windows. — cfr, Nov 29 '14 at 23:55
Any news on this problem? Did you try to run Biber on a minimal example with only two or three citations? — moewe, Dec 28 '14 at 09:00
I'm voting to close this question as off-topic because it is about a problem that is not reproducible and the information provided does not suffice to investigate further (the OP has not gotten back to any information requests lately). — moewe, Jan 30 '15 at 06:43

Aubrey Blumsohn · Answer 1 · 2014-11-30T18:39:34.043

Apologies for getting back so slowly on this. I have explored the problem in detail, and this bug appears very clearly due to runtime data that is stored in the cache. Latex runs clearly does change the cache. I have to put my findings as an answer as the comments do not allow enough explanation. It would be really great if this could be fixed somehow

Here is my last run, and the scenario and its reproduction are completely reproducible. First we start with the scenario as experienced by many users that citations are simply not found.

Trying to run biber by itself yields the following output:

Biber run as : "%texpath%bin\win32\biber.exe" "%dirname%"

Output: INFO - This is Biber 1.7 INFO - Logfile is 'familytree.blg' data source C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\recode_data.xml not found in . Press any key to continue . . .

Then I did nothing more than deleting the cache which was at

C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985

and re-ran the identical biber command which yielded an output

Type option:t
INFO - This is Biber 1.7
INFO - Logfile is 'familytree.blg'
INFO - Reading 'familytree.bcf'
INFO - Found 94 citekeys in bib section 0
INFO - Processing section 0
INFO - Looking for bibtex format file '../bibfiles/testtypes.bib' for section 0
INFO - Decoding LaTeX character macros into UTF-8
INFO - Found BibTeX data source '../bibfiles/testtypes.bib'
WARN - BibTeX subsystem: C:\Users\aubrey1\AppData\Local\Temp\1PpfasL1rn\testtypes.bib_12504.utf8, line 2580, warning: possible runaway string started at line 2575
WARN - BibTeX subsystem: C:\Users\aubrey1\AppData\Local\Temp\1PpfasL1rn\testtypes.bib_12504.utf8, line 2744, warning: possible runaway string started at line 2723
WARN - BibTeX subsystem: C:\Users\aubrey1\AppData\Local\Temp\1PpfasL1rn\testtypes.bib_12504.utf8, line 3394, warning: possible runaway string started at line 3366
INFO - Looking for bibtex format file 'local.bib' for section 0
INFO - Decoding LaTeX character macros into UTF-8
INFO - Found BibTeX data source 'local.bib'
WARN - I didn't find a database entry for '[[bigsheet]]' (section 0)
INFO - Overriding locale 'English_United States.1252' default tailoring 'variable = shifted' with 'variable = non-ignorable'
INFO - Sorting 'entry' list 'none' keys
INFO - No sort tailoring available for locale 'English_United States.1252'
INFO - Writing 'familytree.bbl' with encoding 'UTF-8'
INFO - Output to familytree.bbl
INFO - WARNINGS: 4

Here is a link to the diff on the cache before and after regeneration (html format via winmerge) for anyone who knows what files are supposed to be doing there

Link to cache diff

Here is a comparison of the cache folder (via Winmerge) before deleting and the regenerated cache (as a csv file). Clearly there are hundreds of files that are present in the cache before deletion that are not present on regeneration. Soi the cache cannot be simply an unpacking of program code. I have only shown the first 10th of so of the differences due to limitations on answer length

Compare C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985 with Copy before deleting2\cache-c3e641bac9e7e4b5ab17068122bff38686710985
29/11/2014 22:53:21
Filename,Folder,Comparison result,Left Date,Right Date,Extension,Differences
Algorithm,inc\lib,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib,* 29/11/2014 22:36:44,,,
attributes,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:47,,,
B,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
BerkeleyDB,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
Compress,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
Raw,inc\lib\auto\Compress,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\Compress,* 29/11/2014 22:36:46,,,
Bzip2,inc\lib\auto\Compress\Raw,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\Compress\Raw,* 29/11/2014 22:36:46,,,
Zlib,inc\lib\auto\Compress\Raw,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\Compress\Raw,* 29/11/2014 22:36:46,,,
Crypt,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
SSLeay,inc\lib\auto\Crypt,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\Crypt,* 29/11/2014 22:36:46,,,
Cwd,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
Data,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
Dumper,inc\lib\auto\Data,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\Data,* 29/11/2014 22:36:46,,,
DBD,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
mysql,inc\lib\auto\DBD,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\DBD,* 29/11/2014 22:36:46,,,
ODBC,inc\lib\auto\DBD,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\DBD,* 29/11/2014 22:36:46,,,
Pg,inc\lib\auto\DBD,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\DBD,* 29/11/2014 22:36:46,,,
SQLite,inc\lib\auto\DBD,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\DBD,* 29/11/2014 22:36:46,,,
DBI,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:46,,,
Devel,inc\lib\auto,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto,* 29/11/2014 22:36:47,,,
PPPort,inc\lib\auto\Devel,Left only: C:\Users\aubrey1\AppData\Local\Temp\par-61756272657931\cache-c3e641bac9e7e4b5ab17068122bff38686710985\inc\lib\auto\Devel,* 29/11/2014 22:36:47,,,

A diff won't help here probably because the PAR::Packer module names the unpacked files with hashes which may well change between unpacks, particularly the shared libraries which certain modules need. To really compare, you'd have to compare file contents or sizes. — PLK, Nov 30 '14 at 17:44
Have kept various versions of the cache, so if anyone has a specific compare they would do, happy to try. Should say that the number of files is vastly different (by several hundred), so not just a renaming — Aubrey Blumsohn, Nov 30 '14 at 18:28
I am going to add a link to the complete HTML format report of the file diff (in my original "answer") for anyone who can make sense of what files are involved. — Aubrey Blumsohn, Nov 30 '14 at 18:36
You seem to have an older version of Biber, maybe the problem goes away if you update Biber (and biblatex with it, for compatibility reasons). I do have the feeling that Biber temporarily stores some runtime data in the cache directory, but have never experienced any problems (that could not be handled by deleting the cache folder) with that. I'm also not sure how dynamically the files in the cache are actually unpacked. — moewe, Dec 01 '14 at 05:55
Indeed the warnings you get in your new output could be connected to a faulty .bib file. Can you reproduce the errors with a slightly shorter .bib file (not containing 94+ entries) and also show an MWE of your document. So we can really try to reproduce your issue. — moewe, Dec 01 '14 at 05:58
Did you correct the problems found in the bib file? It is giving you line numbers. Reproducing with a bib with fewer than 3,000+ lines would be a good idea, unless the problem is essentially concerned with the handling of large files. — cfr, Dec 30 '14 at 19:54

Biblatex-biber bug

1 Answers1

Linked