How get rid of kpsewhich searching the whole computer?

Question

I use latexmk in texstudio on win7. Working on a 400+ pages book with a lot of figures and tables, citations and indexterms, it takes more than five minutes to compile the book once. The full compilation (at least for times) takes close to half an hour. This I could stomach easily. However, I noticed that half of the compilation time it is not pdflatex running but kpsewhich.

I have organized the whole book and texlive on a fast ssd. Texlive is updated regularly. Please can anybody tell me why kpsewhich takes over for an extended time? I was told earlier that it searches the entire computer. When this is the case how to tell kpsewhich to focus on the ssd? Are there any environnemental variables to add?

BTW Running PDFlatex directly and not latexmk does not involve kpsewhich.

Kind regards Bernhard

Edit:

Here the (partial) output when removing the -silent option from the latexmk command. Pdflatex stops after

Latexmk: Log file says output to 'VierteDeutscheEdition.pdf'

and kpsewhich takes over. Unfortunately there is no kpsewhich output. After kpsewhich is run the rest of the message appears. ...

Output written on VierteDeutscheEdition.pdf (581 pages, 170933099 bytes).

SyncTeX written on VierteDeutscheEdition.synctex.gz.

Transcript written on VierteDeutscheEdition.log.

Latexmk: Index file 'VierteDeutscheEdition.idx' was written

Latexmk: Found input bbl file 'VierteDeutscheEdition.bbl'

Latexmk: Log file says output to 'VierteDeutscheEdition.pdf'

Stop of pdflatex

Latexmk: List of undefined refs and citations:

Reference `sec:sugarmetabolism' on page 39 undefined on input line 39

Reference `sec:sugarmetabolism' on page 473 undefined on input line 8

Reference `sec:sugarmetabolism' on page 85 undefined on input line 68

Biber warning: [32485] Utils.pm:164> WARN - I didn't find a database entry for 'CSV+87' (section 5) Biber warning: [32486] Utils.pm:164

WARN - I didn't find a database entry for 'THH+13' (section 5)

Biber warning: [39095] Utils.pm:164> WARN - I didn't find a database entry for 'Loveland11a' (section 9) Biber warning: [44320] Utils.pm:164

WARN - month field 'Nov' in entry 'CWT+98' is not an integer - this will probably not sort properly. Biber warning: [44336] Utils.pm:164

WARN - month field 'Nov' in entry 'RYF+98' is not an integer - this will probably not sort properly. Biber warning: [44348] Utils.pm:164

WARN - month field 'Apr' in entry 'HLH08' is not an integer - this will probably not sort properly. Biber warning: [44357] Utils.pm:164

...

Latexmk: Found biber source file(s) [Literatur/EndokrinologieKunde.bib Literatur/FatMetabolism.bib Literatur/Leptin.bib VierteDeutscheEdition.bcf]

Latexmk: Summary of warnings: Latex failed to resolve 5 reference(s)

Latexmk: All targets (VierteDeutscheEdition.pdf) are up-to-date

by default it does not search the disk outside the current directory so you must have a non standard setting for TEXINPUTS I would guess. hard to say with just this information. — David Carlisle, Jan 08 '17 at 11:35
Maybe you need to configure TeXStudio in order to run pdflatexdirectly? — Jan, Jan 08 '17 at 11:35
I have a standard texlive setting AFAIK. The texstudio call for latexmk is: latexmk -pdf -silent -latexoption="-synctex=1" % and I have latexmk configured as the standard compiler. Is there any possibility to get a log of the kpsewhich running? — Bernhard Kleine, Jan 08 '17 at 11:45
I din't use latexmk myself, but can you see what it is searching for (how do you know it is kpsewhich taking the time) kpsewhich uses different search paths depending on the filename or arguments so TEXINPUTS for tex TFMFONTS for .tfm etc, so if one of the search paths is set to search the disk without using the cached ls-r file it depends what you are searching for, — David Carlisle, Jan 08 '17 at 11:50
I see that kpsewhich is running in the windows taskmanager and I compare it to the running when pdflatex is run. The texmf.conf file has only two entries, the osfontdir=$systemroot/fonts// and the TEXLIVE_WINDOWS_TRY_EXTERNAL_PERL = 0 entry. — Bernhard Kleine, Jan 08 '17 at 12:00
What happens if you run latexmk -pdf book.tex (use your own file name), manually via a dos prompt? — daleif, Jan 08 '17 at 12:06
How do you know that that kpsewhich instance comes from the latexmk call? BTW running latexmk by hand without silent also enables you to see what it is doing and where in the compilation it is halting. — daleif, Jan 08 '17 at 12:08
Does latexmk create extra files such as fls or fdb_ something? — daleif, Jan 08 '17 at 13:09
You are also running biber. Could this be one of the cache issues with biber? Such that the correct manual test would be to run pdflatex and then biber — daleif, Jan 08 '17 at 13:10
yes it does, I looked into those for the first time and I found some font relicts from texlive 2015 on a different disk. I wonder where they come from. The texmf.conf in \texlive\2016\texmf-dist\web2c is AFAIS free of 2015 staff. — Bernhard Kleine, Jan 08 '17 at 14:49
@daleif, I removed the fls and fdb_... files and kpsewhich decided to recompile an otherwise not changed document. the new filelist includes only files from the ssd but it still takes minutes for kpsewhich to finish. — Bernhard Kleine, Jan 08 '17 at 16:22
I have now use the -diagnostic option and it seems latexmk looks for files with kpsewhich which he assumes to be input files: it starts in line 6412 of the log file I made available on http://b-kleine.com/wp-content/uploads/2017/01/viertelatexml.zip. Given that kpsewhich starts at that point I am quite sure that searching for files which do not exist is the reason for the extended searching with kpsewhich. Again, who to get rid of this behavior? — Bernhard Kleine, Jan 08 '17 at 17:46
I'll not be able to look further into this. Could you post the contents of the TEXINPUTS variable? There is an interface for it through kpsewhich though I do not remember the syntax. — daleif, Jan 08 '17 at 19:12
Well your dependent list is massive ~900 entries (do you really use that many?). And latexmk keeps track of them all and uses kpsewhich to find them in the LaTeX installations — daleif, Jan 09 '17 at 09:36
For instance you are using a very strange font setup, why is courier, helvetica, palatino, times used? minitoc seems to load 200+ files (I don't use minitoc so I don't know if this is standard) — daleif, Jan 09 '17 at 09:47
The 850 files are sections, figures, tables, chemical structures, etc. They are used. These are not the problem. I think the problem arises when kpsewhich looks for those only supposed files which it captured from the log file. These will never be found.___ I have already ask John Cullins to comment in this thread. — Bernhard Kleine, Jan 09 '17 at 13:52
Could anybody change the thread to: Why looks kpsewhich for file names it assumes from the log files but that are not existing? — Bernhard Kleine, Jan 09 '17 at 13:56
What do you mean not existing? They do exist, there are just loads and loads of them. Have you looked in the log you posted, as I mentioned it looks for 200+ minitoc files in the LaTeX installation. Looping over those takes a long time. I've not sure it it runs it kpsewhich on each file or if it collects it. — daleif, Jan 09 '17 at 14:24
BTW: do you mind posting the preamble for this document? I'd like to test it on my own system (for example to test that minitoc stuff) — daleif, Jan 09 '17 at 14:26
your own stuff is 400 files, then approx 700 from 2016/texmf-dist and approx 100 from texmf-local. Keeping track of 1200 files at each compilation does take some time — daleif, Jan 09 '17 at 14:29
I tested the code, it seems only your own stuff (things via \input, \include and gfx get passed on to kpsewhich. — daleif, Jan 09 '17 at 14:50
Could you post the result from this: kpsewhich -var-value TEXINPUTS, if that variable is not set correctly, kpsewhich may end up behaving badly. BTW: please remember, the @, otherwise I might not catch any answers — daleif, Jan 09 '17 at 14:51
C:\Users\bk>kpsewhich -var-value TEXINPUTS .;{C:/Users/bk/.texlive2016/texmf-config,C:/Users/bk/.texlive2016/texmf-var,C:/Users/bk/texmf,!!k:/texlive/2016/texmf-config,!!k:/texlive/2016/texmf-var,!!k:/texlive/texmf-local,!!k:/texlive/2016/texmf-dist}/tex/{kpsewhich,generic,}// — Bernhard Kleine, Jan 09 '17 at 18:39
the files are in k:\Buchprojekt\Vierteauflage. Latexmk is called from there. — Bernhard Kleine, Jan 09 '17 at 18:53
It looks a little strange, but that might be because it is Windows. Perhaps we should start messing with latexmk.pl (which latexmk.exe is calling), to make kpsewhich more verbose so we'll get a better idea of what on earth it is doing. Might be an idea if we went off list, you can send an email to daleif at gmail.com. How big is the entire project? Is it possible for me to test in on my Linux based system? — daleif, Jan 09 '17 at 20:32
I have asked John Collins, author of latexmk to comment an this thread. Maybe he has the idea.----The project is actually some 4GB large, would still fit on a DVD. That could be made available. — Bernhard Kleine, Jan 10 '17 at 17:48
At work I have access to a service where you can upload large files, and then the recipient get a download link. If you send me an email or give me an address, I can send you a voucher for you to use the service. (please remember the @daleif otherwise I get no notification) — daleif, Jan 11 '17 at 16:01
I sent the questioner a special version of latexmk which lists the invocations of kpsewhich (which are otherwise not visible in the output). Then we can determine definitively whether the excess time is taken by an enormous number of invocations of kpsewhich or whether it is a lot of time taken by one or a few invocations. (For reference, with my own 600+ page book, which has 100s of source files, exactly one invocation of kpsewhich is made. That's to locate a bst file, and that invocation takes a fraction of a second.) — John Collins, Jan 11 '17 at 16:25
@JohnCollins I can recreate Bernards problems using PDF files under texmf-local, why does latexmk search for these using kpsewhich when the full path to them it listed in the log (at least on texlive)? — daleif, Jan 13 '17 at 14:36
@JohnCollins ahh, now I may know, there are several entries for each gfx, <daleif-test2.pdf, id=3, 321.2pt x 240.9pt>, <use daleif-test2.pdf> and </opt/texlive/texmf-local/tex/daleif/daleif-hest0.pdf>. Hmm, but the full path to each is still listed in the fls file. Hmm. — daleif, Jan 13 '17 at 14:38
@JohnCollins I too have this problem with files in texmf-local — samcarter_is_at_topanswers.xyz, Jan 13 '17 at 14:41
@BernhardKleine I guess I have the same problem. I like latexmk very much for all small project, but for my big project I changed the building tool because of this problem. First I switched to ltx2any, which is equally comfortable - has the disadvantage that it compiles at least one time. Now I work with arara, there one has to think a bit to get the same workflow as with latexmk, but in the end you can exactly control anything. — samcarter_is_at_topanswers.xyz, Jan 13 '17 at 14:46
For my project the "time wast" is about 1/3 of the compilation time. latexmk takes 299.31 s, from these only 188.18 s are spent for biber and 4x pdflatex. (measured with latexmk -time) — samcarter_is_at_topanswers.xyz, Jan 13 '17 at 14:51
@samcarter I use latexmk even for large project. Haven't had much issues with it. But I do also use \includeonly a lot. — daleif, Jan 13 '17 at 14:52
@daleif Having my images in texmf-local causes this problem. I went away from \graphicspath, as the images are sorted in many sub folders and I was to lacy to add them all to \graphicspath. (pdf)latex itself is still as fast as always, only latexmk does not like this. — samcarter_is_at_topanswers.xyz, Jan 13 '17 at 14:55
@samcarter same here, my example with 100 files from texmf-local, 0.31s for pdflatex, 7.13 overall. — daleif, Jan 13 '17 at 14:55
@samcarter the issue seems to be the log parsing. LaTeXMK is looking for ^File: (.+) Graphic file \(type, if the file is in texmf-local (.+) ends up as being say daleif-test0.pdf, but later in the file, when pdflatex has included the file then the actual full path is added in <...>. This might though be engine dependant. — daleif, Jan 13 '17 at 15:12
@daleif You tested with softlinked files. I too have only linked the files my files texmf-local. Maybe this is part of the problem. — samcarter_is_at_topanswers.xyz, Jan 13 '17 at 15:39
@daleif Unfortunately not having links but the real files there did not help :( — samcarter_is_at_topanswers.xyz, Jan 13 '17 at 15:49
@samcarter, it is not. The issue is the manner in which latexmk finds the gfx names in the log. The data it takes does not include the path, but the path is written to the log when the backend actually include the file (have not tested with latex+dvips), and that information does include the full path. So if latexmk actually utilises this information as well, then all those kpsewhich calls are unnecessary. — daleif, Jan 13 '17 at 15:51
@samcarter oh, boy, I just got my example doc from 7.3s (0.36s for pdflatex) to 0.48s (0.36s) — daleif, Jan 13 '17 at 16:01
Seems my solution is engine dependent, but it does work with pdflatex at least — daleif, Jan 13 '17 at 16:26
I have now added \graphicspath with the path to the three directories of the pdf-files, however, kpsewhcích still searches for these files. What is wrong with this entry: \graphicspath{{K:/texlive/texmf-local/tex/generic/ChemicalStructures/bkchem/}{K:/texlive/texmf-local/tex/generic/ChemicalStructures/bkchem/melatonin/}{K:/texlive/texmf-local/tex/generic/ChemicalStructures/bkchem/tyroxine/}} — Bernhard Kleine, Jan 13 '17 at 17:54
@daleif You've found the cause of the problem. Thanks. I see several things that can be greatly improved (beyond the improvements that you sent me). I should have an improved release of latexmk soon (i.e., within a day). — John Collins, Jan 13 '17 at 21:03
@JohnCollins greatly appreciated, been a long time user, but still finds gems in it, didn't know the -time option — daleif, Jan 13 '17 at 21:18
A new version of latexmk, 4.51, is available for testing from http://personal.psu.edu/jcc8/latexmk/versions.html I think it solves the problem. Let me know the outcome. — John Collins, Jan 14 '17 at 03:52
In the latexmk version 4.51 there was not any search with kpsewhich when run on the same project. Thank you @JohnCollins. — Bernhard Kleine, Jan 14 '17 at 09:27
@BernhardKleine To add to your comment: When the -recorder option is on, as it is by default, full information on the locations of the graphics files is in the .fls file. Latexmk now knows that it doesn't need to redundantly use kpsewhich to find these same files from poorer information in the .log file. Even when the -recorder option is off, I've made a big improvement by batching the offending calls to kpsewhich into one run. — John Collins, Jan 14 '17 at 15:21
@JohnCollins Thank you so much for the new version! This saves so much time! — samcarter_is_at_topanswers.xyz, Jan 15 '17 at 11:56

score 6 · Answer 1 · edited Apr 13 '17 at 12:35

This behaviour is due to a bug in latexmk which is fixed in version 4.52b. Where possible, you should update your TeX distribution to install the current release. If this is not possible, for example if you want to change only this one package or where you do not have the privileges to update the entire distribution, you will need to install the package locally. This option should be considered a last resort, since for more complex packages there may be package dependencies that will make local installation more complicated and error-prone. (Note that the latexmk script is a single file, so the generally important warning about complex packages is not so critical here.)

(Note: In case of delays in propagation of new files to CTAN mirror sites and to the distributions, you can get the new version at http://personal.psu.edu/jcc8/latexmk/, if you have an urgent need for it.)

Diagnosis and solution of problem

The problem was triggered in a document that included lots of graphics files where the files were found by a non-trivial use of kpsearch to find them in a texmf tree. (Many documents have their graphics files in or under the document directory and don't need the searches; that's why the problem wasn't noticed earlier, even while latexmk's author was writing a 600+ page book.) Latexmk searches for dependency information in both the .fls file and in the .log file. Some relevant information in the .log file doesn't have the necessary path information, so latexmk invoked kpsewhich to find them. But it had the sometimes highly non-optimal behavior of invoking it once per file. At even 0.2s per call (on a recent iMac) that mounts up quickly. It's not that kpsewhich is actually searching the whole computer, but that it was being called far too many times. Strictly speaking, it's not a bug but a lack of an important optimization.

If latexmk's recorder mode is on, the necessary information is in the .fls file, so concerning graphics and other input files, the calls to kpsewhich are no longer made in the new version. If recorder mode is off, invocation of kpsewhich is needed, but the calls are now batched into one invocation, which still gives a dramatic improvement in run time in the problematic case. Remaining uses of kpsewhich are for finding .bst and .bib files (if needed); these are not batched (at least, not yet), but are normally very small in number, so they shouldn't be a problem.

(The new version also some ways of getting diagnostics about calls to kpsewhich.)

Answer based on comments on question and as per http://meta.tex.stackexchange.com/questions/6988/questions-which-are-bug-reports-revisited/7008. — Joseph Wright, Jan 17 '17 at 15:34
I've made a few more updates (and corrected an extra bug I introduced). The result is an official release, v. 4.52b, which is now on CTAN at http://www.ctan.org/pkg/latexmk/ as well on my website at http://personal.psu.edu/jcc8/latexmk/. So I'll take the liberty of updating your answer, which referred to the original fix. I'll also add a summary of how the problem arose. — John Collins, Jan 18 '17 at 15:04

How get rid of kpsewhich searching the whole computer?

1 Answers1

Diagnosis and solution of problem