1

I have the following latex file:

\documentclass{article}
\usepackage[keeptemps, depythontex, runall=false, fvextfile=55]{pythontex}%rerun=always, 
\begin{document}
computed by python: \pys[sname]{1+1=!{1+1}}
\end{document}

and I call it manualLMP.tex. I have a standard config file for latexmk configured for lualatex so that pythontex works. Also this time, all is fine.

If I call the same file manualLatexMavenPlugin.tex then it does not work. The important is that the name is a bit longer. It works still if I run

latexmk manualLatexMavenPlugin -lualatex="pdflatex %O %S"

or the same with xelatex, but not with lualatex.

I must admit, that I don't understand the .latexmkrc file. The core is

$extra_rule_spec{'pythontex'} = [ 'internal', '', 'mypythontex', "%R.pytxcode", "pythontex-files-%R/%R.pytxmcr", "%R", 1 ];

Unfortunately $extra_rule_spec is not mentioned in the latexmk documentation. Also I cannot find any information on the version of latexmk this documentation refers to. I use latexmk 4.79, luahbtex version 1.16. Distribution is texlive.

Just an observation: Only if I run latexmk with lualatex i get the feedback:

Latexmk: Missing input file 'pythontex-files-manualLatexMavenPlugin/manualLatexMavenPlugin.pytxmcr.Run PythonTeX to create it' (or dependence on it) from following:

which may indicate that the Run is interpreted as the extension. Gluing together seems to occur only if the filename is long. It may be a bug also in lualatex or in latexmk.

I think, the first step would be to do research on the mechanism latexmk uses.

user2609605
  • 465
  • I can reproduce your problem. The message about a missing input file is in latexmk's summary of what it found when it looked at the log file. What's happened is that latexmkrc has misinterpreted a pair of lines as a single wrapped line. (There are differences in how latexmk handles wrapped lines from lualatex and xelatex, because the two programs behavior differently.) This is a bug that I'll have to correct. – John Collins Apr 20 '23 at 20:03
  • Workaround, given that you are using texlive: Set the environment variable max_print_line to a large number, say 10000. Then the issue of latexmk being confused by wrapped v. unwrapped lines in the log file won't arise. – John Collins Apr 20 '23 at 20:18
  • The code you are using is from pythontex-latexmkrc in the example_rcfiles in the latexmk distribution. Unfortunately, the only way I could see to get things to work was to use one of latexmk's internal variables (the hash %extra_rule_spec). I've added a comment about this in the file. What I haven't decided is what the appropriate public interface is: the variable itself or a subroutine. – John Collins Apr 20 '23 at 20:24
  • @john colins, the latter is why the solution does not yet show up in the documentation. Maybe you could explain to me the mechanism. I am a bit surprised because I thought, latexmk would be based solely on detecting dependencies in the fls file. So I thought, latexmk does not read log files at all.... I would have suspected that latexmk detects INPUT ./pythontex-files-manualLMP/manualLMP.pytxmcr missing and trying to build this.. and to that end comes to the pythontex rule. This is without reading a log file. which is the charm of latexmk in my opinion. – user2609605 Apr 21 '23 at 12:03
  • Although the fls file is very important, it does not contain all the dependency information. E.g., after a first run of lualatex on your file, the only information about the .pytxmcr file is in a "No file ..." line in the log file (and nothing in the .fls file). This is the clue latexmk needs to know to run pythontex to create the fil, with the line $extra_rule_spec{'pythontex'} ... in your latexmkrc file saying how to do this. Only on a later run of lualatex, does the .fls file contain an INPUT line for the .pytxmcr file. – John Collins Apr 21 '23 at 14:14
  • I see. It is the charm of latexmk to use the fls file and not much parsing of the log file. My intention is that it is difficult to solve that situation in latexmk but in a rather elegant way in pythontex. Unfortunately, i cannot figure out what the essential difference between pythontex (package and utility) and say package biblatex with utility biber is so that latexmk works with biber but not so well with pythontex. Maybe I am naive, but isnt it quite parallel? pythontex package creates pytxcode in a latex run which is then processed by pythontex untily., analogous with bcf file. – user2609605 Apr 23 '23 at 08:38
  • I had contact several times with G poore the author of pythontex. He is an open minded, cooperative guy. If you want a change just try https://github.com/gpoore/pythontex – user2609605 Apr 23 '23 at 08:44
  • There are specific patterns of rule for bibtex and biber that are hard-coded in latexmk. In addition are hard coded (a) the extraction of signatures of when they are needed from a combination of the .log, .fls, and .aux files; and (b) the determination of the dependencies of bibtex and biber, notably on bib files. Production of bibliographies is a core functionality of LaTeX, so it makes sense to hard code what's needed in latexmk. – John Collins Apr 24 '23 at 20:29
  • The differences for the pythontex program compared with bibtex and biber are (a) the set of input and output files , (b) the relationship between the names of the primary source file and the primary output file, and (c) how one goes about determining the set of input files and generated files. That's all implemented in the pythontex-latexmkrc file in the latexmk distribution, in terms of general-purpose mechanisms of latexmk. – John Collins Apr 24 '23 at 20:45

1 Answers1

2

Answering this question gives some information on what is happening "under the hood" in latexmk for a non-trivial case.

Context: Pythontex allows a .tex document to run calculations in Python and then typeset the results. The basic workflow with pythontex is (a) Compile the .tex document with use of the pythontex package. (b) Run the pythontex program to read files generated in step (a) that specify the calculations. (c) Compile the .tex document again to use the results from step (b).

Configuration of latexmk to use pythontex: The code needed is in the pythontex-latexmkrc file in the latexmk distribution. This sets up a "rule" for using the pythontex program. Its primary input file is file of extension pytxcode generated in the *latex compilation of the tex document. Its primary output file has an extension pytxmcr, and is read by the the next compilation of the document. The rule uses a Perl subroutine to run the pythontex program with appropriate arguments, and then determine the dependencies: what files are read, and what files are written beyond the primary input and output?

Functioning: On a first compilation of the .tex file by *latex, the pythontex package writes a warning to the .log file that the pytxmcr file that it needs doesn't exist. Latexmk sees that warning and sees that the missing file can be made by the pythontex rule. So it links that rule into its rule network. The rule gets run, and generates the missing file. So latexmk recompiles the .tex file, and the missing files are read. After that information on what is read and written by the pythontex package appears in the fls file, which gives the main dependency information.

What went wrong with lualatex: Latexmk (v. 4.80) ran lualatex, but didn't run pythontex or do the second run of lualatex. In the output, it states what it is doing and gives important information about what it found

Latexmk: Getting log file 'aux_dir/manualLatexMavenPlugin.log'
Latexmk: Examining 'aux_dir/manualLatexMavenPlugin.fls'
Latexmk: Examining 'aux_dir/manualLatexMavenPlugin.log'
Latexmk: Missing input file 'pythontex-files/manualLatexMavenPlugin/manualLatexMavenPlugin.pytxmcr.Run PythonTeX to create it' (or dependence on it) from following:
No file pythontex-files-manualLatexMavenPlugin/manualLatexMavenPlugin.pytxmcr.Run PythonTeX to create it.

Notice the exact placement single quotes in the line about the missing input file. Latexmk is reporting that it thinks that the log file tells it that there is a missing file. But the extension of the file is not the expected one with extension .pytxmcr but one with the very long extension .pytxmcr.Run PythonTeX to create it (!). Looking at the log file shows that latexmk has concatenated two neighboring lines together. The first line has length 78 bytes, and latexmk has assumed that the pair of lines is a single long line wrapped to keep it within the length limit on lines in log files.

Why there is a problem only with lualatex: Unlike pdflatex and xelatex, lualatex is quite sloppy in its line wrapping, and latexmk allows for the sloppiness. Unfortunately this has caused a bug where latexmk thinks a line has been wrapped but in reality wasn't, and it wasn't programmed into the current version to deal with this. The use of pythontex accentuates the problem because the filenames it deals with are rather long.

Work-arounds: (a) Change the name of the .tex filename to be slightly longer or shorter. (b) With the TeXLive distribution, set the environment variable max_print_line to a large value, e.g., 10000, to stop significant line wrapping. (c) Run pythontex manually once, then latexmk with the -g option. After that latexmk will work, because it now picks up dependency information from the .fls file.

Next version: The next version (probably 4.81) of latexmk will have a fix for this problem, which is more general than just with pythontex.

John Collins
  • 11,183
  • 3
  • 32
  • 33