20

I've been looking into speeding up my pdflatex compilation time and it's lighting fast now.

Only one thing's still very slow: compiling source code using the minted package.

I was wondering if there is no way of speeding this up (perhaps like the tikz externalization \tikzexternalize that only needs to run once and from then on includes the result).

I was thinking of making a new environment that puts the minted environment inside a tikz environment, so that the rules of the \tikzexternalize may apply, but I haven't really figured out how to do this exactly (simply putting it inside a tike node doesn't work).

jub0bs
  • 58,916
romeovs
  • 9,102
  • 3
    For the ConTeXt package t-vim (which is like minted but uses vim editor rather than pygments, though I might support pygments in the future), I create a md5 sum of each external file, and re-run vim only if the external file has changed. Perhaps you can use the same approach in minted. Instead of calling pygmentize --options call mtxrun --ifchanged <filename> --direct pygmentize --options, and mtxrun will do all the book keeping. – Aditya Jun 07 '11 at 15:17
  • @Aditya Yes seems like a smart thing to do.. yet, how would I go about implementing this into a document? I'm not planning on changing my minted.sty file. Perhaps I'll make a request to Konrad Rudolph that he implement this as an option in the next minted version. – romeovs Jun 17 '11 at 13:24
  • Adding an option to minted should be straight forward. @Konard Rudolph: Can you look into this? – Aditya Jun 17 '11 at 18:39
  • @Aditya: I don't think that Konrad Rudolph got notified. IIRC, you can't notify someone who is not participating in the discussion. You can do that, though, on chat. – Bruno Le Floch Oct 23 '11 at 04:07
  • I’ve just now noticed the discussion … my Google Alert must be broken. I’ll look into it when I get time (not likely soon, I’m afraid). Aditya’s approach looks like the proper solution. This could speed up compilation considerably. For reference, this is mentioned in an issue on the minted development website. – Konrad Rudolph Nov 06 '11 at 13:02
  • @KonradRudolph: I added some details in my answer below. Some more details are in the Dealing with slow filters section in the documentation of t-filter. – Aditya Dec 04 '11 at 22:30
  • Since you have some responses below that seem to answer your question, please consider marking one of them as ‘Accepted’ by clicking on the tickmark below their vote count (see How do you accept an answer?). This shows which answer helped you most, and it assigns reputation points to the author of the answer (and to you!). It's part of this site's idea to identify good questions and answers through upvotes and acceptance of answers. – jub0bs Jan 17 '14 at 17:35

5 Answers5

10

I am the author of the ConTeXt module t-vim which is similar to minted but uses vim rather than pygmentize to generate syntax highlighting. The t-vim module actually delegates the task of running external programs to t-filter module, which is provides the necessary pluming to call external programs on the content of an environment.

By default, the t-filter module behaves in the same manner as minted package: it writes the contents of the environment to an external file, calls the external program, and inputs the result back to TeX. However, to deal with slow external program, the filter module provides a continue=yes option. When this option is enabled, the content of each file is written to a separate file and the md5 sum of each file is calculated. The external filter is run only if the md5 sum is changed.

In MkII, this feature is enabled by calling the external program using 

 \doifmode{*first}
    {\executeexternalcommand{mtxrun --ifchanged=\inputfile \externalprogram}}

This calls mtxrun, the wrapper script for ConTeXt, which calculates the md5 sum of the file (and stores it as filename.md5) and the the program only if the md5 sum is stored. This is faster than running vim, but still slow as a new process (mtxrun) must be executed. To speed things up, I wrap the entire command in a \doifmode{*first} so that mtxrun is called only during the first run of a multi-run compilation.

To speed up things further, in MkIV, I use the ConTeXt lua function job.files.run, which stores the md5 in the tuc file (similar to aux file in LaTeX). So the call to the external program is roughly equal to

 \ctxlua{job.files.run("\inputfile", "\externalprogram")}

The same method can, in principle, be implemented in minted. In fact, the mtxrun --ifchanged method can be incorporated easily, provided that minted writes each environment in a separate file (currently it does not do that).

Aditya
  • 62,301
  • 3
    I tried the method, but it doesn't work because of an error in 2context.vim: on line 42 exsists should be exists, at least in my (updated) TeX Live 2011. – egreg Dec 23 '11 at 14:05
  • 2
    @egreg: Thanks. I corrected the typo and made a new release (which is also available on the github repo). – Aditya Dec 23 '11 at 14:31
10

In the document I'm editing right now, I have about twenty code listings, which results in a very slow compilation (about 10 seconds).

However, if I remove these lines from the preamble :

\usepackage{minted}
\newminted{c}{}

and I add these instead:

\usepackage{fancyvrb}
\DefineVerbatimEnvironment{ccode}{Verbatim}{}

then compilation time drops below one second, which is really acceptable. And I get pretty much the same output (though obviously without the colors :-) because the \fvset "directive" needs not be changed.

I wonder if this kind of trick could be integrated into minted itself, maybe in the form of a draft option for instance ? (just like packages like graphicx or beamer honor the draft option to speed up compilation)

EDIT: I've just realized that I could also do the automation myself, by using the ifdraft package (thank you tex.SE !). My preamble now contains this snippet :

\usepackage{ifdraft}
\ifdraft{
  \usepackage{fancyvrb}
  \DefineVerbatimEnvironment{ccode}{Verbatim}{}
}{
  \usepackage{minted} % syntax coloring. 
  \newminted{c}{}
}

This way, I can now use my good-old draft documentclass option to switch between fast-but-imperfect compilation and slow-but-final compilation.

Gyom
  • 743
  • 1
    FWIW, t-vim (actually t-filter) provides this feature. If one says \setupvimtyping[state=stop], old processed files are used. One can selectively disable one class using \setupvimtyping[Ctyping][state=stop] or one code snippet \startCtyping[state=stop]...\stopCtyping. I think that rather than a global draft option, a similar, per class and per code snippet level control is better. This way, one can globally set state=stop and, if some code snippet changes, set state=start for that snippet. (With the lua version, the overhead of md5 check is small, so such tricks are not needed). – Aditya Dec 23 '11 at 13:23
  • @Aditya, the t-filter has not only the overhead of the md5 check, but also the reading and writing to the external files, which should make this approach considerably faster than your solution. – schlamar May 15 '12 at 13:34
  • @ms4py: you are comparing apples to oranges. With the above approach, one gets no syntax highlighting. With caching, one gets syntax highlighting with the overhead of reading and writing to a file. I typically compile big documents with around 50 code snippet with no noticable speed difference from just compiling such a big document. Did you do some timing tests to measure the overhead of writing to, and reading from external filess. In principle, one can donhe md5 check before writing the file and save on that as well, but I think that the savings are not worth the complexity. – Aditya May 15 '12 at 15:08
  • @Aditya You started the comparison ;-) IMO the new explanation was missing in your first comment. – schlamar May 16 '12 at 06:29
  • calm down you guys :-) I have no idea what T-vim etc are in the first place. As for myself, I'm fine with not getting a "final" document when I say "draft", just as I'm happy with the way graphicx handles "draft". – Gyom May 18 '12 at 14:32
5

I am currently maintaining minted, and have just added caching support in the v2.0alpha release. Initial tests indicate that speedups of at least 5x are possible.

Under LuaTeX and pdfTeX, I'm using \pdf@mdfivesum from pdftexcmds to do the hashing. Under XeTeX I'm using a Python script executed via \write18.

G. Poore
  • 12,417
4

My PythonTeX package is another possible alternative when speed is an issue but Pygments is needed. It saves all code to an external file during the first LaTeX pass, then highlights all modified code using Pygments (only using two processes), and then brings in the highlighted results in a second LaTeX pass. All highlighted results are saved in a single external file that is inputted in the TeX document, so you only need to run the Python side of things when code has changed. PythonTeX also has a package-level option to switch off Pygments highlighting and use fancyvrb instead, in the event that highlighting ever does get slow.

As an added benefit, if you are highlighting Python code, PythonTeX can execute your code and bring in the output.

G. Poore
  • 12,417
  • 2
    I answered this question, and in the process did some informal benchmarking of pythontex vs. minted. Summary: on my system, 40 simple uses of \mint take about 12 s to compile, and 80 uses needs about 23 s. With pythontex, 40 equivalent commands need 3.5 s, 80 uses is almost the same, and you only get to around 23 s for 8000 uses. All the pythontex numbers are actually overestimates for typical use, since highlighted results are cached. – G. Poore Nov 21 '12 at 22:02
2

I just ran into the same problem with the minted package. Taking up Aditya's caching approach I came up with a pygmentize wrapper (still really ugly, but working for me).

Maybe it is of use to someone.

Community
  • 1
michi_h57
  • 21