1

I am working on an application where the user learns a topic by interacting with a chatbot. After the completion of the course, the user can generate a personalized notebook using their interaction with the chatbot. The book is generated by compiling user notes/questions using jinja2 to a templated tex files and then proceeding to compile the tex files on the server.

The first course I am releasing is on Python. The problem is: the coursebook is around 465 pages with image pngs, around 90 glossary terms using glossaries package, around 400 exercises using xsim, and even more minted syntax highlighting.

It's taking like 45 minutes to compile just the book without the user notes. I am using latexmk with xelatex to compile the book and using the tufte-book class.

These are some of the things, I have figured out that can possibly reduce the time.

  • Using frozencache of minted exercises
  • Using z 0 compression in the outputdriver option

Any other pointers to reduce the time? Ideally, I would like it to be compiled to under 5 minutes. But this seems impossible now.

Any help at all will be appreciated.

Edit 1

  • Also found the idea of compiling on a ramdisk as shown in the answer interesting. Will try and check it out.

Edit 2

I have finally managed to lower the compilation time from over 40 minutes to 2 minutes.

  1. Replaced xsim with exercise package ( Reduced 20 minutes )
  2. Converted pngs with pdfs ( Reduced 10 minutes )
  3. Replaced minted with listing for syntax highlighting. Although I liked minted much better. The compilation time with the minted package is around 12 minutes.
  4. Ramdisk is an interestingly fast solution but creating a microservice based on it seems complex
Siddharth Kanungo
  • 21
  • Are you running Windows or Linux? Windows seems to be much slower when many files have to be accessed. It is difficult to guess at improvement options without knowing the project structure and running it on ones own computer. Probably you get better answers if you post a link to a sample doucment e.g. on Github. – gernot Jan 21 '21 at 09:24
  • This: https://tex.stackexchange.com/questions/8791/speeding-up-latex-compilation could be useful... or not. – Rmano Jan 21 '21 at 09:29
  • @gernot I have ubuntu installed on an SSD. I am having difficulty creating an MWE given the problem that occurs when the document is particularly large. I was looking for some general tips/best practices to optimize compile time. – Siddharth Kanungo Jan 21 '21 at 10:31
  • @Rmano I had looked at it earlier. It is pretty useful, but most of the answers tend towards faster rebuilds during editing. I am looking for a faster way to compile final versions of large latex files. – Siddharth Kanungo Jan 21 '21 at 10:37
  • It might be an idea to have different versions of the PNGs, such that you can use lowres for when you work and highres for the final version. – daleif Jan 21 '21 at 10:49
  • @daleif I don't have any issues with rebuild times while editing. I want to reduce the time to compile the final build as it will be for users. – Siddharth Kanungo Jan 21 '21 at 10:52
  • Ok, I did not read all the question. I don't think it is a good idea to compile this each and every time, that seems wasteful. – daleif Jan 21 '21 at 10:56
  • @daleif I am trying to embed user notes inside the book itself. To create personalized books. So, each build will contain around 5% of user content. I agree that the process seems wasteful but this is the only way I can think. – Siddharth Kanungo Jan 21 '21 at 11:01
  • Precompile your PNGs into PDFs or EPSs (PNG takes long to be included into a PDF by LaTeX). Also, if the user notes are on separate sheets of paper you can just compile those and merge them into the book. – Skillmon Jan 21 '21 at 11:06
  • If you have any TikZ or pstricks pictures, precompile them using standalone and only include the resulting PDF. – Skillmon Jan 21 '21 at 11:08
  • @Skillmon Great suggestion. I will convert the PNGs into PDFs. Unfortunately, user notes are intimately embedded in the book. They are littered everywhere inside a topic, cannot merge them as separate. – Siddharth Kanungo Jan 21 '21 at 11:09
  • If you have recurring materials (e.g., a logo that is placed on each page), box them up and only use the boxes (maybe with xsavebox). – Skillmon Jan 21 '21 at 11:09
  • If you can make it work without XeLaTeX (because you don't need system fonts, for example) consider using pdfLaTeX (which is generally faster). – Skillmon Jan 21 '21 at 11:11
  • @Skillmon Don't have anything recurring like a logo. – Siddharth Kanungo Jan 21 '21 at 11:11
  • @Skillmon I will compare performance between pdfLatex and XeLaTeX and update the question. Thanks, great suggestion. – Siddharth Kanungo Jan 21 '21 at 11:14
  • I'm afraid even with all these small improvements you won't get the total time down to 5min (as long as you don't have an absurd number of PNG image files). – Skillmon Jan 21 '21 at 11:15
  • @Skillmon I am having the same feeling. – Siddharth Kanungo Jan 21 '21 at 11:18
  • Actually there is something you can do. Note that PDF does not directly support PNG, thus AFAIR latex will convert the PNG into the internal PDF image format. If you can do that PNG conversion in advance, then that will cut out all the auto convertions (that are not saved from compilation to compilation) and it will be a lot faster. – daleif Jan 21 '21 at 11:31
  • @daleif Yeah, I am trying to do that. Thanks ! – Siddharth Kanungo Jan 21 '21 at 11:42
  • I made a small example with a screen shot of my desktop (three monitors so large PNG). If I include and scale the PDF to 0.9\textwidth you can see the compilation (pdflatex) stop at the PNG to convert it. If I convert it to PDF (here I used convert from imagemagick, but the sam2p tool from CTAN is probably better) and use that, it does not stop at the image. – daleif Jan 21 '21 at 11:42
  • @daleif I was trying to convert using ImageMagick. I will try using sam2p now. – Siddharth Kanungo Jan 21 '21 at 11:44
  • convert file.png file.pdf should work. But sam2p creates much smaller images. But it might need manual compiling as it is not distributed anymore. I would hope they include it in TeXLive in the future. – daleif Jan 21 '21 at 12:05
  • do the user notes affect the pagination? If you could arrange that generally they do not then you could include the bulk of the book from a pre-typeset PDF and just regenerate the pages you need. – David Carlisle Jan 21 '21 at 20:36
  • @DavidCarlisle They do affect pagination. I have thought about separating the user notes from the book, but it wouldn't be what I have in mind.

    I guess I have to make peace with the slower compilation and notifying users about their generated personalized book via email.

     – Siddharth Kanungo Jan 22 '21 at 06:10
  • Which version of xsim were you using? – cgnieder Jan 22 '21 at 13:20
  • @cgnieder This one Package: xsim 2020/01/16 v0.16a eXercise Sheets IMproved. Does the latest package compile faster? – Siddharth Kanungo Jan 22 '21 at 13:25
  • @Siddarth yes the performance was improved. v0.19 should compile faster as yours. Probably exercise will still be faster, though. You probably shouldn't update a single package in the middle of a project. That can cause more troubles than it solves – cgnieder Jan 22 '21 at 15:29
  • @cgnieder I updated TexLive to 2020 and then used the latest version. The first time run is around 30 minutes and the second time is around 10 minutes. I think that's manageable for now. Thanks a lot for creating tasks and xsim. – Siddharth Kanungo Jan 23 '21 at 12:07

0 Answers0