0

I have a big issue with project compilation. Instead of "concentrating on getting the right content", I've spend several days on getting the project compiled. My project deadline is near, and I would appreciate any help.

I work with Overleaf and I liked its' interface and it is easy to use. I have a big project. I can compile it by parts, but I can't compile the whole project since Overleaf applies compilation time limitation even for paying customers. I tried all suggested tricks, but it didn't help: Fixing and preventing compile timeouts.

I've made an attempt to compile the project locally on my computer (Windows). I installed TeX Live. Again, I can compile the project by parts, but I can't compile the whole project. I receive the following error:

Unable to read an entire line---bufsize=200000.
Please increase buf_size in texmf.cnf.

I tried to increase the buf_size by running on the Windows cmd:

buf_size=100000 pdflatex main.tex

As proposed in this StackExchange thread

But I get an error:

'buf_size' is not recognized as an internal or external command, operable program or batch file.

Then, I changed texmf.cnf file by adding:

main_memory = 7000000

As proposed in this StackExchange thread. But when I run in cmd:

sudo fmtutil-sys --all

I get the following error:

'sudo' is not recognized as an internal or external command, operable program or batch file.

I have no idea how to compile the project. I worked on it for several months. I will appreciate any assistance.

Best regards, Alex.

F. Pantigny
  • 40,250
ab3001
  • 63
  • 1
    why does your project need such a large bufsize? Do you externalize large graphics? – Ulrike Fischer Jan 28 '24 at 20:55
  • 2
    sudo is a linux command do not use that on windows – David Carlisle Jan 28 '24 at 20:58
  • 2
    is your entire file on one line? why do you need an input buffer of anything like that size? – David Carlisle Jan 28 '24 at 21:00
  • 1
    just use fmtutil-sys --all (no sudo) but first check your tex coding that you are not in an infinite loop that will use up the buffer however large you make it. – David Carlisle Jan 28 '24 at 21:04
  • It is a big project, about 500 pages. Yes I use a lot of graphics, but I am not able to compile it in Oveleaf's draft mode (without graphics). – ab3001 Jan 28 '24 at 21:04
  • 1
    you could normally process a document of thousands of pages with an input buffer of a few hundred characters, why are you producing (after expanding commands) 100000 characters on a single line of input ??? – David Carlisle Jan 28 '24 at 21:06
  • Either way, follow David's comment "just use fmtutil-sys --all" --- post a follow-up if it doesn't work. It appears many people don't use Windows here – user202729 Jan 28 '24 at 21:45
  • I tried fmtutil-sys --all but I get the same error after the compilation. What does it mean "a single line of input"? I think that I am missing something. – ab3001 Jan 28 '24 at 22:04
  • What does "same error" mean? Does the error message change to buf_size=700000 instead? – user202729 Jan 28 '24 at 22:16
  • It is very hard to get a valid tex source to run out of memory. It is much more likely that you have some sort of error. Are you sure you can compile each part? I've compiled a 1000 page book locally with lots of TikZ and not had any trouble (other than the 5 minutes (I think) that I have to wait for each compilation). – Teepeemm Jan 28 '24 at 22:22
  • the error is saying that (effectively) you have a single line of your source file that has 200 thousand characters. That's a very long line. Some expansion constructs need to happen in line so the buffer usage can be longer than the line of the source file but it is always related to a single part of the document the length of the document does not matter, you have an effective line in your source file that is longer than the buffer – David Carlisle Jan 28 '24 at 23:12

1 Answers1

2

You can set the buffer size on the commandline

 pdflatex --cnf-line=buf_size=400000 file

would double the default size of 200000 to 400000.

You can check how much of the buffer is being used by checking a line at the end of the log

 35i,5n,50p,1016b,109s stack positions out of 10000i,1000n,20000p,400000b,200000s

it is the b value here this test used 1016 (already a very abnormally large example created by having 1000 x on a single line)

Note this buffer is the buffer for holding a single line of input not for holding the whole file, it needs to be larger than the actual longest line in the source file, but not usually much larger, you have to be doing something unusual to use 200000 characters in the input buffer.

You may have an ininite loop that is filling this buffer no matter how large you make it. If you comment out parts of the document so that it completes you can check each section and check the b value to see which construct is using more than 200000 input buffer characters.

David Carlisle
  • 757,742
  • Thanks David, it solves the buffer size issue, but now the compilation is stacked with the following message: `[323] [324] No file main.ind. (./main.aux ! Text line contains an invalid character. l.1 ^^@ ^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@^^@...

    ?`

     – ab3001 Jan 29 '24 at 06:16
  • On the other hand it compiled the first 326 pages, and this is a significant progress! – ab3001 Jan 29 '24 at 06:28
  • that is byte 0, ASCII null why have you got nulls in your aux file, but that is easy to fix just delete the aux file – David Carlisle Jan 29 '24 at 08:37
  • oh that probably explains the buffer overrun if you have 200,000 null bytes then tex will see that as a single line with 200 thousand characters and over run the buffer. As said in comments under the question fixing the error makes more sense than increasing the buffer size @ab3001 – David Carlisle Jan 29 '24 at 08:38
  • David, what are "null bytes" and how to find them? I want to find the error and fix it. – ab3001 Jan 29 '24 at 09:43
  • every character in a computer corresponds to a number, but the number 0 is not normally used as a character it is used for special purposes, languages like C for example use position 0 to denote the end of a string. Your .aux file just consists of multiple bytes of 0's so it does not encode text at all: it is an unreadable binary blob, probably due to a system crash corrupting the file, simply delete the file main.aux @ab3001 – David Carlisle Jan 29 '24 at 09:47
  • oh but it's got nulls at the end of the document not the start (the aux file is read twice) if you delete the aux file and this error comes back you have something very strange happening, and you should add more details to your question @ab3001 – David Carlisle Jan 29 '24 at 09:50
  • Removing aux file didn't help. The current status is: The project has several parts in different files that included into the main file using \include command. On my local computer, I can compile each file separately, when I comment out the inclusion of other parts. The compilation output is not good, since some content is simply missing. When I compile all files, I get the same error.

    On Overleaf platform I can compile only is I compile with TeX Live version 2021 (instead of 2023 or 2022) and remove all long tables (longtblr).

     – ab3001 Jan 29 '24 at 19:07
  • you can not write a sequence of null bytes from classic tex you can from luatex but not accidentally, in 99% of times such a sequence comes from a system crash that has trashed the filesystem at that point. do you really get no error before tex tries to read in the aux file and it is full of null? @ab3001 – David Carlisle Jan 29 '24 at 19:33
  • you need to remove the aux file before each test, then gradually move the \end{document} forwards to find the point at which it is being corrupted, then you should be able to show an example and ask people to debug – David Carlisle Jan 29 '24 at 19:37
  • "do you really get no error before tex tries to read in the aux file and it is full of null?" I don't know at which point tex tries to read in the aux file. I am quite new to LaTex. I will try to gradually move the '\end{document}' forwards, as you proposed. – ab3001 Jan 29 '24 at 20:10
  • but you must know if No file main.ind. (./main.aux ! Text line contains an invalid character is the first error reported, – David Carlisle Jan 29 '24 at 20:24