22

I seem to often run into capacity issues and have been upping the limits each time. There are several question on this site pertaining to these. However, there seems to be some conflicting information and no single place where all these limits are detailed.

Main Question:

So, what I am looking for is

  1. Name of the variable
  2. Default value.
  3. Absolute max that it can be set to.
  4. What steps can I take to minimize these usages. What are the DOs and DONTs for each these capacity limit.

The settings I am aware of are the ones reported at the end of the log file:

 106477 strings out of 493042
 2524840 string characters out of 39884661
 12363848 words of memory out of 12435455
 103471 multiletter control sequences out of 15000+600000
 197845 words of font info for 228 fonts, out of 8000000 for 9000
 1143 hyphenation exceptions out of 8191
 86i,17n,86p,10523b,3398s stack positions out of 5000i,500n,10000p,200000b,80000s

but there may be others that are not listed here. My immediate issue is with words of memory as the numbers above are from the first run of my indexing. If I attempt to do a second run then I get TeX Capacity Exceeded.

Examples of Other Variables:

From Davis Carlisle's comments, there is also extra_mem_top and extra_mem_bot that can also be used. But, does this allow for larger words of memory, or are these related to one of other capacity limits in the report. Again, what is the maximum I can set these to?

Possible DOs and DONTs:

Some of the questions I have in regards to minimizing the usage are listed here, so perhaps these can be covered in the DOs and DONTs for each of the variables.

  1. Is it better to use \def\foo{xxx} instead of \newcommand*{\foo}{xxx}. I can perform the check for duplicate macro names outside of TeX.

  2. Do macro names with a large number of characters increase usage of some of these?

    For instance, does \expandafter\newcommand\csname dir1/dir1/long-file-name\endcsname{xxxx} have any issues in regrads to capacity. Would it be better for me to map dir1/dir1/long-file-name to some shorter string such as ZXyw to save on these limits. Yes, readability is lost, but I can automate this replacement.

  3. Does repeated text result in increased use of the limits: Which of the following would be better:

    \MyMacro{../../../../dirA/dirB/dirC/dirD/file0001.tex}
\MyMacro{../../../../dirA/dirB/dirC/dirD/file0002.tex}
 ...
\MyMacro{../../../../dirA/dirB/dirC/dirD/file5000.tex}

    or

    \def\Path{../../../../dirA/dirB/dirC/dirD/}
\MyMacro{\Path/file0001.tex}
\MyMacro{\Path/file0002.tex}
 ...
\MyMacro{\Path/file5000.tex}

  4. Does \undefing macros after they are no longer needed help with these limits?

Certainly, I can experiment with these things that I have thought of, but perhaps there are other things that can be done to reduce the useage of these limited string and words.

Example of Inconsistency

Notes:

  • I am assuming that any changes would be made to the file reported by kpsewhich texmf.cnf as per Leo Liu's answer at How to expand TeX's “main memory size”? (pgfplots memory overload), which in my case is

    /usr/local/texlive/2015/texmf.cnf

    If there are different ways to set some of these I would like to know those as well.

  • My main problem right now is with indexing a large number of entries and the words of memory limit. I am less concerned about run time (within reason), more about exceeding these limits.
  • I am using TeXLive2015 in case it matters. But if the limits are dependent on the distribution it would be good to know that as well.
  • This question is not intended to be about error conditions that result in exceeding the capacity limits (such as my naive question Tex Capacity Exceeded (if remove % after use of macro)).

References:

Community
  • 1
Peter Grill
  • 223,288
  • \expandafter\newcommand*\csname… will expand * rather than \csname. – Henri Menke Feb 11 '16 at 09:43
  • @HenriMenke: opps, have corrected that. – Peter Grill Feb 11 '16 at 09:45
  • I think the 8000000 refers to main_memory but you can up extra_mem_top and extra_mem_bot so the effective memory is beyond that. (or use luatex which uses dynamic allocation) – David Carlisle Feb 11 '16 at 11:08
  • @DavidCarlisle: But the question is what is extra_mem_top and extra_mem_bot? It isn't obvious to me. What is the difference between these extra memory versus the main_memory. What are the maximum values I can set these two. What will increasing these allow me to do? – Peter Grill Feb 11 '16 at 11:55
  • @PeterGrill I think it's just a way of not changing too much of the original pascal tex.web memory layout. Externally in macro usage it's no difference but internally it's an escape hook to allow more "main" memory without having to change the memory layout of everything else to fit it in. Increasing them allows you to have more main memory (beyond the 8000000) – David Carlisle Feb 11 '16 at 12:07
  • @DavidCarlisle: So can I use those to increase words of memory beyond 12435455? – Peter Grill Feb 11 '16 at 12:15
  • 1
    When thinking about this complete overflow topic: Instead of starting a complete summary, would it be a better approach to create 1 question with answers for only one variable or a small set of variables if they are handling the same TeX internal thing or for the TeX (Knuth TeX, e-TeX, pdfTeX, XeTeX, LuaTeX engine) overflow error message? Adding a small sample code to trigger the error, description of cause, constants for the different engines, how to avoid overflow concepts? – Bernd Raichle Jan 09 '23 at 08:14
  • @BerndRaichle: Yeah, I agree that that is a better approach. – Peter Grill Jan 09 '23 at 08:17
  • mem_bot and mem_top are INITEX only special constants for the dumped pre-loaded format file - in the old times when INITEX and VIRTEX were two different executables. IMHO most (or all?) TeX systems in use have both integrated in one executable and then these two variables need to be set to mem_min and mem_max (or the implementation handles the main_memory in a different way replacing these TeX.web parts in their used change file). – Bernd Raichle Jan 09 '23 at 08:21

2 Answers2

6

Is it better to use \def\foo{xxx} instead of \newcommand*{\foo}{xxx}. I can perform the check for duplicate macro names outside of TeX.

No, comes to the same thing

For instance, does \expandafter\newcommand\csname dir1/dir1/long-file-name\endcsname{xxxx} have any issues in regards to capacity.

in theory yes, the csname has to be stored somewhere so this eats in to 2524840 string characters out of 39884661

but you are presumably only going to make it 20 or so characters less so not much difference out of 39884661

Which of the following would be better:

again marginally the first uses more as it defines one more csname (\path) which takes one out of 

 103471 multiletter control sequences out of 15000+600000

but you are nowhere near that limit

Does \undefing macros after they are no longer needed help with these limits?

yes that's why latex removes lots of commands at \begin{document} the hash entry is not reclaimed (number of multi-letter commands) but the main memory used to save the definition is reclaimed.

David Carlisle
  • 757,742
  • That is useful info, but does't address the main thrust of the question of what are the absolute maximums and what programming practices to follow to keep their usage down. Also, my particular use case makes tens of thousands of calls to\index so saving 20 here and there helps. Am experimenting now, and replacing the long paths in with macros reduces the words of memory from 12,400,429 to 11,458,517. – Peter Grill Feb 11 '16 at 11:16
  • 1
    @PeterGrill also I realise i misread long macro names for \long macro names So I'll delete one "no" and just leave the following quote. Not sure what to say about the actual limits they are all in texmf.cnf – David Carlisle Feb 11 '16 at 11:23
  • @PeterGrill any reduction in main mem from using a macro in the argument to MyMacro is related to whatever that macro is doing not to the call shown. if you for example you save that argument in another definition, that definition will have a smaller replacement text so use less memory. If you just use the argument for typesetting or writing to a file, the fact that the argument was passed as a macro should make no difference. Indexing is normally just a matter of writing to a file and shouldn't use that much memory at all? – David Carlisle Feb 11 '16 at 11:29
  • The above results are just from indexing -- nothing else. My indexing has a lot of formatting applied so that is probably what is causing the problem. And the reduction almost 1 million words of memory is strictly from replacing repeated text with a macro -- exactly as shown in the question. No other there differences in terms of what the index looks like. So, my results seem to be contradicting your last comment -- perhaps I am not understanding the last comment. – Peter Grill Feb 11 '16 at 11:49
  • Also, I don't see where in the absolute MAX settings of these variables are defined in the texmf.cnf. – Peter Grill Feb 11 '16 at 11:53
  • @PeterGrill my assumption is that you are just limited by the size of integer and/or your physical memory unless otherwise stated. (don't have time to test now though:-) – David Carlisle Feb 11 '16 at 11:57
  • @PeterGrill replacing a string of characters by a macro in itself increases main memory (as the macro definition has to be stored) I an sure you are correct that in your usage memory is going down, but that just means there is some code you haven't described that is using the macro. \def\zz{\foo} takes less memory to store \zz than \def\zz{abcdefg} although the total memory to store \zz and \foo is higher. So I can only assume you are making a macro and then using it in lots of definitions. (just using it in lots of macro arguments doesn't count) – David Carlisle Feb 11 '16 at 12:01
  • Yes you are correct, I am replacing a string that is used in macro definitions many times with a macro. So the numerous definitions such as \newcommand{\foo}{some long path/XX} was modified to use \Path defined by \def\Path{some long path}. Sorry for not clarifying this earlier, but I didn't know that it mattered. Since \foo may be used 5-10 times I thought it better to define a macro. If I understood your comments correctly, perhaps I should skip defining \foo and \Path and just pass some long path/XX to every invocation of \index? – Peter Grill Feb 11 '16 at 12:13
  • @PeterGrill I would expect "passing stuff straight to index" would save essentially all your memory use. – David Carlisle Feb 11 '16 at 12:27
  • My experiments seem to indicate that passing "stuff" straight to indexing is making things worse. I think it is because the entire index needs to be read in so that sorting can take place and that runs into issues earlier. – Peter Grill Feb 12 '16 at 05:53
  • @PeterGrill possibly. Profiling unseen code is not a 100% accurate operation:-) – David Carlisle Feb 12 '16 at 07:48
3

Many years ago I gave a talk about "TeX capacity exceeded" problems with some background information, root causes and possibilities how to avoid them.

The first part of the talk was published in "Die TeXnische Komödie" issue 1995/3, pages 22ff.

Page 24 has a table with all (standard TeX.web) overflow error messages, the assciated TeX.web parameters and the parameter settings.

Negative: (1) the talk and the paper are in german only, (2) I had never enough time to finalize the article series including all remaining topics in a 2nd and 3rd part, and (3) it is based on standard TeX.web resp. the web2c "bigTeX" version, without e-TeX, pdfTeX, XeTeX, LuaTeX and similar extended variants.

Bernd Raichle
  • 469