Has the time come for a LaTeX3 successor to 'listings'?

Question

I'm currently looking for a nice and challenging open-source project to work/collaborate on after my thesis. I don't have a Comp. Sci. background, but I'd like to learn more about compiler construction and I also would like to improve my TeX skills, so I figured: Two birds, one stone.

I was thinking of a LaTeX package that would allow for more advanced syntax highlighting of listings than is currently offered by the listings package. I use the latter a lot and find it mostly pretty good, but I find it also very frustrating for some more advanced stuff. Implementing proper syntax highlighting of a language with a context-sensitive grammar is really a pain in the neck, in listings. Also, Unicode isn't supported "out of the box".

Sure, we've got minted, verbments, pythontex. Those packages successfully unleashed the power of Python and Pygments in the world of TeX; however, customisation of the syntax highlighting (colour, etc.) involves writing/customising a Pygments lexer; in other words, it requires some Python coding outside the .tex file. Wouldn't it be nice if everything could be done without using -shell-escape? Or is there no point in trying to replicate Pygments in TeX?

Arguably, more powerful syntax highlighting seems more within reach, with a combination of fancyvrb and powerful LaTeX3 packages such as l3regex.

1. Am I just fooling myself? Is there even a point in such a project? Or should we be content to use existing tools (listings, minted, etc.)?
2. Without turning this into a biglist, what, if anything, do you find frustrating about listings? What would be on your wishlist for a hypothetical new package meant for typesetting source code?
3. Would anybody interested in collaborating on such a project please stand up? Anybody? Hello...?
4. What limitations of l3regex should I know about before deciding to use it for such a package?

Please do chime in below...

Answer 1

I don't think syntax highlighting should be done purely in Tex for following reasons:

• Languages are complicated (context sensitive) and to proper highlight the syntax, you have to implement a full compiler front-end. As an example, here is some simple C++ code:

  struct foo; // foo is the name of a data type
  void f(foo param); // function f, taking a foo as parameter
  
  f(foo()); // 1 \color{type}{f}(\color{type}{Foo});
  foo foo;  // 2 \color{type}{foo} \color{variable}{foo};
  f(foo()); // 3 \color{type}{f}(\color{variable}{foo});
  

  A good highlighter needs to know whether foo is a class name or the name of a variable. To decide that, you need to know the context. This example may seem a bit artificial, but this kind of highlighting problem occurres all over the place in normal code and proper highlighting can help to understand the code and wrong highlighting confuses the reader.

  Good highlighting is something people really want, see for example this question for C# where types and non-types should have different colors.

  To solve the issue, you have to keep track of all declarations (and even more in many languages (for example C++)). Basically, you have to implement a whole compiler front-end and I'm not sure, if that can be done efficiently in TeX. To give you an example, only 5 compilers so far understand the grammar of the recent version of the C++ language (released 2011). And for a good highlighting, you have to do that.

• With a heuristic approach, you can provide a package that highlights code kinda ok, but this is what pygments already does, so there's not really point in copying that. Especially if things get worse, I don't think someone is going to change from minted to a pure LaTeX package. You said that having TeX-only would allow better customization. I don't think so, as color changes can be done in e.g. minted equally easy and more in depth changes are not so common and should be easier in python for many people that know programming but not TeX (and these are likely to include code in their documents).

• There are many excellent syntax highlighters out there for every language. But they usually only provide IDE highlighting, not a LaTeX backend. For C++, one could easily do one with clang and I'm sure that's possible for other languages too. There's no point in rewriting them in TeX!

• What I think needs to be addressed is a way to communicate with other programs. -shell-escape is not the best way of handling this. IMHO, installed packages should have the right to perform shell-escape whenever they want to. Because I installed them and I trust the people that maintain them. You can't exploit the fact that minted executes pygmentize (assuming there are no security holes). There is nothing wrong in formatting code with external tools.

• External tools are faster than TeX, even tough interfacing with them is solved poorly in teX. As far as I know, caching the output can't be done effectively in LaTeX yet. This is something that should to be done IMO.

To sum up, I don't think you can provide a high quality highlighter in pure TeX. And regarding the effort to write one for occasional uses where quality doesn't matter, I don't think it's worth it.

Answer 2

Limitations of l3regex

On page 12 of the documentation, under "The following features of PCRE or Perl will definitely not be implemented":

• Recursion: this is a non-regular feature.
• Back-references: non-regular feature, this requires backtracking, which is prohibitively slow.

So it doesn't look like you will be able to track matched/unmatched parentheses using l3regex alone (at least to an arbitrary depth), and some other things will be complicated without additional tools.

Ultimately, the question may be, "Are the extra things we can easily do with l3regex worth the effort, given that we won't be able to match Pygments (at least not without a lot of work...and even if we could, it might be too slow)?"

State of syntax highlighting

As the author of pythontex and the new maintainer of minted, here are my thoughts on the state of syntax highlighting.

• The main disadvantage of tools that use Python, in my view, has been performance. They either require two compiles (pythontex), or are slow for one compile (minted). But I've added caching to the development version of minted, so I think that is solved.
• A second disadvantage of minted is the potential security issues of using \write18. Maybe someone can figure out a way to have things like Pygments added to a whitelist of sorts. (pythontex doesn't use \write18 due to the way it uses two compiles with a Python script run in between, so it could be secure for highlighting. But I haven't tried to make it secure, because it's made for executing Python code, and syntax highlighting is just there for convenience.)
• I don't see any reason that a TeX interface for customizing a Pygments lexer couldn't be created, particularly if the power of LuaTeX is invoked. I expect that settings could be collected on the TeX side, and then automatically plugged into a Python template that customizes the Pygments lexer. (That being said, I don't have any plans to attempt this, at least in the near future, due to time constraints.)
• The thing I would really like to see for syntax highlighting is an updated, easily customized version of fancyvrb. It would be a nice basis for building future syntax highlighting packages. In particular, the following features would be useful.
  • Built-in support for Unicode (patch VerbatimOut to use \detokenize, etc.)
  • Support for automatic breaking of long lines.
  • Error checking for font-related issues (tildes can be raised like a superscript due to font issues, backticks can require upquote, etc.)
  • Built-in support for creating an environment with a custom name and then automatically numbering the lines for such environments consecutively, so that, for example, all Python code uses one numbering while all C code uses another.
  • Default setup that plays nicely with framing packages like framed, mdframed, tcolorbox, etc. Or at least a setup that is made to work with one of them really well.
  • Built-in macros for creating styles.

Answer 3

I am, in general, relatively happy with listings. In fact, listings was the reason that made me switch from Word to LyX and then to LaTeX.

I also have the feeling that, over the years, it has become a pretty well engineered piece of code that provides a great interface and good extensibility. When I asked my question How to automatically skip leading white spaces in listings, I was quite astonished about the elegance and simplicity to integrate such a feature in the answer of Martin Scharrer, which eventually became part of his lstaddons bundle.

• To put a long story short: There are many, many things just done right in listings, such as the interface. Lets keep them!

• What I miss most is (besides proper unicode support, but so far I could always work around that with the literate trick) is a better interface to beamer overlays. I mostly use listings for CS lectures these days. While I have, over the years, developed quite some idioms and tricks in this respect (such as here, here,and here) I still consider them as workarounds.

• I do see, however, the point regarding context-sensitive languages and I also think there is little hope in solving this problem in general using l3regex and other advanced packages.

• I am not a big friend of minted and other packages that rely on external programs. Besides the compilation-time issue (which can become quite big if you use them in beamer frames with dozens of overlays) and the general security concerns induced by --shell-escape there is always the problem of external dependencies: Especially in the Windows world, one can not assume that users have a proper python installation. This renders minted and similar packages practically unusable in larger collaborative settings.

So I think the way to go is to (a) use a real programming language for lexing and parsing, (b) but use one that is built-in: Lua!

• Basically all TeX distributions today also contain lualatex, so even in the Windows world (or in collaborative online editors, such as WriteLatex) they are available. Also all common IDEs provide built-in support to use lualatex as engine.

• The Lua language in lualatex provides a good interface to the TeX world, both can interact with each other on a much finer granularity than with external programs.

• lualatex provides built-in support for unicode.

• Even though most users (including me) still employ pdflatex, I am convinced that the transition to lualatex is already there. The upcoming TikZ version with its Lua-based graph layout engine will certainly be an accelerator in this respect.

So my suggestion would be: Take the existing listings package and extend it so that language-specific scanners and parsers could be written in Lua.

Ideally, a first version of this package would remain compatible to listings and require lualatex only for advanced features or complicated. This would allow for a lot of reuse and foster the transition. Later versions then may generally depend on lualatex, if that is necessary or significantly eases the implementation.

Answer 4

I have written codehigh package and have submitted it to CTAN: https://ctan.org/pkg/codehigh.

This package uses l3regex to parse and highlight source code and demos. It is more powerful than listings package, also easier to use than minted package.

\begin{codehigh}[language=latex/latex3]
\cs_new_protected:Npn \__codehigh_typeset_demo:
  {
    \__codehigh_build_code:
    \__codehigh_build_demo:
    \dim_set:Nn \l_tmpa_dim { \box_wd:N \g__codehigh_code_box }
    \dim_set:Nn \l_tmpb_dim { \box_wd:N \g__codehigh_demo_box }
    \par\addvspace{0.5em}\noindent
    % more code
  }
\end{codehigh}

In this example, you can see that private functions/variables and public functions/variables in LaTeX3 are highlighted with different colors.

But using l3regex makes this package slower than listings and minted. Therefore in LuaTeX the package provides another way to highlight code: using LPeg (Parsing Expression Grammars for Lua). LPeg is much more powerful and faster than l3regex.

With l3regex the source code to define a language is like this:

\AddCodeHighRule[latex/latex3]{1}{Argument}  {\#+\d}
\AddCodeHighRule[latex/latex3]{2}{PrivateVar}{\\[cgl]__[A-Za-z_:@]+}
\AddCodeHighRule[latex/latex3]{5}{PrivateFun}{\\__[A-Za-z_:@]+}
\AddCodeHighRule[latex/latex3]{4}{PublicVar} {\\[cgl]_[A-Za-z_:@]+}
\AddCodeHighRule[latex/latex3]{6}{PublicFun} {\\[A-Za-z_:@]+}
\AddCodeHighRule[latex/latex3]{8}{Brace}     {[\{\}]}
\AddCodeHighRule[latex/latex3]{3}{Bracket}   {[\[\]]}
\AddCodeHighRule[latex/latex3]{9}{Comment}   {\%.*?\r}

With LPeg the source code to define a language is like this:

language["latex/latex3"] =
  {
    {1, "Argument",   P"#" ^ 1 * digit},
    {2, "PrivateVar", P"\\" * S"cgl" * P"__" * (alpha + S"_:@") ^ 1},
    {5, "PrivateFun", P"\\" * P"__" * (alpha + S"_:@") ^ 1},
    {4, "PublicVar",  P"\\" * S"cgl" * P"_" * (alpha + S"_:@") ^ 1},
    {6, "PublicFun",  P"\\" * (alpha + S"_:@") ^ 1},
    {8, "Brace",      S"{}"},
    {3, "Bracket",    S"[]"},
    {9, "Comment",    P"%" * (P(1) - S"\r\n") ^ 0 * (S"\r\n" + -1)},
  }

And the source code to define a theme is like this:

\SetCodeHighStyle[default]{0}{black}
\SetCodeHighStyle[default]{1}{brown3}
\SetCodeHighStyle[default]{2}{yellow3}
\SetCodeHighStyle[default]{3}{olive3}
\SetCodeHighStyle[default]{4}{teal3}
\SetCodeHighStyle[default]{5}{azure3}
\SetCodeHighStyle[default]{6}{blue3}
\SetCodeHighStyle[default]{7}{violet3}
\SetCodeHighStyle[default]{8}{purple3}
\SetCodeHighStyle[default]{9}{gray3}

The package also provides demohigh environment for typesetting code and result at the same time, and \dochighinput command for inputting and highlighting source file.

Answer 5

As teknokrat said above, it would be difficult to write a C or C++ processor in TeX. However, you could run your program through the libclang C library. This is a library which uses the clang C/C++ compiler to compile source files. The result is a set of pointers into the text called cursors. These cursors have much more information about the source code than anything TeX could ever extract, because it comes from the real compiler. Doing listings would be a fairly trivial application of this. It would also be a reasonably good introduction to a production quality compiler.