5

The official repository of LuaTeX seems to be located on a private GitLab site, here. It doesn't appear to allow contributions, not even in the form of defect reports or similar.

While this is unconventional for a FLOSS project -- and per its license it is one -- and I appreciate and comprehend the remarks from the FAQ I was wondering about these items in particular:

  1. Several of the points mentioned around the topic of contributions and even user feedback appear out of date (both sockets and UTF-8 no longer seem to be a concern; ancient versions seem to be referenced; trunk ist referenced instead of the master default branch).
  2. Having specific plans and a specific design doesn't generally preclude outside contributions. The only thing it does is to provide a boundary within which contributions should (?) be welcome.
  3. Small team of decision-makers doesn't preclude outside contributors or contributions. I've even heard of instances where FLOSS projects gained new core team members by taking in contributions and promoting regular high-quality contributors.

Considering that the upstream project -- as per their own statements -- doesn't seem to be inclined to take any outside contributions, is there any way to contribute at all?

In particular I imagine the TeXlive project could be willing to maintain patches rebased against the upstream (just a guess with lots of wishful thinking).

0xC0000022L
  • 985
  • 1
    I repeat my comment from meta: "It is nice that you want to help, maybe asking on the luatex-dev list if there is anything to help with is the best thing to do?" After all, there you will also reach the people actually involved. – mickep May 09 '23 at 10:55
  • @mickep: thanks for the comment! I noticed it also on meta and will turn there once I have something tangible to contribute. Makes little sense to approach someone saying "Hey, I have this brilliant idea. But you gotta implement it". Also I don't just need to get this into a shape to be "presentable" but also to think about it from all sides and consider potential issues such as backwards compatibility concerns. – 0xC0000022L May 09 '23 at 12:06
  • 1
    (In fact, are you sure that luatex is the right project to contribute to? There is maybe more work going on with tagging. I don't know if you are into math, but many people would love to see more work on the unicode-math package. Just to give a few directions...) – mickep May 09 '23 at 13:20

1 Answers1

3

(Mostly copied from my answer on Meta. And sorry, this answer is a little long/rambly.)

Responses to the question

The official repository of LuaTeX seems to be located on a private GitLab site, here.

Yes, the main "upstream" repo is indeed https://gitlab.lisn.upsaclay.fr/texlive/luatex. Throughout the year, commits/updates are made to this repo. However, sometimes changes are made directly to https://tug.org/svn/texlive/trunk/Build/source/. Occasionally, the GitLab repo pulls in the changes from the TL repo, then once a year (ish) the TL repo pulls in the changes from the GitLab repo.

It doesn't appear to allow contributions, not even in the form of defect reports or similar.

The GitLab site doesn't allow any contributions directly, but the LuaTeX team still accepts them.

and I appreciate and comprehend the remarks from the FAQ

Most of luatex.org is pretty old, so I wouldn't take anything there too literally.

The story here is a little complicated, but here's a short version. LuaTeX started when NTS was going to be the next big thing, but NTS ultimately failed because there was more discussion than actual coding. This was a big problem since computers had changed a lot since 1982, but the only major new engine was pdfTeX, and TeX couldn't afford to stagnate if it wanted to survive.

Also, LuaTeX was originally developed only for ConTeXt. So at the start, LuaTeX gained whatever features that it needed for it to work with ConTeXt. Pretty much every previous TeX engine had near-perfect backwards compatibility with Knuth's original TeX, but LuaTeX intentionally had a few minor backwards incompatibilities. These incompatibilities are to this day controversial.

These days, things are much different. At the start, LuaTeX would change radically from year-to-year; now, there haven't been any major changes for about 5 years. At the start, LuaTeX was entirely driven by ConTeXt's needs; now, the most recent update broke ConTeXt to fix bugs with LaTeX and Plain.

1. Several of the points mentioned around the topic of contributions and even user feedback appear out of date (both sockets and UTF-8 no longer seem to be a concern; ancient versions seem to be referenced; trunk ist referenced instead of the master default branch).

Correct, this is all ancient.

2. Having specific plans and a specific design doesn't generally preclude outside contributions. The only thing it does is to provide a boundary within which contributions should (?) be welcome.

3. Small team of decision-makers doesn't preclude outside contributors or contributions. I've even heard of instances where FLOSS projects gained new core team members by taking in contributions and promoting regular high-quality contributors.

This was written back when LuaTeX was only for ConTeXt, and to this day ConTeXt has very different wants/needs compared to the other TeX engines. Generally speaking, some of the loudest users in the TeX community want nothing to ever change, but this conflicts with ConTeXt's goal of continuous improvement.

Outside contributions are definitely welcome in LuaTeX; I made a first-time contribution (bug fix) two weeks ago. And if you look through the changelogs, LuaTeX still adds new features every year or so.

Considering that the upstream project -- as per their own statements -- doesn't seem to be inclined to take any outside contributions,

Yeah, I agree that this page makes them look semi-hostile to outsiders. But the LuaTeX team (these days, Luigi and sometimes Hans) are actually quite friendly. Keep in mind that there are also some language/cultural issues here too.

is there any way to contribute at all?

Send a patch (in an attachment, not inline) or a bug report to dev-luatex@ntg.nl (current archive is stale, use this one instead). If it's something simple/obvious, Luigi will usually merge/fix it in about a day (although it won't end up in a binary until the next annual TL release).

If it's a feature request or something more complicated, then it will probably take some more time. Generally speaking, LuaTeX doesn't like to add new features anymore, but anything simple or important has decent odds at being accepted.

Patches generally aren't needed for bug reports, but they'll help a lot for a feature request.

In particular I imagine the TeXlive project could be willing to maintain patches rebased against the upstream

TL never patches upstream projects (except for things like web2c, kpathsea, pdftex, etc. where TL itself is the upstream). But if you did manage to get a patch into TL, it would eventually get pulled back into the main LuaTeX repo.

Other responses

[me] How do you build/install LuaTeX from source?

Assuming that you already have a newish TL installation, the commands below should replace your current luatex binaries with new ones compiled from master:

$ git clone --depth 1 //gitlab.lisn.upsaclay.fr/texlive/luatex.git
$ cd luatex
$ ./build.sh --parallel --luahb
# cp build/texk/web2c/luatex build/texk/web2c/luahbtex "$(kpsewhich --var-value=SELFAUTOLOC)"
# fmtutil-sys --byengine luahbtex
# fmtutil-sys --byengine luatex

There isn't really any developer documentation, so you're kind of on your own for the rest.

(You can email me echo bXNldmVuIGF0IHRlbHVzIGRvdCBuZXQK | base64 -d if you have any questions here. I'm just a random user—not on the LuaTeX team or anything—but I might be able to answer a few questions if you get stuck anywhere.)

[0xC0000022L] Joseph Wright suggested https://tug.org/mailman/listinfo/lualatex-dev, would that also be okay for patches?

That list is designed for discussing things like luaotfload, luatexbase, fontspec, etc., so it's not exactly the right place to post patches. But Luigi is the only one who is applying patches right now, he appears to be active on the list, and anything in LuaTeX will ultimately affect LuaLaTeX, so it should be fine.

[Joseph Wright] On the specifics of LuaTeX, the FAQ date from about 2006/7, so the situation is not quite as described: that was when things were very much developmental. LuaTeX is largely 'done', but changes can and are made: you are likely looking for the LuaTeX dev mailing list. (Like much of the TeX ecosystem, LuaTeX development discussion is still much more active on mailing lists than e.g. GitHub issues.)

I agree with all of this.

[mickep] luatex is as far as I know frozen (no new functionality, supposed to be stable). There is the luatex-dev mailing list for bug reports and similar. Maybe that is how the developers prefer to communicate? It is nice that you want to help, maybe asking on the luatex-dev list if there is anything to help with is the best thing to do?

LuaTeX claims to be frozen since TL2019, but this isn't quite true. If you look through the TeX Live Guide, LuaTeX has added a few minor features every year. Just in TL2023 (released 3 months ago) it added a new primitive. Nothing major will be added, but they still seem to be willing to add small new features if they think that it's important enough. [originally from a comment on Meta]

[Joseph Wright] BTW, I see that https://luatex.org/roadmap.html has a pretty clear statement at the end about the status - perhaps you are looking to contribute to LuaMetaTeX?

Contributing to LuaMetaTeX is definitely an option, but keep in mind that

  1. LuaMetaTeX pretty much only works with ConTeXt. There is a way to make it work with LaTeX and Plain, but this is only if you are very adventurous and willing to do lots of debugging. Probably nothing that you're too interested in.

  2. LuaMetaTeX generally only adds features that Hans wants to use in ConTeXt. It is certainly possible to to get patches into LuaMetaTeX (I've sent in a few tiny patches), but you'll need to persuade Hans that it's a good feature. In some ways this is easier since no one is concerned about backwards compatibility or how useful it is to other people, but this is also trickier since if Hans wants a feature, he has usually already added it himself.

[mickep] It is nice that you want to help, maybe asking on the luatex-dev list if there is anything to help with is the best thing to do?" After all, there you will also reach the people actually involved.

[0xC0000022L] Makes little sense to approach someone saying "Hey, I have this brilliant idea. But you gotta implement it". Also I don't just need to get this into a shape to be "presentable" but also to think about it from all sides and consider potential issues such as backwards compatibility concerns.

It's a very low volume mailing list, and everyone is generally pretty friendly, so as long as you aren't too pushy or demanding, I don't see any problem asking on the list—even without having an implementation ready to go.

[mickep] In fact, are you sure that luatex is the right project to contribute to? There is maybe more work going on with tagging. I don't know if you are into math, but many people would love to see more work on the unicode-math package. Just to give a few directions...

I think that this is related to this question, which would definitely need to go into the LuaTeX core binary.

FWIW, I think that there's a good chance that LuaTeX would add what you're talking about in that question there. They'd certainly appreciate a patch, but what you're asking for is maybe simple enough that they'd accept a mere feature request.

Max Chernoff
  • 4,667
  • Regarding your last part: If virtual file handling is the thing, then one could question if that really should go into the engine. ConTeXt has, as you probably know, support for it. For example the buffer system uses it. – mickep May 10 '23 at 12:51
  • @mickep Yeah, that question seems to be looking for pretty much exactly what buffers do in ConTeXt. In ConTeXt, all of the file code is completely custom Lua, but 0xC0000022L wants for LuaTeX to provide something like emulate_default_open_read_file which would be probably needed for perfect LaTeX compatibility. You could probably get a pretty close emulation with Lua, but it would be fairly easy to modify the engine to just fallback on the default behaviour if you return true in the function. – Max Chernoff May 10 '23 at 21:11
  • @mickep I guess I don't have "the purist's eyes" for this sort of thing, as I mainly use LaTeX and friends to produce pretty documents. However, the beauty of embedding Lua is that -- with its notion of functions being first class citizens, having closures etc -- is uniquely equipped to tackle this. Actually what Max wrote would be one change, I'd like to see. But there's another. Considering that I have the existing (!) callbacks and am able to feed content (linewise or multi-line) to TeX, shouldn't I be able to achieve the same behavior, no matter whether I use \input and TeX reads ... – 0xC0000022L May 10 '23 at 22:13
  • @mickep ... the file line-wise or whether I feed the content line-wise through the env.reader callback? Alas, that's not what's happening, because the callback seemingly does expansion differently and at a different point than if I had f = io.open()'d a file f and then iterated over its f:lines(). In fact, I think you just gave me an idea for another question related to the first one, which is more spot-on to the thing I want to get solved. Having emulate_default_open_read_file() -- instead of tapping into default behavior -- is a nuisance, but no blocker. – 0xC0000022L May 10 '23 at 22:17
  • 1
    @0xC0000022L TeX treats files very differently than pretty much every mainstream programming language, so I'm not really surprised that something works differently than it does with Lua. \endlinechar and catcodes come to mind as obvious examples, and these can both be dynamic within a read file. The LuaTeX callbacks can also have some very surprising behaviour/limitations; section 13.4 shows a particularly tricky example. – Max Chernoff May 10 '23 at 22:37
  • 1
    @0xC0000022L You may also want to look at ConTeXt. ConTeXt buffers already do most of what you're looking for, and it has some other neat VFS features (like zip:// to read a file out of a zip file, or http:// for direct network requests). The ConTeXt Lua manual is far better than anything available for LaTeX/Plain, and ConTeXt also has a number of additional features that make working with Lua much easier. You can also look at OpTeX, which also has tight integration with Lua and really clear source/documentation. – Max Chernoff May 10 '23 at 22:43
  • @MaxChernoff reading out of a ZIP is an easy trick (really!). Thanks for the pointers. I will have a look, but I am a little afraid of taking the plunge. As I understand it, ConTeXt means re-learning basically everything I know about about LaTeX. My main point, however, wasn't about how TeX is different. I think Knuth had this metaphor of eyes, mouth and stomach. From past experience I value when an extension mechanism (such as Lua) allows me to do exactly what the "extendee" does. However, with these callbacks it isn't clear at which layer (eye, ...) they act and no way to control that. – 0xC0000022L May 10 '23 at 22:50
  • In NEW (luatexdir) one can find BIG CHANGE: \directlua now runs \scantokens internally.. Given Joseph's comment, I reckon that's part of my problem. It goes on the old behaviour is still present, under the new name \detokenizedlua. Alas, it does no longer exist. Which -- I think -- means that anything I emit from the env.reader is already (in TeX terms) tokenized. – 0xC0000022L May 10 '23 at 22:54
  • 1
    @0xC0000022L "\directlua now runs \scantokens internally" I think that's referring to the tokens (Lua code) inside the {braces} right after \directlua, not to anything that happens from Lua. If env.reader takes a string, then it is either going to be pre-tokenized text or catcode-12-only text. If it takes tokenized data, then you'd need to pass it tokens created with token.create. It might accept both strings and tokens (maybe?); this is an area of LuaTeX that I've never really explored before. – Max Chernoff May 10 '23 at 23:08
  • 2
    @0xC0000022L I neither have the purist's eye nor understand these things in detail. My point is rather: Since the engine developers and the context developers are essentially the same people, I would assume that they would have thought through these kinds of questions very well. Also, since context works very well (with both luatex and luametatex) with virtual files (for example buffers), I would start by looking at how it was done there before suggesting to change the engine. – mickep May 11 '23 at 06:06
  • @mickep "Since the engine developers and the context developers" ... I had no idea :-o ... and looking at the ConTeXt implementation is actually one takeaway from you and Max' comments for me. So: thanks! – 0xC0000022L May 11 '23 at 07:56
  • @MaxChernoff so formally as per documentation, env.reader is supposed to return lines. At least the documentation seems to suggest this behavior. However, it is certainly possible to feed it a string with line breaks. Alas, something happens to that string, which diverges from how files are read internally using \input. That's what I am investigating. Essentially instead of reading from disk, I want to read from a string and use the (existing?) callbacks to feed that to TeX in a fashion that corresponds to \input. env.reader is the "file" reader one has to provide to LuaTeX. – 0xC0000022L May 11 '23 at 08:01