What are the advantages of Lua over other scripting languages for a TeX implementation?

Question

The question is inspired by Martin Schröder's answer to I am new to TeX. Should I use LaTeX, XeLaTeX, …? as well as my two previous questions on this forum:

1. Automatic document update
2. Generate LaTeX Truth Table with Python Cheetah

In retrospect, I am truly surprised that no LuaTeX user offered any answers to my questions as it seems to me that that could be easily done in LuaTeX by embedding Lua scripts into the LaTeX code. Can it be really done (question for LuaTeX users only) and is it easy?

My second question is motivated by the fact that I hate very thought of having to learn another scripting/general programming language (Lua) if I already know Python (you can substitute Perl if you like) just to use it with TeX. Is anybody cooking a Python interface for TeX call it PythonTeX for short (PerlTeX actually does exist)? As far as I know PythonTeX doesn't exist. There is something called PyTeX but that just look like a tool for people who want to type TeX documents using Python syntax.

Is there an advantage of using Lua over Python as an embedded scripting language for TeX (apart of the fact that people who started LuaTeX probably were more familiar with Lua)?

Answer 1

I have been coding "PythonTeX" since last May, and am planning the first public release sometime between next weekend and the beginning of March. I'm actually planning to call it PythonTeX, by analogy to PerlTeX and SageTeX.

I've created a LaTeX package, and accompanying Python scripts, that provide most functionality of python.sty, SageTeX, and SympyTeX, but with emphasis on speed and usability. Here's a brief summary of what PythonTeX will bring.

• Persistence between commands and environments. For example, each environment \begin{pythoncode} ... \end{pythoncode} picks up where the last left off, so all variables, functions, etc. persist. You can optionally name commands and environments, and then there is only persistence between commands and environments with the same name. For example, \begin{pythoncode}[name] ... \end{pythoncode} only shares persistence with other commands and environments called name.

• Automatic inclusion of printed content. For example,

  \begin{pythoncode}
  print(1+1)
  \end{pythoncode}
  

  is automatically replaced by 2 in the compiled document.

• Speed. Each set of named commands and environments is executed in its own process, using Python's multiprocessing package. Also, each set is hashed, so code is only executed when changed. The results of running the code are saved, and reused when no changes are detected.

• Code typesetting. All code that is executed can also be typeset, using fancyvrb and its internals. Optionally, code can be typeset using Pygments, which is a Python package that does syntax highlighting (Pygments is used by the minted package). When Pygments is used, all code is hashed and the typeset results are saved, so that only changed code must be processed by Pygments in subsequent runs. This helps prevent Pygments from slowing things down.

• Meaningful error messages. All error messages are parsed by code that determines the line of your document where the error occurred (as opposed to the line of the *.py file that is actually executed). So you know exactly where things went wrong in your document.

• Minimal files. The code cleans up after itself, so the number of permanent files created by Python is kept to a minimum and all temporary files are deleted after each run.

I can post a link as soon as PythonTeX is released. I will also be submitting it to CTAN shortly after the first public release.

Answer 2

The LuaTeX developers have commented on their choice of Lua over other languages, including Python, on their home page.

Embedding the interpreter is one thing, and apparently no fun with Python. Another is to actually make the innards of TeX visible to the embedded interpreter. While much of the communication code could likely be adapted in some way from LuaTeX to PythonTeX, it will still be a lot of work.

A better approach may be to cheat a bit, and simply piggyback on the LuaTeX interpreter. There is a bridge that 'embeds' Python inside Lua called Lunatic Python, and maybe it can be made to work with LuaTeX also. However, it may be easier to just use XML-RPC: Let LuaTeX spawn a Python process, which acts as the XML-RPC server. LuaTeX commands can then make calls to the Python process. Bonus feature: Once this is implemented on the LuaTeX side, it can be used with any other scripting language, not just Python.

Answer 3

Here is a way to call into a persistent Python process from LuaTeX via JSON-RPC. 1)

Required setup:

1. Download and install jsonrpclib. This provides the Python server.

2. Download a Lua json-rpc client. I adapted some Lua modules I found on the web and zipped them up and put them on my server. Unzip into one of the directories specified in this answer: "Lua tree" (analogue of texmf tree)

Now, we can write a Python server that exposes some functions:

import sys
from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCServer

server = SimpleJSONRPCServer(('localhost', 8080))

server.register_function(lambda: "Hello from Python!", 'ping')
server.register_function(lambda *args: sum(args), 'add')

# the counter generator shows that the server can be used
# to preserve state between requests, and even across multiple
# LaTeX compiler runs (which we may not want)
counter = iter(xrange(sys.maxint))
server.register_function(lambda: counter.next(), 'count')

server.serve_forever()

Save this as testserver.py and run.

Here is test document that shows how to call the server:

\documentclass{article}

\directlua{ % make the running python server available as a proxy
    require "jsonrpc"
    pythonserver = jsonrpc.proxy("http://localhost:8080")
}

\newcommand{\python}[2][]{\directlua{%
    result = pythonserver.#2(#1)
    tex.print(result)
}}

\begin{document}

\python{ping}

The sum of 1,2,3,4,5,6,2,4,5,6 is \python[1,2,3,4,5,6,2,4,5,6]{add}

\directlua{%
    for x=0,100,1 do
        result = pythonserver.count()
        tex.print(result)
    end
}

\end{document}

Addressing the original question about how to implement pythontex, all that would be required now is a shell script that first starts the server, then calls LuaTeX, and then shuts down the server again. This is left as an exercise to the reader ;)

1) The reason why I used JSON-RPC instead of XML-RPC is that I couldn't find a Lua-only XML-RPC client; the one client that I did find used a C extension library, and I and couldn't figure out how to load that into LuaTeX (blush). The JSON-RPC modules were simpler, and after some kicking and screaming (on my part) could be persuaded to cooperate. Maybe someday I can figure out how to make XML-RPC work also.

Answer 4

My take is: Lua as a language was created as an embedded language and is well suited for the job. Now that with LuaTeX a scriptable TeX implementation exists, there is little principled reason to create another one. Of course, you can go ahead and create PythonTeX. But it seems that the community feels that this would not add enough value over LuaTeX to justify the cost.

Answer 5

Actually, a good way to use Python code into (plain) TeX code is:

1. for the typesetting work (boxes, glue and penalties) use only TeX;
2. for calculation, databases interface and so on you can build Python modules that you can call from the TeX program via \write18. The Python modules have to leave the output into files that have to be read via the usual \newread and then used into the TeX code. This solution work best on Linux, as far as on Linux Python is natively installed.

As an example try this: prepare the module `adder.py' this way:

#!/usr/bin/env python
# file 'adder.py'

import sys

def save(data):
    tempFile = open("temp.dat","w")
    tempFile.write(str(data))
    tempFile.close()

def double(x):
    result = x * 2
    save(result)


if __name__ == "__main__":
    n = int(sys.argv[1])
    double(n)
    sys.exit

now, in the same directory, you can type a TeX file like this:

% file `doubler.tex'
\newread\tempfile
%
\def\double#1{%
    \immediate\write18{python adder.py #1}\relax
    \immediate\openin\tempfile=temp.dat
    \read\tempfile to \var
    \var}%
%
Double 2 is \double2\par
Double 3 is \double3\par
Double 4 is \double4\par
Double 5 is \double5\par
Double 6 is \double6\par
\bye

This TeX file has to be compiled with shell-escape, e.g.:

tex -shell-escape doubler

pdftex -shell-escape doubler

xetex -shell-escape doubler

Answer 6

I think that the pdfTeX community analyzed the situation correctly when they chose Lua. Lua is intended to be embedded into other applications, and it provides a robust, easy to use C API. The Lua API is straightforward and its design eliminates the need for manual reference management in C code, unlike Python's API.

Python is an opinionated language controlled by essentially one person and many people, myself included do not favour the significance of whitespace.

Yes, for you it maybe easier to use Python as it would have been easier for me if everyone stayed with the original Pascal code. Pascal would have provided everything one needed, it is a beautiful language and everyone is almost using it daily via pseudo-code and it would have provided a good way to migrate TeX to a new generation.

If there is any merit to suggest another language, I would recommend JavaScript. It is gaining in popularity via the web and easy to learn. The maths part of it has already been written, at least for browsers. It also provides full unicode support.

Answer 7

Pweave

Much like the original poster, I also do regret the absence of a LaTeX interpreter capable of directly interpreting Python code.

Since PythonTeX got mentioned here in one of the other answers, I also would like to call your attention to Pweave. It is easier to use than PythonTex and also offers a couple more environment options.

Compiling a document that uses PythonTeX involves three steps: run latex, run pythontex.py, and finally run latex again. Pweave only needs two steps: run Pweave, then run pdflatex.

I got started with Pweave by reading a concise but excellent introductory article. You will also find my minimal working example over here on tex.StackExchange.

I have also posted an easier example involving currency notations and demonstrating the power of locales, Python and Pweave over here.

EDIT:

$ Pweave -l

Pweave supported output formats:

• sphinx: reStructuredText for Sphinx
• pandoc: Pandoc markdown
• tex: Latex with verbatim for code and results
• html: HTML with pygments highlighting
• rst: reStructuredText
• texpweave: Latex output with user defined formatting using named environments (in latex header)
• texminted: Latex with predefined minted environment for codeblocks

More info: http://mpastell.com/pweave/formats.html