Coding non-trivial algorithms within TeX

Question

This is kind of a hard question to formulate correctly, so forgive me if I have to clarify this question later.

I want to draw a lot of somewhat complex TikZ diagrams, which to be done effectively needs a certain amount of "intelligent behaviour" on the part of the macros that will drawing them. To achieve this I think I have several options, none of which is ideal:

• code the whole thing in TikZ and/or TeX/LaTeX macros. This is a huge challenge because while I'm fairly adept at using LaTeX and TikZ for typesetting, I have no idea how you'd go about implementing a non-trivial algorithm and it feels like attaining the needed expertise would require a major investment of time.

• use LuaLaTeX. This is the obvious Right Solution, but unfortunately it would mean I couldn't post anything written using the system to arXiv, and hence it has to be dismissed.

• write the layout code in Julia or Python etc. and spit out TikZ code that I paste into my document. This is by far the easiest solution but it means I have to manage all the diagrams separately from the main text, which I'm hoping to avoid.

• write code in my language of choice that acts as a preprocessor for .tex files. This avoids the issues with the above option but means I can't use Overleaf for collaborations.

• [added in edit] use PythonTeX - this is even slicker than writing my own preprocessor and probably by far the best option, but it does have the same issue that I have to teach my colleagues to use git because it doesn't work with Overleaf.

Because none of these are ideal, I was wondering if an additional option might exist: TeX is a Turing-complete macro processing language, so in principle it should be possible to implement a more modern-feeling basic scripting language within TeX.

I realise it's a long shot, but my question is just whether this has been done: is there a package that lets you code in TeX in a way that "feels" like writing code in a modern scripting language, in such a way that I could auto-generate TikZ diagrams without needing to know all the deep intricacies of how TeX macros work?

I'm open to other suggestions also and am happy for this to become a general question about how to get things done when you want to automate things but aren't a TeX wizard.

Answer 1

What workflows to suggest depends on many circumstances.

In case there is the requirement of recipients of your .tex-input-files being able to obtain the .pdf-file simply by running LaTeX on the main .tex-file on whatsoever (probably more recent) TeX-platform, then you probably cannot get around implementing things in TeX because only the availability of the TeX-platform itself can be taken for granted.

If you can make some assumptions regarding the computer-platform, the TeX-platform and the possibility of running/executing external programs from within TeX, then things might look different:

In LaTeX you can, e.g., use the filecontents*-environment or \immediate, \openout, \write and \closeout for creating an external text file which contains the source code for a program, which in turn is the implementation of the algorithm needed by you in your preferred programming language. The output of the program can be an external text-file which contains .tex-code representing the result of carrying out the algorithm. (\immediate\write, however, might require a bit of knowledge of how and when TeX writes tokens to file and of expansion that hereby might take place.)

Using \write18 (traditional TeX and XeTeX) or os.execute (LuaTeX) —the package shellesc provides a unified interface—you can have executed commands on your computer for running the compiler of your choice on that source code and afterwards running the resulting executable and this way obtaining the text-file which contains the .tex-code representing the result of carrying out the algorithm.

This text-file in turn can be processed by TeX via \input.

However, such approaches are highly platform-dependent and you might need to do a great deal of (shellscript-)programming for catching up the situation of something going wrong while having TeX run the external programs.

Configuring things (rights to run external programs, enabling \write18/os.execute) might also turn out cumbersome on some of the nowadays' computer platforms.

If you use overleaf or the like online-platform, workflows of this kind are not really an option to you.

---

I the following I take the title of this question to be a question about general strategies/procedures when implementing algorithms in TeX.

When implementing things in TeX, you might want to think of things "materially":

TeX reads its input and thinks of the input as a set of instructions to put pretty glittery things on an assembly line one after another. These pretty glittery things are called "tokens".

(Excursus: These tokens are either control sequence tokens or character tokens. Control sequence tokens are those things where you usually write a backslash first in the TeX source code. So for example \LaTeX, \section, ... Some control sequence tokens, e.g. macros, e.g. some primitives, like \romannumeral or \string or \csname..\endcsname are expandable. That is, they and their arguments are replaced by other tokens at an early stage of processing.
Some control sequence tokens are not expandable. For example \relax or \hbox{...}. They are not replaced in said early stage but remain and serve in later stages e.g. as basic directives for typesetting the text.
Character tokens have a character code which in the TeX-internal character encoding scheme (either ASCII or Unicode) corresponds to the number of the code point of the respective character, and belong to a category on which depends what TeX does when the respective character token is processed. Character tokens of category 13 (active) can be used like control sequence tokens. Together with the control sequence tokens they form the control sequences. Excursus end.)

TeX processes these pretty glittery things. The first station the tokens go through on the assembly line is the "expansion department". Here expandable tokens and their arguments are taken off the assembly line and constellations of tokens are put on the assembly line to replace those tokens. How the constellations of replacement tokens are composed depends on the definition or meaning of the replaced expandable tokens.

When the replacing is over in the expansion department, there are no more expandable tokens on the assembly line. The (non-expandable) tokens resulting from going through the expansion department on the assembly line are transported to departments where they are considered as directives for typesetting text or for defining macros or for assigning values to registers/parameters or for shipping typeset pages into the .pdf or .dvi output file or the like.

When I implement algorithms in TeX and want to use only macros/expandable tokens, so that the algorithm is fully processed in the expansion department alone, and only tokens representing the result should reach subsequent departments, I think of it this way:

A macro can have, depending on the definition, up to nine arguments, which must be placed on the assembly line right behind the respective macro token, each argument enclosed in curly braces. The curly braces are also tokens, and the arguments themselves are also constellations of tokens. (If an argument is to consist of only one token, the curly braces around it can be omitted).

A macro denotes a step of the algorithm.
The arguments of a macro, numbered from #1 to (at most) #9, can be thought of as variables used in that step of the algorithm. The tokens that make up the arguments in an expansion step can be thought of as the values of the variables in that step.

An expansion step, i.e., a replacement step, represents a step of the algorithm. In such a step, the macro token denoting the step of the algorithm and the tokens that are its arguments and thus denote the values of the variables are taken off the assembly line and are replaced on the assembly line by the macro token that denotes the next step, followed by arguments required in the next step, consisting of curly brace tokens and sequences of tokens nested within them that represent the values of the variables in the next step.

You can often do this recursively. That means, after a few intermediate replacement-steps, which are only there to rearrange tokens for arguments, which means to change values of variables of the algorithm, and to put the arguments in the right order on the assembly line, the token is put in front of everything, which denotes that step of the algorithm, from which the expansion-cascade/replacement-cascade was started. In this way, one can have steps of an algorithm repeated recursively until the arguments/values of variables meet certain conditions.

In principle it is a matter of replacing tokens by other tokens expansion step by expansion step until those tokens are there which are considered as the result of the algorithm. If you use subroutines, you may need to know after how many expansion steps their result is there.

It is very helpful to know how \expandafter works, how \romannumeral and in some contexts \the and with recent TeX-engines \expanded can be (mis)used to trigger further expansion steps (, e.g. all expansion steps of an expansion based "sub-routine"), how \csname. .\endcsname , \number and the \if-\else-\fibranches work respectively which tokens behind tokens \if/\else/\fi are removed from or left on the token assembly line under which conditions in which expansion-steps.

The figurative idea that tokens are items that lie one behind the other on an assembly line, combined with the question how an expansion step that gets applied to the token currently processed in the expansion department changes what tokens lie on the assembly line, and how an expansion step itself triggers further expansion steps (\romannumeral, \number, \csname, \if, \ifcat) or triggers an expansion-step to the next but one token (\expandafter) and so influences the time order in which tokens on the assembly line are expanded is helpful when it comes to macro programming in TeX.