Is it possible to call external programs from LaTeX?

Question

I'm trying to write very specific LaTeX package that does a lot of complex computation. It encloses some parts of text (or whatever else user wants to print) into boxes and then runs quite complex computations to determine where to lay out those boxes in the document.

It is not practical to program that computation in LaTeX - they will be challenging even in high-level programming language. Is there a way to delegate that computations to some external program? The desired workflow would be like this:

1. In LaTeX code collect necessary parameters: dimensions of all the boxes to be print, textwidth and textheight.
2. Send it to external program.
3. Based on received parameters the program would compute placement of all the boxes and send the results back to LaTeX somehow.
4. LaTeX would use externally computed positions of boxes and print stored boxes at those locations.

Is there a way to communicate with external programs? It does not have to be anything elegant, storing parameters in file and then reading computed results from another file would be sufficient. I do not plan to redistribute the package yet so it does not matter that the external program cannot be redistributed.

Answer 1

My pythontex package might be an option (latest version on GitHub; will be on CTAN soon). It offers the following features.

• Enter Python within your LaTeX document, and bring the output back into LaTeX.
• User-defined sessions. If you have independent calculations that are slow, just put them in separate sessions, and they will automatically run in parallel, using all available cores.
• All calculations are saved or cached; code is only executed when modified (unless you specify otherwise).
• Any errors or warnings are synchronized with your document, so that you know which line in your document caused problems.

The easiest thing to do would be to pass whatever parameters you need to Python, do your calculations, and then print LaTeX code back to the TeX side. There are a few examples in the documentation, and I've also used pythontex to answer a few questions on this site.

If you just want to get values back from Python, stored in macros, here's an example. This allows you to pass a length to Python, perform some calculations on it, and then return the output to TeX, stored in a macro. Again, usually it's probably easier to just assemble a complete block of LaTeX code within Python, and print that back instead (maybe have a template, and perform substitutions.)

\documentclass{article}

\usepackage{pythontex}

\makeatletter

% Define a Python function that takes a TeX length and a name,
% extracts the numerical part of the length and performs
% a calculation with it, and finally prints a macro definition
% back to TeX that stores the calculated value, with its 
% units, using the specified name for the macro.
\begin{pylabcode}
def my_calc(length, name):
    num = float(length[:-2])
    unit = length[-2:]
    print(r'\def\{0}{{{1}{2}}}'.format(name, str(sqrt(num**2/2)), unit))
\end{pylabcode}

\newcommand{\mycalc}[2]{%
    % Create a default macro for the output.
    % This avoids errors before PythonTeX runs
    % and creates the real value
    \expandafter\def\csname #2\endcsname{0pt}%
    % When using PythonTeX commands inside macros,
    % it's sometimes easiest to employ a helper 
    % macro to get everything expanded properly.
    \expandafter\mycalc@i\expandafter{\the#1}{#2}%
}
\newcommand{\mycalc@i}[2]{%
    \pylabc{my_calc('#1','#2')}%
}

\makeatother


\begin{document}

% This passes the value of \textwidth to Python,
% performs a calculation on it, and stores the result,
% including units, in the macro \returnvalue
\mycalc{\textwidth}{returnvalue}

\returnvalue

\end{document}

Answer 2

(This is a ConTeXt solution, not a LaTeX solution; I am posting this in case someone wants to achieve the same in ConTeXt).

Is there a way to communicate with external programs? It does not have to be anything elegant, storing parameters in file and then reading computed results from another file would be sufficient.

For ConTeXt, I have written a module---filter (at github)---that does exactly that. As an example, suppose you want to create figures using Graphviz and include them in ConTeXt. For that, define a graphviz environment as follows:

\usemodule[filter]

\defineexternalfilter
  [graphviz]
  [
    filter=\externalfilterparameter{layout} -Tpdf -o\externalfilteroutputfile,
    layout=dot, % neato or twopi
    cache=yes,
    output=\externalfilterbasefile.pdf,
    readcommand=\ReadPDF,
  ]

\def\ReadPDF#1%
    {\externalfigure[#1]}

and then typeset a graph as:

\startgraphviz
   digraph G {
       a_1-> a_2 -> a_3 -> a_1;
    }
\stopgraphviz

If you want to change the layout program (different algorithm used by graphviz), use:

\startgraphviz[layout=neato]
   ....
\stopgraphviz

You can pass arbitrary options to the external program using its command line options.

This package is just a wrapper around \write18, but provides some useful features:

• Caching of results (rerun the external program only if the content has changed)
• Specify output directory used to store all the temporary files (current directory used by default)
• Enable diagnostic information if something goes wrong.

Answer 3

You could use pipes for this. Assuming you use Miktex you will have to add -enable-pipes flag to your tex command. You can then use pipes when opening files. You can open files to read and write. To read from a file, that really is a command you can use:

\newread\source
\openin \source = "|echo Text"
\loop
\read\source to \currentline
\ifeof \source
\global\morefalse
\else

What you do with the input (probably assign it to a length \setlength{\linewidth}{\currentline} or something)

\fi
\ifmore\repeat
\closein\source

Here echo Text would be your executed command. To write to a file or the command line input of a program works the same way, just with \newwrite, \openout, \write and \closeout. If the first character of the file is not | than Tex will look for an actual file. Be careful with escape characters though, \input{File_You_just_wrote_with_external_tool} might be easier.

Answer 4

If you have complicated calculations, I would tend to think you want either the python package or the sagetex package. There's an example of using Python here but sagetex uses Python commands plus math commands created for the CAS known as Sage so it's ideal for complicated problems. There's a Sage Cell Server where you can experiment with Sage yourself. From here you'll find "A concise description of sagetex is that it is a collection of TeX macros that allow a LaTeX document to include instructions to have Sage compute various objects and/or format objects using the latex() support built in to Sage. So as an intermediate step of compiling a LaTeX document, all of the computational and LaTeX-formatting features of Sage can be handled automatically.". The documentation of the sagetex package states on page 5 and 6: "Note that since LaTeX will do macro expansion on whatever you give to \sage,you can mix LaTeX variables and Sage variables! If you have defined the Sage variable foo to be 12 (using, say, the sageblock environment), then you can do something like this: The prime factorization of the current page number plus foo is $\sage{factor(foo + \thepage)}$." Here's a small working example I put together showing that off:

MAJOR EDIT MADE TO BETTER ANSWER THE QUESTION

\documentclass{article}
\usepackage{sagetex}
\pagestyle{empty}
\begin{document}
The documentation of the sagetex package gives a simple example
showing how you can combine \LaTeX and Sage variables similar
to this:
\begin{sagesilent}
foo=99
\end{sagesilent}
The prime factorization of the current page number plus foo
is $\sage{factor(foo + \thepage)}$.

The 2nd footnote (bottom of page 6) indicates there can be some
``weirdness'' depending on what you're processing. 
\end{document}

After running MyFile.tex with pdfLaTeX, a MyFile.sage file is created. Process this with Sage from the command line: load 'MyFile.sage' and Sage will create MyFile.sout and instruct you to LaTeX on your document again. When you do so, your document will print your document with the Sage calculations/text/pictures it created. This is the output from the code above.

It's also possible to "...get input and send it to Sage at the time of compilation" (quote from this on page 271). There's a PDF you can download with more examples of sagetex; the important application on page 9: "Make Sage write your LaTeX for you". There might be other packages that can do what you want but when if work with complicated math, sagetex is a package you need to learn.

Answer 5

There are a number of packages that allow this. The most general is probably bashful.

bashful provides a convenient interface to TEX’s primitive \write18—the execution of shell commands from within your input files, also known as shell escape. Text between \bash and \END is executed by bash, a popular Unix command line interpreter. Various flags control whether the executed commands and their output show up in the printed document, and whether they are saved to files.

in that you can then do anything that you can do from a BASH CLI.

If you are willing to limit yourself to Python there is the python package

This package enables you to embed Python code (www.python.org) in LaTeX and insert the output into your LaTeX document.