11

How expressive is TeX in terms of being a compiler or a preprocessor of other languages? One splendid example is the listings package in which TeX is changed so as to kind of recognize the kind of tokens of almost all programming languages. So, yes, TeX can be made into a kind of preprocessor.

But can TeX be a parser? Can you implement something like YACC in TeX, so that you define a BNF, and then the rest of the input is parsed by this BNF.

Let's simplify the question a bit. is at least theoretically possible (can't cheat by emulating a Turing machine) to make such lex.tex file so that the input file

 \input lex.tex
 DEFINITIONS
   letter=[a-zA-Z];
   digit=0-9;
   alphanum=$(Letter)|$(digit);
   id=$(letter)$(alphanum)*       
RULES
   id -> { some($); }
INPUT
   hello world

will actually behave such as a lexical analyzer with three sections: defintions, rules, and input. The first two sections define how the INPUT section is processed, producing 

some(hello) some(world)

If this is possible and realistic in some way, then one may aspire for more, yacc.tex, and then perhaps, a universal processor/compiler all written in tex? Wouldn't that be exciting?

Martin Schröder
  • 15,156
Yossi Gil
  • 15,951
  • 4
    TeX is Turing-complete. While it is possible, it is not immediately realistic. Even if implemented, I imagine such a parser would be incredibly slow. – Sean Allred Feb 05 '17 at 20:05
  • 5
    I've implemented a more or less complete parser for XML in TeX, including DTD syntax and namespace resolution, for example so can tex be a parser sounds like it ought to have "yes" as an answer, but I didn't implement lex(which is part of a parser generator rather than a parser, doable but tedious) – David Carlisle Feb 05 '17 at 20:07
  • 4
    while it would be theoretically possible to do in tex, if doing this now it would make far more sense to do it in luatex and use lpeg – David Carlisle Feb 05 '17 at 23:36
  • 1
    Your question triggered my memory of this answer of mine, http://tex.stackexchange.com/questions/332012/translate-in-line-equations-to-tex-code-any-package/332061#332061. I mention this because the listofitems package provides powerful parsing capability, and in the [updated] answer cited, is used to digest alternative syntax for equations and [tries to] convert it to LaTeX, by producing a long string of "spaghetti" or "DNA" code that compiles into the desired result. – Steven B. Segletes Feb 09 '17 at 19:48
  • BaSiX - An Interpreter Written in TeX – Henri Menke Feb 09 '17 at 21:15
  • TeX is more complex than ordinary programming languages: even parsing TeX is Turing-complete. – gernot Feb 10 '17 at 16:07
  • This is related with the literate programming paradigm of Donald Knuth, but actually is mostly done without force to TeX to give mortal leaps. First, foreign chunks of code are parsed by another engine (R,bash,etc.), then converted to LaTeX code to produce a pure LaTeX file, and finally TeX compilers only have to deal with TeX code. Today, knitr is the most useful example of this concept, afaik. – Fran Feb 14 '17 at 09:51
  • Have you looked at the TUGboat article by Pedro Palao Gostanza: Fast scanners and self-parsing in TeX? – Udo Wermuth Apr 24 '17 at 14:56
  • My answer here is possibly related: https://tex.stackexchange.com/questions/332012/translate-in-line-equations-to-tex-code-any-package/332061#332061 – Steven B. Segletes Apr 26 '17 at 01:08

0 Answers0