Syntax of TeX/LaTeX

Question

{} and [] do some kind of grouping, and comma seems to be special as well. Being a programmer I find it frustrating to use Latex on incantation-only basis. I would like to know how does it exactly work.

What is exactly the syntax of TeX/LaTeX?
How is it parsed?
Does Latex introduces new commands only or changes the syntax as well?
Or maybe changing syntax locally is a inherent feature of Tex?

Answer 1

For a programmer, I would recommend reading the book "TeX by Topic" (available for free at http://www.eijkhout.net/tbt/). I think that that will give you the best answer to your general questions (which really are, in my opinion, a bit too general for this site; I would advise you to read TbT and then ask more focussed questions on particular aspects).

However, you mention a couple of specifics so let me try to answer them. {} are lexical:

\def\hello{world}
{\def\hello{hello}
\hello}
\hello

produces: "hello world".

The grouping given by square brackets is an illusion. Some commands in LaTeX start with "If the next character is [, gobble the rest up to ] and use it as the first argument.". This can cause problems if, for example, what you want to pass has a square bracket in it:

\newcommand{\hello}[2][hello]{#1 #2}
\hello{world}
\hello[greetings]{world}
\hello[Greetings [1] to the]{world}

Produces

hello world
greetings world
Greetings [1 worldo the]world

because the first argument to "hello" in the last line is "Greetings [1".

(This can be remedied by using braces to do proper grouping, \hello[{Greetings [1] to the}]{world} would work fine).

I've never heard of commas being special. Can you give an example?

Answer 2

I would recommend reading The TeXbook, by Donald Knuth, at least for two reasons:

1. It is a big pleasure to read. Knuth's writing style is very clear and explanatory. Not to mention he is the author of TeX.
2. It has a excellent chapter which is almost literally the answer to your question: ``Chapter 7. How TeX Reads What You Type.''

I consider this book to be the best writing about the TeX low-level features.

Answer 3

The excellent TeX by Topic has already been mentioned, and I'd start there if you are a programmer looking for enlightenment. What you need to bear in mind is that TeX works with tokens with category codes. In the case of grouping, usually { and } are tokens with category codes 1 (begin group) and 2 (end group), respectively. So TeX will start a group when it reads { and finish it when it reads }.

For a simple macro such as

\newcommand*\example[1]{Stuff with #1}

the argument will be either a single token or a single group. Thus

\example{stuff}

will absorb 'stuff' as #1, but

\example stuff

will only pick up the 's' of 'stuff', as 's' is a single token.

LaTeX introduces the idea of optional arguments inside [ ... ]. As pointed out by Andrew, this isn't done by grouping, and so can give odd results. You can get around this: see how xparse manages optional arguments for example. LaTeX also uses ( ... ) for co-ordinate arguments, but again with no grouping.

As I said, category codes are important. ConTeXt uses [ and ] for grouping rather than { and }. So it's not the character you need to worry about, but the token (including the category code). This is a rather TeX-specific concept, so looking at TeX by Topic and The TeXbook is a good way to get to grips with it.

Answer 4

Since TeX by Topic and the TeXbook have already been mentioned as particularly relevant to programmers, I think it would be egregious not to mention TeX: The Program. This is a beautiful—probably the crowning—example of Knuth's philosophy of literate programming; I am at absolute best an amateur programmer, and I found it very easy to dive in and read the parts I wanted (I, too, am deeply concerned with the parsing, and not so much right now with the layout) and leave out the rest.

As a warning, as Juan Navarro mentioned, any visions of getting a clean view of parsing, separate from the actual ‘compilation’ per se, will soon be dashed. What one would hope to think of from a modern point of view as distinct parts of the typesetting process are hopelessly intermingled; this is at least part of why people say that TeX is a terrible programming language. Chris Rowley argues for a modern approach to separating what should be separate (and people like Taco Hoekwater have done some beautiful things that should count as steps in that direction, even though I'm not sure that was their intention).