Fully expand macro argument

Question

I am sure this has been answered already, but I cannot seem to find any answer that explains it in core TeX primitives and that I understand on a token level, e.g. I definitely don't want to get into expl3 before I got a grip on TeX itself.

What I am trying to do is basically just being able to write macros that can be used in a nested fashion using braces like \amacro{\bmacro{bla}}, first evaluating \bmacro{bla} to e.g. la (argument without first token = \@cdr) and then plugging that into \amacro, evaluating to l (only first token of argument = \@car).

\catcode`\@=11 %
\def\@car#1#2\@nil{#1}
\def\@cdr#1#2\@nil{#2}
\def\amacro#1{%
    \edef\tempa{#1}%
    \@car\tempa\@nil%
}
\def\bmacro#1{%
    \edef\tempa{#1}%
    \@cdr\tempa\@nil%
}
\catcode`\@=12 %

    a: \amacro{bla}\par
    b: \bmacro{bla}\par
    a(b): \amacro{\bmacro{bla}}

\bye

Using \edef I try to ensure that the argument is fully expanded before I go and operate on it, but while the individual macro calls work fine, compilation fails with the nested way with

! Missing control sequence inserted.
<inserted text> 
                \inaccessible 
l.16 a(b): \amacro{\bmacro{bla}}

? 

It's not that unexpected, since it kind of is a nested \edef on the same name so I could very well see it breaking that way, but I didn't really get around what \inaccessible means in this context and why it arises.

And then, of course, the main point, there must be a proper way to do this in TeX core primitives, right? There has to be some way of fully expanding an argument before doing something with it.

Answer 1

First of all, you're missing \expandafter:

\documentclass{article}

\makeatletter
\def\amacro#1{%
  \edef\tempa{#1}%
  \expandafter\@car\tempa\@nil 
}
\def\bmacro#1{%
  \edef\tempa{#1}%
  \expandafter\@cdr\tempa\@nil 
}
\makeatother

\begin{document}

a: \amacro{bla}

b: \bmacro{bla}

%    a(b): \amacro{\bmacro{bla}}

\end{document}

otherwise \@car will just produce \temp and \@cdr nothing.

You cannot nest \bmacro in \amacro, because \edef is not expandable.

On the other hand, with LuaTeX there is \expanded (and it will be also in the other engines in a near future).

% compile with lualatex
\documentclass{article}

\makeatletter
\def\amacro#1{%
  \expandafter\@car\expanded{#1}\@nil
}
\def\bmacro#1{%
  \expandafter\@cdr\expanded{#1}\@nil
}
\makeatother

\begin{document}

a: \amacro{bla}

b: \bmacro{bla}

a(b): \amacro{\bmacro{bla}}

\end{document}

Answer 2

Basically, I’m posting this answer because it is way too long for a comment.

As usual, @egreg’s post already answers your question in essence. However, I think that you are having difficulties in understanding exactly what expansion means, and that you tend to confuse expansion with digestion (i.e., execution in the stomach), two concepts that are quite different from each other. This is absolutely excusable, since TeX’s behavior in this regard is really hard to understand, and the answer that David Carlisle has linked to is very helpful in explaining this subtle topic; nevertheless, I think that it could also be helpful to show you exactly what happens when your code is executed, in a step-by-step fashion. This is precisely what this answer does.

---

What happens with \amacro{bla}

The tokens

\amacro{bla}

are replaced with

\edef\tempa{bla} \@car\tempa\@nil

Then TeX processes \edef: this is an unexpandable token, so it is forwarded to the “stomach”, where it is executed. Since the tokens in the replacement text are all unexpandable (they are just character tokens), this means that \tempa is defined to expand into bla. Next, TeX finds \@car, which is defined as

\def\@car#1#2\@nil{#1}

In particular, being a macro, \@car is expandable. The first argument of \@car is undelimited, the second is delimited by \@nil; in our case, \tempa (!) is taken as the first argument, and the second argument is empty. Note that \@nil is gobbled as well as part of the expansion process. The result of the expansion is the first argument, that is, \tempa. In conclusion,

\@car\tempa\@nil

is replaced with

\tempa

This is again an expandable token, so it is expanded: the current definition of \tempa is bla, so

bla

is what remains in the token stream after the expansion. These are all unexpandable tokens, so they are sent to the stomach, where they are executed. The final result is that TeX typesets “abc” (note that it was already in horizontal mode) and that a temporary definition has been made.

---

What happens with \bmacro{bla}

This is very similar to the previous case, hence I‘ll leave it “as an exercise” to you. The final result is that

\bmacro{bla}

is replaced by nothing, i.e., it is removed from the token stream, but always with the side effect of defining \tempa to expanda into bla.

---

What happens with \amacro{\bmacro{bla}}

This is, of course, the most interesting case. Initially, the token stream contains

\amacro{\bmacro{bla}}

According to the definition of \amacro, this is replaced with

\edef\tempa{\bmacro{bla}} \@car\tempa\@nil

Again, the next thing that TeX processes is \edef, which, I repeat, is not expandable and proceeds therefore to the stomach, where it is executed (not expanded!): \tempa is defined as a macro that will expand to the current full expansion of \bmacro{bla}. So TeX begins to fully expand \bmacro{bla}; according to the current definition of \bmacro, the first expansion step yields \edef\tempa{bla} \@cdr\tempa\@nil, and TeX looks at each of these ten tokens, in turn, to see if it happens to be expandable. Now:

• \edef is unexpandable, so it is left alone;

• \tempa, on the other hand, is expandable, so (aha!) it is expanded according to its current definition, which, at this moment, is still the definition set, as a side effect, by the execution of the line

  b: \bmacro{bla}\\
  

  found in your source file, that is, bla; these three tokens are unexpandable and therefore they remain as they are;

• a { token follows, which is unexpandable, and hence is left alone;

• similarly, the next four tokens, b, l, a, and } are copied ”verbatim” into the replacement text;

• \@cdr is expandable, so it is expanded: \tempa becomes its first argument, while the second argument of \@cdr is empty; \@nil is also gobbled as the argument delimiter; according to its definition, \@cdr is expanded into its second argument, that is, into nothing;

• at this point, the replacement text of the definition that we were executing (recall that it was

  \edef\tempa{\bmacro{bla}}
  

  see above where we said “it is executed (not expanded!)”) has been expanded fully, yielding \edef bla{bla}.

In conclusion, \tempa is defined as a macro whose replacement text is

\edef bla{bla}

(nine tokens). This completes the execution (not expansion) of the code

\edef\tempa{\bmacro{bla}}

What remains now in the token stream is

\@car\tempa\@nil

\@car is expandable, so it is expanded (obviously…): its first argument is \tempa and its second argument is empty; the expansion is the first argument, that is, \tempa. All in all, what remains in the token stream is

\tempa

(of course, \@nil has been gobbled when the arguments of \@car where fetched). This again is an expandable token, whose current expansion is

\edef bla{bla}

because of the definition we have just performed. So TeX processes \edef, which is unexpandable and is hence executed in the stomach… and, well, I think that now the reason of the error is perfectly clear.

(Deep breath.)