Why doesn't \makeatletter work inside \newcommand?

Question

After hours of banging my brain against my skull, I narrowed down an issue I'm having to the fact that \makeatletter doesn't work inside \newcommand. Commands that I call after \makeatletter are registered as undefined, as if \makeatletter didn't do anything.

Why is this the case?

In the example below, I define two commands. Calling one of them from the document body causes the undefined error, while the other doesn't.

\documentclass[11pt]{book}
\usepackage{kvsetkeys}

\newcommand{\commandThatFails}{
    \makeatletter \comma@parse{}{} \makeatother
}

\makeatletter
    \newcommand{\commandThatWorks}{
        \comma@parse{}{}
    }
\makeatother

\begin{document}

Hi there.

\commandThatFails
\commandThatWorks

\end{document}

Because you can't change the catcode of @ inside the body of the document. The second version is the correct one since it allows you to include control sequences with @ character and doesn't cause any problem upon execution. The first one is trying to change the catcode upon execution and failing. — percusse, Sep 16 '14 at 00:06
@percusse I'm not sure that's the correct explanation… I mean, it partially is, but it's a bit of a mess. — Manuel, Sep 16 '14 at 00:07
@percusse I know why that happens but I'm not sure I know how to explain it in english without writing a whole paragrph. The thing is that tokens are stored when the macro is defined, in the second one you store \comma@parse, but in the first one, since \makeatletter is only saved (not executed) the following that is saved are the tokens \comma, @, p, a, r, s, e. — Manuel, Sep 16 '14 at 00:13
this must be a duplicate, but it's the same reason you can't use \verb in a command argument. catcodes change the tokenisation of characters they do not affect tokens tat have already been parsed. when the argument to \newcommand is scanned you get token \makeatletter token \comma token @ token p token a token r token s token e when the command is executed the catcode of @ is changed but that has no affect on tokens — David Carlisle, Sep 16 '14 at 00:14
@Manuel Oh in that sense. Yes I meant as a cs parsing. My bad you are right, wrong wording. — percusse, Sep 16 '14 at 00:16
@percusse you can change the catcode at any point in the document or in the preamble, but catcodes don't affect tokens. — David Carlisle, Sep 16 '14 at 00:16
@Manuel I leave the floor to you since I deserve to be punished :) Can you also add that \csname comma@parse \endcsname doesn't require cat change? — percusse, Sep 16 '14 at 00:20
Related/duplicate: http://tex.stackexchange.com/questions/73851/can-the-makeatletter-makeatother-mechanics-lead-to-parsing-problems — egreg, Sep 16 '14 at 09:39

score 44 · Accepted Answer · edited Feb 03 '20 at 17:43

When \makeatletter is executed, it changes the catcode of @ to 11 (letter) so it can be part of a macro name. However, when defining a macro or using an argument of a macro, the tokens are stored without being executed.

So here,

\newcommand{\commandThatFails}{
    \makeatletter \comma@parse{}{} \makeatother
}

\makeatletter is stored as a token in the command rather than being executed. Because @ is still a non-letter, the command parses as the tokens \comma, @, p, a, r, s, and e, which is not \comma@parse

However, in your second example, \makeatletter is executed before the command is defined, so that \comma@parse{}{} is parsed correctly:

\makeatletter
    \newcommand{\commandThatWorks}{
        \comma@parse{}{}
    }
\makeatother

This is for the same reason that \verb|...| fails within arguments: it’s not executed, just stored, causing any strange characters to cause all sorts of errors.

As noted by @percusse, in this cases where you need @-macros just once or twice, you can use \csname comma@parse\endcsname which expands all the macros and then converts every token between \csname and \endcsname to a macro name. So this would also work:

\newcommand{\commandThatDoesntFail}{\csname comma@parse\endcsname{}{}}

Where the command resolves to \comma@parse{}{} (note that the tailing {}{} is treated as arguments to \comma@parse, not \endcsname).

Why doesn't \makeatletter work inside \newcommand?

1 Answers1

Linked

Related