What can go wrong when using macros as arguments of other macros?

Question

Thats a problem I encountered several times during the last weeks. I try to use macros as arguments of other macros, and it doesn't work.

The most recent example: I try to use \addcontentsline to put a name from a datatool database into the table of contents.

MWE 1:

\documentclass{article}

\usepackage{datatool}

\DTLnewdb{database}

\DTLnewrow{database}
\DTLnewdbentry{database}{shortcut}{JD}
\DTLnewdbentry{database}{name}{John Doe}

\begin{document}

\tableofcontents

%this works and produces the output "John Doe"
\DTLfetch{database}{shortcut}{JD}{name}

%if I use the same command inside \addcontentsline,
%it doesn't work and returns errors
\addcontentsline{toc}{section}{\DTLfetch{database}{shortcut}{JD}{name}}

\end{document}

I'm not (only) asking for a solution to this particular problem, but for a more general explanation what I should consider when using macros as arguments of other macros and what can cause errors when doing so.

I'm sorry if that's a stupid question, but I wasn't able to find any information about that topic, neither in The LaTeX Companion nor the internet, although I spent many hours searching.

EDIT: Please provide a general answer instead of only solving the problems in the MWEs, if possible. I'd like to understand a bit of what causes those errors and how to avoid them, and I imagine there are many unexperienced users out there who might also find this useful.

To illustrate what kind of problems I'm talking about, here is another MWE with a problem similar to MWE 1. I tried to solve it by adding \protect randomly on several places (as egreg did in his answer with MWE 1, except I guess HE knew what he was doing ;-) ), but I couldn't make it work in this way.

MWE 2:

\documentclass{article}

\usepackage{datatool}

\DTLnewdb{database}

\DTLnewrow{database}
\DTLnewdbentry{database}{shortcut}{JD}
\DTLnewdbentry{database}{name}{John Doe}

%command which reads and prints the name belonging to the shortcut
\newcommand{\readname}[1]{\DTLfetch{database}{shortcut}{#1}{name}}

%environment for some text written by the author whose shortcut is in the argument
\newenvironment{authorstext}[1]{%
    \addcontentsline{toc}{subsection}{#1}%when this line is removed, everything works
    {\textbf{#1} has written this text:}%
}{}

\begin{document}

\tableofcontents

\section{Authors}

%this doesn't work
\begin{authorstext}{\readname{JD}}
Some text by John Doe.
\end{authorstext}

%this works
\readname{JD}

\end{document}

Answer 1

\DTLfetch is a fragile command and the third argument to \addcontentsline is a moving argument. So you want

\addcontentsline{toc}{section}{\protect\DTLfetch{database}{shortcut}{JD}{name}}

The problem is that \addcontentsline needs to fully expand its third argument, that might contain data that should be stored with its current value, but might change when the appropriate \write instruction is performed (that is, when TeX is shipping out a page), typically a section number.

With \protect, a fragile command is protected “forever” from expansion and written as is in the .toc file.

On the other hand, \protect does nothing when typesetting is involved, so you can solve your more general second problem by defining

\newcommand{\readname}[1]{\protect\DTLfetch{database}{shortcut}{#1}{name}}

Answer 2

The problem is that expansion of macros are (during normal processing) mixed with primitive commands and this produces desired output. But when a parameter is written to the file, it is only expanded but the primitive (and noexpandable) commands are not processed. They are only rewritten to the file. Suppose, you have the macro:

\newcount\num  \num=0
\def\macro{\num=113 \ifnum\num=113 YES\else NO\fi}

When the \macro is processed during normal processing then the primitive command (assigment) sets \num to 113 and after that the expansion command \ifnum evaluates the conditional as true and YES is printed. But now, suppose that the macro is a parameter which is outputted to the file, for example:

\num=0
\typeout{\macro}

Then \macro is expanded, but \num is unexpandable command (assignment) which isn't processed now (only reprinted) and \ifnum is expandable and it is processed.. The result is

\num =133 NO

If you process this output later on, the assignemt is processed but the result is NO.

Another example. Suppose, that the \macro is defined by the following way:

\def\macro{...\def\foo{something}...}

When normal processing is in progress, then \macro expands to \def\foo{something} and this primitive command is executed, i. e. the \foo si defined. On the other hand, if you try to type \typeout{\macro} then \def command isn't processed (it is only rewritten) and \foo is expanded. But it is typically undefined so you get Undefinded control sequence error.

Finally suppose that the \macro is defined as \nobreak\space and \nobreak is defined by \penalty\@M. Normal processing gives nobreakable space, because \@M is defined as a 10000 constant. When you type \typeout{\macro} then you get primitive commands and declared constants: \penalty\@M followed by expanded space, i.e. invisible space. When you need to read this output again (this is typically done during TOC processing) then you get error \@ is undefined or it isn't constant (because catcode of @ is different at this moment). If the catcode of @ is suitable set then the space is vanished. So there is a problem.

This is a reason why you need to write the \macro unexpanded to the file. This can be done by two different way: the \macro can be prefixed by \noexpand when it is written to the file or it has temporary different meaning (\relax for example). LaTeX uses the first method and provides the \protect command which expands (contextual dependent) to \noexpand or something similar. The \macro itself (not the expansion of the \macro) is stored to the file. When the contents of the file is read again, the \macro is normally processed.

LaTeX provides \DeclareRobustCommand\macro which defines \macro as \protect\internal-version-of-macro. The internal version has the same name but space in the name of the command is appended. I'll not comment this decission of authors of the LaTeX.