Delaying a sequence of tokens via \expandafter

Question

Use case: define a command that loops over a list of input names, defining a command from each name.

In the working code below, note the \expandafter\definput\expandafter{\x} within a loop with \x as the loop variable.
The necessity is clear: we need to expand \x to each value in the loop range before expanding \definput on each one.

As the programmer, I wanted to say "Please substitute the loop value for \x everywhere before expanding the rest of the loop body."
To do so is non-intuitive, requiring an explicit "Wait to expand me" on every other macro in the loop body that ends up consuming the loop variable.

To complicate this, the tool to delay expansion is \expandafter, which only operates on the following 2 tokens, so you need to keep chaining \expandafter until you reach the loop variable.
Written functionally, the solution is: expandafter('\definput', expandafter('{', '\x')) (followed by normal token '}').
This expands \x to each iteration's value, then { to itself, then \definput, consuming {, the iteration value, and }.

Coming from other programming languages where you can easily group things together, this input chaining feels like pulling teeth.

Questions:

• Is there any way to group together the \definput and { to say "don't expand this group" with one \expandafter?
• Is this problem fundamental to LaTeX's token-processing design?

\documentclass[11pt]{article}
\usepackage{tikz}
% On input x, define \x as "the definition of x"
\newcommand{\definput}[1]{%
  \expandafter\gdef\csname#1\endcsname{the definition of #1}%
}
\newcommand{\defallinputs}[1]{\foreach \x in {#1} {\expandafter\definput\expandafter{\x}}}
% Define \x and \y
\defallinputs{x, y}
\begin{document}
\x\par\y
\end{document}

becomes:

the definition of x
the definition of y

For reference, the use case that inspired this question is: Using \xdef in \foreach errors when using another \foreach inside the \xdef

Answer 1

It is easier to control expasion using expl3, also you can use mapping forms that do not define loop variables that need expanding

\documentclass[11pt]{article}
\ExplSyntaxOn
\def\defallinputs#1{
   \clist_map_inline:nn{#1}{\cs_new:cpn{##1}{the~ definition~ of~ ##1}}}
\ExplSyntaxOff
% Define \x and \y
\defallinputs{x, y}
\begin{document}
\x\par\y
\end{document}

Answer 2

With \expandafter you anticipate, not delay, expansion.

The \foreach loop was developed in PGF in order to do repetitive tasks, typicall adding statements to a tikzpicture.

In this way the \x (or whatever control sequence you use) is “used” immediately because the statements are immediately executed and the result (in some internal format) is appended to the list of graphic instruction to be delivered when \end{tikzpicture} is processed.

With time, \foreach started to be used also for other tasks, but the \x is a quite big limitation. The other known limitation is that each cycle is processed inside a group.

A common problem (and I believe it's what you're trying to do) is to define commands for letters in some math alphabet. Say you want do define all at once

\C \F \H \N \Q \R \Z

where each stands for \mathbb{<letter>}. You can do it with \foreach, but it's clumsy. Of course the attempt

\foreach \x in {C,F,H,N,Q,R,Z}{%
  \expandafter\gdef\csname\x\endcsname{\mathbb{\x}}%
}

won't work, because it results in all the commands to have replacement text \mathbb{\x}.

For this particular case you might do

\foreach \x in {C,F,H,N,Q,R,Z}{%
  \expandafter\gdef\csname\x\expandafter\endcsname\expandafter{\expandafter\mathbb\expandafter{\x}}%
}

Gasp! In your case, the number of \expandafter tokens would be embarassingly high.

You might use tricks of argument inversion, but there are much better ways.

\documentclass{article}
\usepackage{amssymb}
\ExplSyntaxOn
\NewDocumentCommand{\definelettercommands}{O{}mO{}m}
 {% #1 = optional prefix
  % #2 = list of letters
  % #3 = optional postfix
  % #4 = template
  \fink_dll:nnnn { #1 } { #2 } { #3 } { #4 }
 }
\cs_new_protected:Nn \fink_dll:nnnn
 {
  \clist_map_inline:nn { #2 }
   {
    \cs_set_protected:Nn __fink_dll_temp: { #4 }
    \cs_new_eq:cN { #1 ##1 #3 } __fink_dll_temp:
   }
 }
\ExplSyntaxOff
\definelettercommands{C,Q,R}{\mathbb{#1}}
\definelettercommands[c]{A,B,C}{\mathcal{#1}}
\definelettercommands{a,b,c}[frak]{\mathfrak{#1}}
\begin{document}
$\C\Q\R$
$\cA\cB\cC$
$\afrak\bfrak\cfrak$
\end{document}

Here #1 in the fourth argument will become the current item in the loop, because by rule it becomes ##1 when passed to \clist_map_inline:nn, so in the first case you get the equivalent of

\clist_map_inline:nn { #1 ##1 #3 } { \mathbb{##1} }

The indirection via first defining \__fink_del_temp: and then using \cs_new_eq:cN is in order to get better error messages and recovery in case a command is already defined.

It takes some time to digest this abstraction, but it's rewarding.

Without the “complication” of prefixes and postfixes and not using indirections it would be

\ExplSyntaxOn
\NewDocumentCommand{\definelettercommands}{mm}
 {
  \clist_map_inline:nn { #1 }
   {
    \cs_new_protected:cpn { ##1 } { #2 }
   }
 }
\ExplSyntaxOff

and the call \definelettercommands{C,Q,R}{\mathbb{#1}} would become

\clist_map_inline:nn {C,Q,R}
 {
  \cs_new_protected:nn { #1 } { \mathbb{#1} }
 }

where you recognize the analog of your \foreach cycle, but without the pesky \x.

Answer 3

Coming from other programming languages where you can easily group things together, this input chaining feels like pulling teeth.

I believe you. ;-) However TeX was not intended to be a programming language. It was intended to be a typesetting language. The programming paradigm of TeX differs from the programming-paradigm of functional/procedural/object-oriented programming-languages. You do yourself a favor by not attempting to transfer concepts/terminology of programming languages like Pascal, C, C++, Java, whatsoever to TeX but strictly sticking to the concepts and terminology introduced in Donald E. Knuth's TeXbook while in the stage of getting acquainted to TeX.

Questions:

• Is there any way to group together the \definput and { to say "don't expand this group" with one \expandafter?

No. \definput and { are two tokens. As long as it is about expansion and macro-programming of TeX think about TeX as if it was a factory and think of tokens as if they were items that are placed one behind another on an assembly line.
The assembly line goes through several departments of the factory where different things are done.
In the expansion-department expandable tokens and those subsequent tokens that form their arguments are removed from the assembly line and in their place tokens are put on the assembly line that—according to the macro-definition or the meaning of the expandable token in question—form the replacement text. "text" might be misleading as the replacement text is also made from tokens.
In the expansion-department it is all about nice little items (tokens) on the assembly line that get removed from the assembly line/replaced by other nice little items.
If in the expansion-department \expandafter is encountered, it is removed from the assembly-line and then TeX does not focus on the next token on the assembly line but on the next but one token on the assembly line, i.e., expands the next but one token if that is expandable so that on the assembly line you have the next token trailed by either the next but one token (in case that is not expandable) or by those tokens that make the replacement of the next but one token (in case the next but one token was expandable).

Seems with "grouping together" you mean a mechanism similar to \expandafter but causing TeX to focus at the next but second token instead of focusing at the next but one token.
TeX does not have such a facility. It is not needed.

As you already noticed yourself in your question you can chain \expandafter: \expandafter\definput\expandafter{\x}.

If you wish to avoid long \expandafter-chains or if the length of the token-sequence which is to be chained is not predictable, e.g., as it comes from a macro argument that might hold tokens coming from arbitrary user-input, you can exchange macro-arguments after applying a short \expandafter-chain—e.g.:

\expandafter\tokenA\expandafter\tokenB\expandafter\tokenC\expandafter\tokenD
\expandafter\tokenE\expandafter\tokenF\expandafter\tokenG\expandafter\tokenH
\expandafter\tokenI\expandafter\tokenJ\expandafter\tokenK\expandafter\tokenL
\expandafter\tokenM\expandafter\tokenN\expandafter\tokenO\expandafter\tokenP
\expandafter\tokenQ\expandafter\tokenR\expandafter\tokenS\expandafter\tokenT
\expandafter\tokenU\expandafter\tokenV\expandafter\tokenW\expandafter\tokenX
\expandafter\tokenY\expandafter\tokenZ\ExpandThisFirst{Argument A}{Argument B}%

yields the same as

\newcommand\ExchangeArgs[2]{#2#1}%
%
\expandafter\ExchangeArgs
\expandafter{\ExpandThisFirst{Argument A}{Argument B}}%
            {\tokenA\tokenB\tokenC\tokenD\tokenE\tokenF\tokenG\tokenH\tokenI\tokenJ
             \tokenK\tokenL\tokenM\tokenN\tokenO\tokenP\tokenQ\tokenR\tokenS\tokenT
             \tokenU\tokenV\tokenW\tokenX\tokenY\tokenZ}%

If you are picky about the amount of expansion-steps that need to be triggered for obtaining the desired resulting set of tokens—with the further only one expansion-step needs to be triggered on the first \expandafter, with the latter two expansion-steps need to be triggered, the first one triggering expanding the \expandafter-chain, the second one triggering expansion of \ExchangeArgs, you can do s.th. like:

\newcommand\ExchangeArgs[2]{#2#1}%
\csname @ifdefinable\endcsname\stopromannumeral{\chardef\stopromannumeral=`\^^00}%
%
\romannumeral
\expandafter\ExchangeArgs
\expandafter{\ExpandThisFirst{Argument A}{Argument B}}%
            {\stopromannumeral
             \tokenA\tokenB\tokenC\tokenD\tokenE\tokenF\tokenG\tokenH\tokenI\tokenJ
             \tokenK\tokenL\tokenM\tokenN\tokenO\tokenP\tokenQ\tokenR\tokenS\tokenT
             \tokenU\tokenV\tokenW\tokenX\tokenY\tokenZ}%

This way also only one expansion-step needs to be triggered on the token \romannumeral as \romannumeral itself triggers expansion until finding \stopromannumeral which denotes a TeX-⟨number⟩-quantity whose value is not positive so that \romannumeral triggers its removal from the token-assembly-line without delivering any tokens in return.

You might be interested in How can I know the number of expandafters when appending to a csname macro?

• Is this problem fundamental to LaTeX's token-processing design?

Yes. LaTeX's token-processing design in turn is an aspect of TeX's underlying programming paradigm.

Btw: I might probably try s.th. like this:

\documentclass[11pt]{article}
\usepackage{tikz}
\newcommand{\defallinputs}[1]{%
  \foreach \x in {#1} {%
    \expandafter\gdef\csname\x\expandafter\endcsname\expandafter{\x}%
  }%
}
% Define \x and \y
\defallinputs{x, y}
\begin{document}
\x\par\y
\end{document}

Answer 4

In general, if you want to expand 'after some tokens', you end up using an auxiliary macro to do the work. As David has commented, expl3 provides a large library of macros for expansion control and defined expansion behaviours. For a simple case of 'expand \x then do stuff', one could do something like

\def\foo#1{\expandafter\fooaux\expandafter{#1}}
\def\fooaux#1{<code>}
\foo\x