How to \futurelet the token after a space

Question

Suppose we want to process a list of tokens given as a parameter argument, and we want to inspect each token with \futurelet before processing it. How does one \futurelet the token after a space?

Consider the following example:

\def\process#1{\doA#1}
\def\doA  {\show\fl\futurelet\fl\doB}
\def\doB#1{\show\fl\futurelet\fl\doC}
\def\doC  {\show\fl}
\process{ ab}
\bye

We get the following output by the \show commands:

\doA: \fl=undefined    % clearly, \fl hasn't yet been defined
\doB: \fl=blank space  % \fl has been letted to the blank space
                       % If it were a normal token, we could consume it with an argument,
                       % and would have #1=space in \doB and \fl=a in \doC
\doC: \fl=the letter b % However, the space is skipped when parsing the argument,
                       % so we end up with #1=a in \doB and \fl=b in \doC

So how do we get into a state where \fl=the letter a, after having been able to detect and process the space?

Related question: What is the best way to define \myspace such that it compares equal to \fl (when \fl=blank space) via \ifx (or any other if)? The following kind of works (apart from a spurious space).

\def\defmyspace#1{\futurelet\myspace\relax#1}
\defmyspace{ }
\show\myspace          % \myspace=blank space

i am not quite clear what you want here, if you just want to skip over space tokens see the latex definition of \@ifnextchar similarly the latex definition of \@sptoken answers your related question I think. — David Carlisle, Dec 13 '20 at 17:57
@DavidCarlisle I want to iterate over all three tokens in { ab}. \@ifnextchar hides the space and lets me access only a and b. The code in my question gives me access to the space, but I can't find a way to proceed (=skip the space) in a way such that I can futurelet a. Yes, \@sptoken is what I was looking for. It took me a bit to understand why it is defined using \: instead of \somemacro; clever. — gernot, Dec 13 '20 at 20:28

Ulrich Diez · Answer 1 · 2020-12-20T10:04:34.160

Inspired by wipet's answer I suggest a \let-\afterassignment-loop:

\def\process#1{\afterassignment\doA\let\next= #1\process\process}
\def\doA{%
  \ifx\next\process\else
    \message{\meaning\next}%
    \afterassignment\doA
  \fi
  \let\next= %
}
\process{ ab}
\message{done}
\end

(It is assumed that the argument of \process does not contain tokens whose meaning equals the meaning of \process.)

Console output is:

This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=pdftex)
 restricted \write18 enabled.
entering extended mode
(./test.tex blank space   the letter a the letter b done )
No pages of output.
Transcript written on test.log.

To your "related question":

Instead of

\def\defmyspace#1{\futurelet\myspace\relax#1}
\defmyspace{ }
\show\myspace

take into account that \let ... = ... consumes exactly one optional space behind = and do something like:

\def\myspace#1{#1}
\myspace{\let\myspace= } %
\show\myspace

As long as the \lccode of the space-character and the \lccode of the =-character do not indicate that \lowercase affects these characters you can also do:

\lowercase{\let\myspace= } %
\show\myspace

In LaTeX 2ε \@sptoken is available.

As Wipet already provided an excellent answer exhibiting the usage of \let, \futurelet and \afterassignment I decided to provide a totally different approach to the matter—one which probably is suitable if you actually wish to "look" at things argumentwise.

Depending on what you intend to do exactly, you can probably tail-recursively iterate without assignments/without \futurelet/\let:

%%-----------------------------------------------------------------------------
%% Paraphernalia:
%%.............................................................................
\long\def\firstoftwo#1#2{#1}%
\long\def\secondoftwo#1#2{#2}%
\long\def\exchange#1#2{#2#1}%
\chardef\stopromannumeral=`\^^00%
\secondoftwo{}{\long\def\removespace} {}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \CheckWhetherNull{<Argument which is to be checked>}%
%%                  {<Tokens to be delivered in case that argument
%%                    which is to be checked is empty>}%
%%                  {<Tokens to be delivered in case that argument
%%                    which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\long\def\CheckWhetherNull#1{%
  \romannumeral\expandafter\secondoftwo\string{\expandafter
  \secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \secondoftwo\string}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\expandafter\stopromannumeral\secondoftwo}%
  {\expandafter\stopromannumeral\firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is an explicit 
%% catcode-1-character-token
%%.............................................................................
%% \CheckWhetherBrace{<Argument which is to be checked>}%
%%                   {<Tokens to be delivered in case that argument
%%                     which is to be checked has leading
%%                     explicit catcode-1-character-token>}%
%%                   {<Tokens to be delivered in case that argument
%%                     which is to be checked does not have a leading
%%                     explicit catcode-1-character-token>}%
\long\def\CheckWhetherBrace#1{%
  \romannumeral\expandafter\secondoftwo\expandafter{\expandafter{%
  \string#1.}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\expandafter\stopromannumeral\firstoftwo}%
  {\expandafter\stopromannumeral\secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with a space-token
%%.............................................................................
%% \CheckWhetherLeadingExplicitSpace{<Argument which is to be checked>}%
%%                                  {<Tokens to be delivered in case
%%                                     <argument which is to be checked>
%%                                    does have a leading explicit space-
%%                                    token>}%
%%                                  {<Tokens to be delivered in case
%%                                     <argument which is to be checked>
%%                                    does not have a leading explicit 
%%                                    space-token>}%
\long\def\CheckWhetherLeadingExplicitSpace#1{%
  \romannumeral\CheckWhetherNull{#1}%
  {\expandafter\stopromannumeral\secondoftwo}%
  {%
    % Let's nest things into \firstoftwo{...}{} to make sure they are nested in braces
    % and thus do not disturb when the test is carried out within \halign/\valign:
    \expandafter\firstoftwo\expandafter{%
      \expandafter\expandafter\expandafter\stopromannumeral
      \romannumeral\expandafter\secondoftwo
      \string{\CheckWhetherLeadingExplicitSpaceB.#1 }{}%
    }{}%
  }%
}%
\long\def\CheckWhetherLeadingExplicitSpaceB#1 {%
  \expandafter\CheckWhetherNull\expandafter{\firstoftwo{}#1}%
  {\exchange{\firstoftwo}}{\exchange{\secondoftwo}}%
  {\expandafter\expandafter\expandafter\stopromannumeral
   \expandafter\expandafter\expandafter}%
  \expandafter\secondoftwo\expandafter{\string}%
}%
%%-----------------------------------------------------------------------------
%% Extract first inner undelimited argument:
%%
%%   \romannumeral\ExtractFirstArgLoop{ABCDE\SelDOm} yields  A
%%
%%   \romannumeral\ExtractFirstArgLoop{{AB}CDE\SelDOm} yields  AB
%%
%%   \romannumeral\ExtractFirstArgLoop{\SelDOm ABCDE\SelDOm} yields  \SelDOm
%%.............................................................................
\long\def\RemoveTillSelDOm#1#2\SelDOm{{#1}}%
\long\def\ExtractFirstArgLoop#1{%
  \expandafter\CheckWhetherNull\expandafter{\firstoftwo{}#1}%
  {\expandafter\stopromannumeral\secondoftwo{}#1}%
  {\expandafter\ExtractFirstArgLoop\expandafter{\RemoveTillSelDOm#1}}%
}%
%%-----------------------------------------------------------------------------
%%
\long\def\process#1{%
  \CheckWhetherNull{#1}{}{%
    \CheckWhetherLeadingExplicitSpace{#1}{%
      The first token of {\tt\string\process} is an explicit space token.\hfill\break
      \expandafter\process\expandafter{\removespace#1}%
    }{%
       \CheckWhetherBrace{#1}{%
          The first token of {\tt\string\process} is an opening brace.\hfill\break
          \expandafter\process\expandafter{\romannumeral\ExtractFirstArgLoop{#1\SelDOm}}%
          The first token of {\tt\string\process} is a closing brace.\hfill\break
       }{%
          The first token of {\tt\string\process} is: {\tt\expandafter\string\romannumeral\ExtractFirstArgLoop{#1\SelDOm}}\hfill\break
       }%
       \expandafter\process\expandafter{\firstoftwo{}#1}%
    }%
  }%
}%
\noindent{\tt\string\noindent}\hfill\break
{\tt\string\process\string{ ab\string{cd\string}e f \string\TeX\string}}
\bigskip
\noindent
\process{ ab{cd}e f \TeX}%
\bye

From a comment of yours I learned that the issue is about stringification of an argument in LaTeX 2ε.

Be aware that the approach above is not suitable, e.g., for proper stringification of arguments because matching curly braces are consumed without actually examining the closing brace.

I can offer a stringification-routine which examines opening- and closing braces. (Instead of {₁/}₂ there could be something like X₁/Y₂ or even worse ␣₁/␣₂ ...)

%% Copyright (C) 2019, 2020 by Ulrich Diez (eu_angelion@web.de)
%%
%% This work may be distributed and/or modified under the
%% conditions of the LaTeX Project Public Licence (LPPL), either
%% version 1.3 of this license or (at your option) any later
%% version. (The latest version of this license is in:
%% http://www.latex-project.org/lppl.txt
%% and version 1.3 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.)
%% The author of this work is Ulrich Diez.
%% This work has the LPPL maintenance status 'not maintained'.
%% Usage of any/every component of this work is at your own risk.
%% There is no warranty - neither for probably included
%% documentation nor for any other part/component of this work.
%% If something breaks, you usually may keep the pieces.
\errorcontextlines=10000
\documentclass{article}
\makeatletter

%%=============================================================================
%% Paraphernalia:
%%    \UD@firstoftwo, \UD@secondoftwo,
%%    \UD@PassFirstToSecond, \UD@Exchange, \UD@removespace
%%    \UD@CheckWhetherNull, \UD@CheckWhetherBrace,
%%    \UD@CheckWhetherLeadingSpace, \UD@ExtractFirstArg
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
\newcommand\UD@Exchange[2]{#2#1}%
@ifdefinable\UD@removespace{\UD@Exchange{ }{\def\UD@removespace}{}}%
@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=\^^00}% %%----------------------------------------------------------------------------- %% Check whether argument is empty: %%............................................................................. %% \UD@CheckWhetherNull{&lt;Argument which is to be checked&gt;}% %% {&lt;Tokens to be delivered in case that argument %% which is to be checked is empty&gt;}% %% {&lt;Tokens to be delivered in case that argument %% which is to be checked is not empty&gt;}% %% %% The gist of this macro comes from Robert R. Schneck's \ifempty-macro: %% &lt;https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J&gt; \newcommand\UD@CheckWhetherNull[1]{% \romannumeral\expandafter\UD@secondoftwo\string{\expandafter \UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter \UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@secondoftwo}{% \expandafter\UD@stopromannumeral\UD@firstoftwo}% }% %%----------------------------------------------------------------------------- %% Check whether argument's first token is a catcode-1-character %%............................................................................. %% \UD@CheckWhetherBrace{&lt;Argument which is to be checked&gt;}% %% {&lt;Tokens to be delivered in case that argument %% which is to be checked has a leading %% explicit catcode-1-character-token&gt;}% %% {&lt;Tokens to be delivered in case that argument %% which is to be checked does not have a %% leading explicit catcode-1-character-token&gt;}% \newcommand\UD@CheckWhetherBrace[1]{% \romannumeral\expandafter\UD@secondoftwo\expandafter{\expandafter{% \string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@firstoftwo}{% \expandafter\UD@stopromannumeral\UD@secondoftwo}% }% %%----------------------------------------------------------------------------- %% Check whether brace-balanced argument starts with a space-token %%............................................................................. %% \UD@CheckWhetherLeadingSpace{&lt;Argument which is to be checked&gt;}% %% {&lt;Tokens to be delivered in case &lt;argument %% which is to be checked&gt; does have a %% leading explicit space-token&gt;}% %% {&lt;Tokens to be delivered in case &lt;argument %% which is to be checked&gt; does not have a %% a leading explicit space-token&gt;}% \newcommand\UD@CheckWhetherLeadingExplicitSpace[1]{% \romannumeral\UD@CheckWhetherNull{#1}% {\expandafter\UD@stopromannumeral\UD@secondoftwo}% {% % Let's nest things into \UD@firstoftwo{...}{} to make sure they are nested in braces % and thus do not disturb when the test is carried out within \halign/\valign: \expandafter\UD@firstoftwo\expandafter{% \expandafter\expandafter\expandafter\UD@stopromannumeral \romannumeral\expandafter\UD@secondoftwo \string{\UD@CheckWhetherLeadingExplicitSpaceB.#1 }{}% }{}% }% }% \@ifdefinable\UD@CheckWhetherLeadingExplicitSpaceB{% \long\def\UD@CheckWhetherLeadingExplicitSpaceB#1 {% \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}% {\UD@Exchange{\UD@firstoftwo}}{\UD@Exchange{\UD@secondoftwo}}% {\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter\expandafter\expandafter}% \expandafter\UD@secondoftwo\expandafter{\string}% }% }% %%----------------------------------------------------------------------------- %% Extract first inner undelimited argument: %% %% \UD@ExtractFirstArg{ABCDE} yields {A} %% %% \UD@ExtractFirstArg{{AB}CDE} yields {AB} %%............................................................................. \@ifdefinable\UD@RemoveTillUD@SelDOm{% \long\def\UD@RemoveTillUD@SelDOm#1#2\UD@SelDOm{{#1}}% }% \newcommand\UD@ExtractFirstArg[1]{% \romannumeral\UD@ExtractFirstArgLoop{#1\UD@SelDOm}% }% \newcommand\UD@ExtractFirstArgLoop[1]{% \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}% {\UD@stopromannumeral#1}% {\expandafter\UD@ExtractFirstArgLoop\expandafter{\UD@RemoveTillUD@SelDOm#1}}% }% %%----------------------------------------------------------------------------- %% In case an argument's first token is an opening brace, stringify that and %% add another opening brace before that and remove everything behind the %% matching closing brace: %% \UD@StringifyOpeningBrace{{Foo}bar} yields {{Foo} whereby the second %% opening brace is stringified: %%............................................................................. \newcommand\UD@StringifyOpeningBrace[1]{% \romannumeral \expandafter\UD@ExtractFirstArgLoop\expandafter{% \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter\expandafter \expandafter {% \expandafter\UD@firstoftwo \expandafter{% \expandafter}% \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter\string \expandafter}% \string#1% \UD@SelDOm}% }% %%----------------------------------------------------------------------------- %% In case an argument's first token is an opening brace, remove everything till %% finding the corresponding closing brace. Then stringify that closing brace: %% \UD@StringifyClosingBrace{{Foo}bar} yields: {}bar} whereby the first closing %% brace is stringified: %%............................................................................. \newcommand\UD@StringifyClosingBrace[1]{% \romannumeral\expandafter\expandafter\expandafter \UD@StringifyClosingBraceloop \UD@ExtractFirstArg{#1}{#1}% }% \newcommand\UD@CheckWhetherStringifiedOpenBraceIsSpace[1]{% %% This can happen when character 32 (space) has catcode 1... \expandafter\UD@CheckWhetherLeadingExplicitSpace\expandafter{% \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter\UD@secondoftwo \expandafter{% \expandafter}% \expandafter{% \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter\UD@firstoftwo \expandafter{% \expandafter}% \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter\string \expandafter}% \string#1% }% }% \newcommand\UD@TerminateStringifyClosingBraceloop[2]{% \expandafter\expandafter\expandafter\UD@stopromannumeral% \expandafter\expandafter \expandafter{% \expandafter\string \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter#1% \string#2% }% }% \newcommand\UD@StringifyClosingBraceloopRemoveElement[4]{% \expandafter\UD@PassFirstToSecond\expandafter{\expandafter {\romannumeral\expandafter\UD@secondoftwo\string}{}% \UD@CheckWhetherStringifiedOpenBraceIsSpace{#4}{% \UD@Exchange{\UD@removespace}% }{% \UD@Exchange{\UD@firstoftwo\expandafter{\expandafter}}% }{% \expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter#1% \romannumeral\expandafter\expandafter\expandafter\UD@stopromannumeral \expandafter }% \string#4% }{\expandafter\UD@StringifyClosingBraceloop\expandafter{#2#3}}% }% \newcommand\UD@StringifyClosingBraceloop[2]{% \UD@CheckWhetherNull{#1}{% \UD@CheckWhetherStringifiedOpenBraceIsSpace{#2}{% \UD@TerminateStringifyClosingBraceloop{\UD@removespace}% }{% \UD@TerminateStringifyClosingBraceloop{\UD@firstoftwo\expandafter{\expandafter}}% }% {#2}% }{% \UD@CheckWhetherLeadingExplicitSpace{#1}{% \UD@StringifyClosingBraceloopRemoveElement {\UD@removespace}{\UD@removespace}% }{% \UD@StringifyClosingBraceloopRemoveElement {\UD@firstoftwo\expandafter{\expandafter}}{\UD@firstoftwo{}}% }% {#1}{#2}% }% }% %%----------------------------------------------------------------------------- %% Apply &lt;action&gt; to the stringification of each token of the argument: %% %% \StringifyNAct{&lt;action&gt;}{&lt;token 1&gt;&lt;token 2&gt;...&lt;token n&gt;} %% %% yields: &lt;action&gt;{&lt;stringification of token 1&gt;}% %% &lt;action&gt;{&lt;stringification of token 2&gt;}% %% ... %% &lt;action&gt;{&lt;stringification of token n&gt;}% %% %% whereby &quot;stringification of token&quot; means the result of applying \string %% to the token in question. %% Due to \romannumeral-expansion the result is delivered after two %% \expandafter-chains. %% If you leave &lt;action&gt; empty, you can apply a loop on the list formed by %% {&lt;stringification of token 1&gt;}% %% {&lt;stringification of token 2&gt;}% %% ... %% {&lt;stringification of token n&gt;}% %%............................................................................. \newcommand\StringifyNAct{% \romannumeral\StringifyNActLoop{}% }% %%............................................................................. %% \StringifyNActLoop{{&lt;stringification of token 1&gt;}...{&lt;stringification of token k-1&gt;}}% %% {&lt;action&gt;}% %% {&lt;token k&gt;...&lt;token n&gt;} %%............................................................................. \newcommand\StringifyNActLoop[3]{% \UD@CheckWhetherNull{#3}{% \UD@stopromannumeral#1% }{% \UD@CheckWhetherBrace{#3}{% \expandafter\expandafter\expandafter\UD@Exchange \expandafter\expandafter\expandafter{% \UD@StringifyClosingBrace{#3}% }{% \expandafter\StringifyNActLoop\expandafter{% \romannumeral \expandafter\expandafter\expandafter\UD@Exchange \expandafter\expandafter\expandafter{\UD@StringifyOpeningBrace{#3}}{\StringifyNActLoop{#1}{#2}}% }{#2}% }% }{% \UD@CheckWhetherLeadingExplicitSpace{#3}{% \expandafter\UD@PassFirstToSecond\expandafter{\UD@removespace#3}{% \StringifyNActLoop{#1#2{ }}{#2}% }% }{% \expandafter\UD@PassFirstToSecond\expandafter{\UD@firstoftwo{}#3}{% \expandafter\StringifyNActLoop\expandafter{% \romannumeral% \expandafter\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter\UD@PassFirstToSecond \expandafter\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter{% \expandafter\expandafter\expandafter\string \expandafter\UD@Exchange \romannumeral\UD@ExtractFirstArgLoop{#3\UD@SelDOm}{}% }{\UD@stopromannumeral#1#2}% }% {#2}% }% }% }% }% }% %%............................................................................. %% Now a routine which you can apply as &lt;action&gt; within \StringifyNAct: %%............................................................................. \newcommand\printstringifiedtoken[1]{% A token was stringified as \UD@CheckWhetherLeadingExplicitSpace{#1}{% \fbox{\texttt{\char\ \strut}} (explicit space token\strut)%
  }{%
    \fbox{\texttt{#1\strut}}%
  } 
 .\
}%
%%.............................................................................
%% Now a routine which you can apply when prefering iterating on the result
%% of \StringifyNAct
%%.............................................................................
\newcommand\printstringifiedtokenloop[1]{%
  \ifx\relax#1\expandafter@gobble\else\expandafter@firstofone\fi
  {\printstringifiedtoken{#1}\printstringifiedtokenloop}%
}%
\makeatother
\pagestyle{empty}
\begin{document}
\vspace*{-4cm}\enlargethispage{4cm}
\begin{verbatim}
\noindent
\StringifyNAct{\printstringifiedtoken}{%
  \textbf{\csname @firstofone\endcsname{\LaTeX} is funny.}
}
\end{verbatim}
yields:\bigskip
\noindent
\StringifyNAct{\printstringifiedtoken}{%
  \textbf{\csname @firstofone\endcsname{\LaTeX} is funny.}
}
(The last explicit space token is due to the \verb|\endlinechar|-thingie 
while the state of \LaTeX's reading-apparatus is in state M (middle of line) 
after a curly closing brace. It also is in that state after an opening
curly brace.)
\newpage
\vspace*{-4cm}\enlargethispage{4cm}
\begin{verbatim}
\noindent
\expandafter\expandafter
\expandafter\printstringifiedtokenloop
\StringifyNAct{}{%
  \textbf{\csname @firstofone\endcsname{\LaTeX} is funny.}
}%
\relax
\end{verbatim}
yields:\bigskip
\noindent
\expandafter\expandafter
\expandafter\printstringifiedtokenloop
\StringifyNAct{}{%
  \textbf{\csname @firstofone\endcsname{\LaTeX} is funny.}
}%
\relax
(The last explicit space token is due to the \verb|\endlinechar|-thingie 
while the state of \LaTeX's reading-apparatus is in state M (middle of line) 
after a curly closing brace. It also is in that state after an opening
curly brace.)
\end{document}

Wow, thanks for this elaborate solution that I still have to digest. The original problem is a LaTeX problem, but I find it easier to understand (and debug) using plain TeX, in particular since there is an expl3 layer that is great if everything works but which I find hard to debug. I was hoping for a tiny TeX trick to skip over the space. — gernot, Dec 13 '20 at 20:49
Maybe you want to tackle to original problem, Show math macros and result in a table. My first two solutions where to use \detokenize (introduces unwanted spaces) and applying \string to all tokens (problem with braces and spaces, see the OP's comment). — gernot, Dec 13 '20 at 20:59
Thanks a lot, I noted immediately since I'm still brooding over your solution. — gernot, Dec 13 '20 at 21:16
Thanks for solving the original problem, now I can rest in peace ;-) — gernot, Dec 13 '20 at 22:13
@DonaldArseneau As the link in the comments to \CheckWhetherNull indicates, this macro is an outgrowth of a correspondence that took place between you and Robert R. Schneck in comp.text.tex in 2003. I took the idea shown there to apply \string to the first token of the argument and apply some brace-hacking tricks and just tinkered some \romannumeral-expansion around it.\CheckWhetherBrace is just a variant of \CheckWhetherNull, where the case of the argument being empty is excluded by appending a trailing dot...Thank you for the inspiration from the mentioned usenet correspondence ;-) — Ulrich Diez, Dec 15 '20 at 18:54
@DonaldArseneau "to the first token of the argument" is not sufficiently precise as in case of emptiness of the argument there is no "first token of the argument" and what is "hit" by \string is the closing brace of the braces surrounding the empty argument... I tried to elaborate on how \CheckWhetherNull/CheckWhetherBlank/\CheckWhetherBrace macros work in my answer to Expandable test for an empty token list—methods, performance, and robustness. Your critical comments on my explanations there and suggestions for improvement are welcome. — Ulrich Diez, Dec 15 '20 at 19:14

wipet · Accepted Answer · 2020-12-14T06:34:59.647

6

You can \process{ ab} with \futurelet by following macros:

\def\process#1{\doA#1\end}
\def\doA{\futurelet\fl\doB}
\def\doB{\ifx\fl\end\else \reportfl \afterassignment\doA \fi \let\next= } 
\def\reportfl{\message{\meaning\fl}}
\process{ ab}
\end

I corrected a bug in my code (Ulrich's and Donald's comments) and moved the reporting about \fl to a macro in order to tokens like \if, \fi can be processed too.

edited Dec 14 '20 at 06:34

answered Dec 13 '20 at 21:34

wipet

74,238

Close, but your \end token is left dangling and actually performed! It should be consumed or changed to an innocuous delimiter. – Donald Arseneau Dec 14 '20 at 00:54
1

@DonaldArseneau There does not exist innocuous delimiter. This implies of the concept of TeX: you must process the token list in the input queue and there is no end mark of token lists reserved in TeX. So we need to reserve one token with concrete meaning as end-mark. We do the test by \ifx so the meaning is important, no the control sequence name. If you reserve \def\stop{something crazy} then you cannot be 100% sure that user does not define \def\my{comething crazy} and do \process{a\my b}. \end is defined as something crazy in LaTeX and it has primitive meaning in normal format. – wipet Dec 14 '20 at 05:46
1

You've already followed the first suggestion to "consume" it, fixing the bug. About "innocuous", I just meant something that expands to nothing, but something rarer than \end is certainly desirable. Mathematically you're never 100% sure, but who cares? It is better to go close to 100%, in a statistical sense, rather than give up and choose a commonly-defined token that might be expected to be in the argument. – Donald Arseneau Dec 14 '20 at 07:56

Steven B. Segletes · Answer 3 · 2020-12-14T02:09:30.473

The tokcycle package is set up for cycling through tokens. This is the simplest demonstration, showing that you define what to do with each token...here, I bracket characters, and paren the spaces.

A more sophisticated directive can test the tokens and branch according to their charcode or catcode.

Spaces have their own directive because the way TeX absorbs them them is different, as you noted in your question.

I should note that the normal way one uses tokcycle, the processed tokens are added to a toks register, and you can regurgitate it at the end. Here, because of the simplicity of the demonstration (the tokens did not interact with each other, as arguments, for example), I forewent the \cytoks and just processed the tokens on the fly.

Finally, I will note that the MWE uses the macro form \tokcyclexpress{...} that takes an argument. The package also provides a pseudo-environment form \tokencyclexpress...\endtokencyclexpress.

\documentclass {article}
\usepackage{tokcycle}
\begin{document}
\Characterdirective{[#1]}
\Spacedirective{(#1)}
\tokcyclexpress{ ab}
\end{document}

Here is an example of a slightly more sophisticated character directive, in which a tokens are, additionally, made italic:

\Characterdirective{[\tctestifx{a#1}{\textit{#1}}{#1}]}

SUPPLEMENT

And now, for fun, I demonstrate how tokcycle can track the active and implicit nature of tokens, as well. Note in example 3, the a is not italicized, because it is handled by the \Macrodirective, being defined via \def, rather than \let. The macro directive is set up to provide the token between braces.

\documentclass {article}
\usepackage{tokcycle}
\begin{document}
\newcommand\checkAI{$\ifimplicittok_\mathrm{implicit}\fi
   \ifactivetok^\mathrm{active}\fi$}
\Characterdirective{[\tctestifx{a#1}{\textit{#1}}{#1}]\checkAI}
\Macrodirective{\{#1\}\checkAI}
\Spacedirective{(#1)\checkAI}

\tokcyclexpress{ ab}

2.\let\q a
\tokcyclexpress{ \q b}
3.\catcode`Q=\active
\defQ{a}
\tokcyclexpress{ Qb}
4.\letQa
\tokcyclexpress{ Qb}

\def:{\let\z= }:

\tokcyclexpress{\z Qb}

\catcode`Z=\active

\def:{\let Z= }:
\tokcyclexpress{ZQb}

\tokcyclexpress{\space Qb}

\end{document}

Thanks for drawing my attention to your package. In the past, I could have made good use of it at several occasions. I accepted the other answer because it addresses exactly the question. — gernot, Dec 13 '20 at 22:10
@gernot Understood. I should also note that I did my MWE in LaTeX, while yours was Tex. Note that tokcycle works in TeX as well: \input tokcycle.tex. — Steven B. Segletes, Dec 13 '20 at 22:19

How to \futurelet the token after a space

3 Answers3

Linked