How can I implement a \futurelet that ignores spaces?

Question

This is a question about token processing in TeX. What am I doing wrong when implementing \FutureLetNoSpace?

The book TEX in Practice: Volume III: Tokens, Macros, by Stephan v. Bechtolsheim, really does a good job explaining TeX in detail. However, I ran into a stumbling block that I cannot solve using \tracingall alone.

Could someone demonstrate how I might use the code from the book. I also thought \FutureLetNoSpace is the user-level/document-level command. It seems to imitate \futurelet.

Code

Here I write the text "Hello there!", and inject my token checker immediately after the "t". It checks for the token "h". If found, it should yield a nice log message and "TRUE" in the document.

\documentclass{article}
\usepackage{fontspec}% xelatex

%\tracingall % all hell breaks loose

\catcode`@=11 % or \makeatletter to change category code of @ to 11 and temporarily to access kernel macro \@tabularcr

\long\def\DoLongFutureLet #1#2#3#4{%
  \def\@FutureLetDecide{% hangs here
    #1#2\@FutureLetToken% becomes \ifx#2\@FutureLetToken, which compares two expanded tokens
      \def\@FutureLetNext{#3}%
    \else
      \def\@FutureLetNext{#4}%
    \fi% the \@FutureLetNext gets grabbed into \futurelet below
    \@FutureLetNext
  }%
    \futurelet\@FutureLetToken\@FutureLetDecide
}

\def\DoFutureLet #1#2#3#4{%
  \DoLongFutureLet{#1}{#2}{#3}{#4}
} % identical to \DoLongFutureLet

\def\FutureLetNoSpace #1#2{%
  \def\@FutureLetNoSpaceA{#1}% save arg
  \def\@FutureLetNoSpaceB{#2}% save arg
  \@FutureLetOne
}

\def\@FutureLetOne{%
  \DoFutureLet{\ifx}{ }%
    {\@FutureLetThree}{\@FutureLetOk}% \@FutureLetThree if it is a space token.
}

\edef\@FutureLetNoSpaceTemp{%
  \def\noexpand\@FutureLetThree\space{\noexpand\@FutureLetOne}% force expansion of space into a space token and recall \@FutureLetOne
}
\@FutureLetNoSpaceTemp% why are we calling this macro here?

\def\@FutureLetOk{% called when no space is found
  \expandafter\futurelet\@FutureLetTokenA\@FutureLetTokenB
}

\long\def\DoLongFutureLetNoSpace #1#2#3#4{%
  \def\@FutureLetDecideNoSpace{% \@FutureLetTokenNoSpace is self-contained by this macro
    #1#2\@FutureLetTokenNoSpace% becomes \ifx#2\@FutureLetTokenNoSpace, which compares two expanded tokens
      \def\@FutureLetNextNoSpace{#3}%
    \else
      \def\@FutureLetNextNoSpace{#4}% whatever should get executed on match
     \fi
     \@FutureLetNextNoSpace
    }
    \FutureLetNoSpace{\@FutureLetTokenNoSpace}%
      {\@FutureLetDecideNoSpace}% call \@FutureLetDecideNoSpace instead of \futurelet
}
\def\DoFutureLetNoSpace #1#2#3#4{%
  \DoLongFutureLetNoSpace{#1}{#2}{#3}{#4}%
}

\catcode`@=12 % or \makeatother to restore category code of @ to 12

\begin{document}
Hello t\DoFutureLetNoSpace{\ifx}{h}{\typeout{\noexpand\@FutureLetTokenNoSpace value: \meaning\@FutureLetTokenNoSpace}}TRUE}{\typeout{\noexpand\@FutureLetTokenNoSpace value: \meaning\@FutureLetTokenNoSpace}FALSE}here!
\end{document}

Trace

The last four lines of the trace in the console:

\DoFutureLet #1#2#3#4->\DoLongFutureLet {#1}{#2}{#3}{#4}
#1<-\ifx
#2<-
#3<-\@FutureLetThree

"futurelet that ignores spaces" is more or less the definition of latex's \@ifnextchar — David Carlisle, May 18 '18 at 13:04
@GuM So awesome. Thanks! It is not the same implementation as in TEX in Practice, but useful. It uses \afterassignment like in Manuel's answer. — Jonathan Komar, May 18 '18 at 13:41
What's the problem with \afterassignment? It's more secure to do \def\foo{\afterassignment\nextstep\let\gobble= } than \expandafter\def\expandafter\foo\space{\nextstep}. — Manuel, May 18 '18 at 16:35
@Manuel I have no problem with it. As I said the comments to your answer, it seems more elegant. I am just trying to grasp the code from the book. I thought I understood its recursive calls, but it keeps choking. Safer is better, of course! — Jonathan Komar, May 18 '18 at 16:42
Seeing the code, it seems like overcomplicating things, and I don't see the problem at first sight, In any case you can use Hello t\@ifnextchar{h}{TRUE}{FALSE}here! — Manuel, May 18 '18 at 16:48

Manuel · Accepted Answer · 2018-05-19T07:37:57.197

4

\makeatletter
\def\futureletignorespaces#1#2{\def\nextaction{#2}%
  \def\checkspace{\ifx\nexttoken\@sptoken
    \expandafter\@firstoftwo
   \else
    \expandafter\@secondoftwo
   \fi
   {\afterassignment\futureletagain\let\nexttoken= }
   {\futurelet#1\nextaction}}%
  \futurelet\nexttoken\checkspace}
\def\futureletagain{\futurelet\nexttoken\checkspace}
\makeatother

This started as just a substitute for \futurelet\nexttoken\action ignoring spaces \futureletignorespaces\nexttoken\action. But I changed the code a bit so that it lets you define the second argument on the fly:

\futureletignorespaces\nexttoken\action % or you could define \action in the argument
\futureletignorespaces\nexttoken{\ifx\nexttoken!yes\fi}

edited May 19 '18 at 07:37

answered May 18 '18 at 12:07

Manuel

27,118

Doesn't work yet. – Manuel May 18 '18 at 12:10
+1 for effort haha. I am well aware that my question is a pain. – Jonathan Komar May 18 '18 at 12:13
It does work now. – Manuel May 18 '18 at 12:16
Note that this has the same syntax as \futurelet. \futurelet\nexttoken\action or \futureletignorespaces\nexttoken\action. – Manuel May 18 '18 at 12:17
Then why switch to \afterassignment? I seem to remember one of them puts a token back into the token stream. – Jonathan Komar May 18 '18 at 12:20
There's no “switch”: if the next token is a space token, remove it and call again \ftureletignorespaces. If there are three space tokens it would gobble all before checking what's next (if for some reason you want to leave the space tokens there we can put them back after checking what's next). – Manuel May 18 '18 at 12:43
I mean that you used \afterassignment instead of answering "What am I doing wrong when implementing \FutureLetNoSpace [as defined in TEX in Practice]." Not to imply that your answer is not good--it is arguably more elegant. – Jonathan Komar May 18 '18 at 13:49
Oh, true, I didn't read your question, just the title. – Manuel May 18 '18 at 14:04

Steven B. Segletes · Answer 2 · 2021-08-24T17:24:30.547

While this problem is older and solved, I wanted to see if I could get my tokcycle package to perform this function. I employed my recently introduced package functionality of popping, pushing, and peeking (futureletting) information from and to the input stream. I even used it to push the cycle-ending \endtokcycraw onto the input stream, which allowed for an undelimited-argument approach required of \futurelet.

In the MWE, I \let at one point an active-Q to be \mytest, so that I can easily introduce space tokens following \mytest, yet before the tokens we want futureletted. In the MWE, the input stream is allowed to execute and then, at the end of the line, I show in parens, the token that was peeked at with \futureletignorespaces.

The logic is as follows, \futureletignorespaces, when invoked, absorbs one argument (#1), the token into which the \futurelet is to be placed. It then starts the token cycle (looking eventually for an \endtokcycraw to close out the cycle). It absorbs the next token in the input stream into the cycle, which is the token to be executed upon completion of the futurelet. If this absorbed token is cat-0, it will be passed to the \Macrodirective as "#1" (or in this case, as ####1). If the absorbed token is a character, it is passed to \Characterdirective. In either case, I want to perform the same actions, which are characterized in my definition of \mydirective.

It absorbs white space and places the result of that into \spacesbsorbed. It then performs the \futurelet by way of \tcpeek#1 (where #1 was the argument passed to \futureletignorespaces, the specified destination for the futurelet). It then makes this token-let global.

Now, it starts reassembling the input stream in LIFO fashion: It pushes back any white space that was earlier absorbed. It then pushes back the ####1, which is the token to be executed at the conclusion of the process. Finally, it pushes \endtokcycraw onto the input stream, so that when it is absorbed in the very next iteration of the token cycle, it will terminate the cycle. The use of the leading \empty in two of those pushes is because \tcpush is expecting to push something that was earlier popped into a macro (requiring one expansion to recover the replacement text), whereas I want it to push literal tokens. Thus, the \empty gets expanded, leaving the desired literal token to be pushed.

Having defined the cycle directives thus, the token cycle is unleashed with \tokencyclexpress, which takes no directive-arguments, instead using predefined directives, which I described above.

\documentclass{article}
\usepackage{tokcycle}
\newcommand\futureletignorespaces[1]{%
 \def\mydirective{%
  \tcpopwhitespace\spaceabsorbed
  \tcpeek#1%
  \global\let#1#1%
  \tcpush\spaceabsorbed
  \tcpush{\empty####1}%
  \tcpush{\empty\endtokcycraw}}%
 \expandafter\Characterdirective\expandafter{\mydirective}%
 \expandafter\Macrodirective\expandafter{\mydirective}%
 \tokencyclexpress
}
\def\mytest{\ifx*\nexttok\expandafter\mytestSTAR
  \else\expandafter\mytestNOSTAR\fi}
\def\mytestSTAR#1#2{STAR VERSION, ARGUMENT ``#2''}
\def\mytestNOSTAR#1{NOSTAR VERSION, ARGUMENT ``#1''}
\begin{document}
\catcode`\Q=\active
\letQ\mytest
\futureletignorespaces\nexttok Q{xyz} (\ifx\bgroup\nexttok\{\else\nexttok\fi)
\futureletignorespaces\nexttok Q {xyz} (\ifx\bgroup\nexttok{\else\nexttok\fi)
\futureletignorespaces\nexttok Q*{xyz} (\ifx\bgroup\nexttok{\else\nexttok\fi)
\futureletignorespaces\nexttok Q *{xyz} (\ifx\bgroup\nexttok{\else\nexttok\fi)
\futureletignorespaces\nexttok Xy (\nexttok)
\futureletignorespaces\nexttok X y (\nexttok)
\futureletignorespaces\nexttok Xxy (\nexttok)
\futureletignorespaces\nexttok X xy (\nexttok)
\end{document}

For a direct comparison, this is the output if \futureletignorespaces is replaced with \futurelet in the document. The even-numbered lines differ, because the \futurelet sees the space, not the token following the space.

SUPPLEMENT

Since the question is tagged tex-core, I can show that the tokcycle approach works in plain tex as well:

\input tokcycle
\def\futureletignorespaces#1{%
 \def\mydirective{%
  \tcpopwhitespace\spaceabsorbed
  \tcpeek#1%
  \global\let#1#1%
  \tcpush\spaceabsorbed
  \tcpush{\empty####1}%
  \tcpush{\empty\endtokcycraw}}%
 \expandafter\Characterdirective\expandafter{\mydirective}%
 \expandafter\Macrodirective\expandafter{\mydirective}%
 \tokencyclexpress
}
\def\mytest{\ifx*\nexttok\expandafter\mytestSTAR
  \else\expandafter\mytestNOSTAR\fi}
\def\mytestSTAR#1#2{STAR VERSION, ARGUMENT ``#2''}
\def\mytestNOSTAR#1{NOSTAR VERSION, ARGUMENT ``#1''}
\catcode`\Q=\active
\letQ\mytest
\futureletignorespaces\nexttok Q{xyz} (\ifx\bgroup\nexttok\string{\else\nexttok\fi)
\futureletignorespaces\nexttok Q {xyz} (\ifx\bgroup\nexttok\string{\else\nexttok\fi)
\futureletignorespaces\nexttok Q*{xyz} (\ifx\bgroup\nexttok\string{\else\nexttok\fi)
\futureletignorespaces\nexttok Q *{xyz} (\ifx\bgroup\nexttok\string{\else\nexttok\fi)
\futureletignorespaces\nexttok Xy (\nexttok)
\futureletignorespaces\nexttok X y (\nexttok)
\futureletignorespaces\nexttok Xxy (\nexttok)
\futureletignorespaces\nexttok X xy (\nexttok)
\bye

I like multiple approaches! Thanks for exploring this possibility. — Jonathan Komar, Aug 24 '21 at 16:53

How can I implement a \futurelet that ignores spaces?

Code

Trace

2 Answers2

Linked