A Question About \futurelet

Question

Here is something I don't understand about \futurelet, in this case, into macro \xfl. I can successfully do the archetypal test of

\ifx\xfl x...

to test whether the next token is an x. When true, this would indicate that \xfl replaces to an x. And yet, if I do a

\detokenize\expandafter{\xfl}

I do not get an x as I would expect, but only an unexpandable \xfl.

Why is that? Are there other tests I can perform than \ifx on a \futureletted token? Can't I truly capture (and save) that token?

\documentclass{article}
\usepackage[T1]{fontenc}
\def\fltest{\futurelet\xfl\pdecide}
\def\pdecide{%
  \ifx\xfl x\relax[Next character is x]\else[Not x]\fi\par
  (Can't detokenize\detokenize\expandafter{\xfl})\par
}
\begin{document}
\fltest xyz
\end{document}

Where I would really like to end up is to be able to take the token which has been \futureletted and pass it as an argument to a \readlist. But this code hangs:

\documentclass{article}
\usepackage{listofitems}
\def\fltest{\futurelet\xfl\pdecide}
\def\pdecide{%
  \setsepchar{x}% SEARCH FOR x IN NEXT LIST
  \readlist\mylist{\xfl}\listlen\mylist[]:
}
\begin{document}
\fltest xyz
\end{document}

Note: siracusa rightly notes that \xfl doesn't replace with x, it is x. Nonetheless, there are things I can do with x that I cannot do with \xfl. For example, I can `x to get the ascii value of x, but cannot do a `\xfl to get the same thing.

Note that after a \let or \futurelet the control sequence is basically the same thing as the following token, just with a different name. That's why \ifx yields the true branch, and that's also why you get \xfl back when you use it with \expandafter, because \xfl already is fully expanded. — siracusa, Feb 12 '19 at 03:50
@siracusa Yes, I see that now, but my question still stands...what can I do other than \ifx to probe the quantity? — Steven B. Segletes, Feb 12 '19 at 03:53
This is similar to my question: https://tex.stackexchange.com/questions/474444/convert-token-to-string-in-latex3 (which is not yet answered). — David Purton, Feb 12 '19 at 05:33
@StevenB.Segletes \meaning\xfl might give you more data about what it has inside. — Manuel, Feb 12 '19 at 07:18

Joseph Wright · Accepted Answer · 2019-02-12T16:32:39.093

11

The \let and \futurelet primitives create implicit character tokens. Thus when does

\futurelet\foo\baz a

the token \foo is an a. It is not a macro which expands to an a.

One can see this by using \show and \meaning:

\def\baz{\edef\test{\meaning\foo}\show\foo\show\test}
\futurelet\foo\baz a

You can do anything with \foo that doesn't require an explicit character token. For example, if we do

\let\bgroup={

we can do

\hbox\bgroup <content>

but not

\def\foo\bgroup <definition>

One can use \meaning to disect the text, for example if we know it's a letter

\def\baz{%
  \edef\test{\meaning\foo}%
  \edef\test{\expandafter\bazaux\test\stop}%
  \show\test
}
\edef\bazaux{\def\noexpand\bazaux\detokenize{the letter }##1\noexpand\stop{##1}}
\bazaux
\futurelet\foo\baz a

(One could of course test first for this case.)

edited Feb 12 '19 at 16:32

answered Feb 12 '19 at 08:08

Joseph Wright

259,911
34
706
1,036

Is there a place I can read up a bit more on the distinctions between what one can do with implicit vs. explicit tokens? It seems I am quite limited in what I can do with an implicit token. – Steven B. Segletes Feb 12 '19 at 11:19
1

To be precise, I think the terminology The TeXbook uses is “implicit character” (and “implicit character token”), rather than “implicit token”. (That is, what's implicit is that it's a character. The contrast is with having that character present explicitly, rather than categorizing tokens as implicit and explicit. AFAICT.) – ShreevatsaR Feb 12 '19 at 16:16
1

@ShreevatsaR You are correct: I've updated – Joseph Wright Feb 12 '19 at 16:32
Your answer here, https://tex.stackexchange.com/questions/449597/when-an-implicit-token-is-allowed/449642, seems very related. – Steven B. Segletes Feb 12 '19 at 19:23
You may have interest in my answer to my own question, where I provide some code of a colleague that converts implicit letter/character tokens into macros. – Steven B. Segletes Feb 12 '19 at 21:35

score 2 · Answer 2 · answered Feb 12 '19 at 21:24

My LaTeX colleague, Christian Tellechea, as I was asking him about the limitations of implicit character tokens, quickly shot back a macro (using the listofitems package) that he calls \implicittomacro#1 that takes an implicit token (limited to catcode 11 or 12) and turns it into a macro.

Thus, the MWE I posted at the end of my question becomes do-able with Christian's help. In it, the macro \fltest uses \futurelet to capture the undigested next token, and then can test it against a multiplicity of match characters at once (here, both x and y). This is cooler than a mere \ifx test, which can only compare against a single token at a time.

\documentclass{article}

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{listofitems}
\def\implicittomacro#1{%
\def\next{\expandafter\implicittomacroi\meaning#1\implicittomacroi#1}%
     \unless\ifcat a\noexpand#1%
         \unless\ifcat.\noexpand#1%
             \def\next{\errmessage{\string#1\space is not catcode 11 or 
12}}%
         \fi
     \fi
     \next
}
\expandafter\def\expandafter\implicittomacroi\expandafter
#\expandafter1\detokenize{er} #2\implicittomacroi#3{%
\def\implicittomacroii##1\implicittomacroii{\endgroup\def#3{##1}}%
     \begingroup\endlinechar-1\everyeof{\noexpand}%
\expandafter\implicittomacroii\scantokens{#2\implicittomacroii}%
}

\def\fltest{\futurelet\xfl\pdecide}
\def\pdecide{%
  \implicittomacro\xfl%
  \setsepchar{x||y}% SEARCH FOR x or y AS THE NEXT CHARACTER
  \readlist\mylist{\xfl}%
  \ifnum\listlen\mylist[]>1\relax%
    (Following undigested token is: \mylistsep[1])%
  \else
    (Following undigested token is NEITHER x nor y)%
  \fi
}
\begin{document}
\fltest xyz

\fltest yzx

\fltest zxy
\end{document}

A Question About \futurelet

2 Answers2

Linked