3

I am trying to work with \scantokens, but apparently, I don't really understand it. I want to detokenize some sequence into a register \S:

\let\ea\expandafter
\newtoks\S
\ea\S\ea{\detokenize{\textbf{f}}}

manipulate it (omitted), and retokenize. My first attempt

\ea\def\ea\T\ea{\ea\scantokens\ea{\the\S}}

yields \T=macro: \scantokens {\textbf {f}}.. That's not wrong, but probably we can get the \scantokens out of \T. I thought, expanding once more should do it, but apparently not. So far, also what's written here about \everyeof{\noexpand} does not help me:

\documentclass{article}
\let\ea\expandafter
\newtoks\S
\ea\S\ea{\detokenize{\textbf{f}}}

\ea\def\ea\T\ea{\ea\scantokens\ea{\the\S}}%
\show\T %\T=macro: \scantokens {\textbf {f}}.


%Runaway definition?
%->\textbf {f}
% ! File ended while scanning definition of \R.
\ea\ea\ea\def\ea\ea\ea\R\ea\ea\ea{\T}


%Runaway definition?
%->\textbf {f} \noexpand 
%! File ended while scanning definition of \R.
\everyeof{\noexpand}%
\ea\ea\ea\def\ea\ea\ea\R\ea\ea\ea{\T}
\show\R
\everyeof{}
\begin{document}
    \R
\end{document}
Bubaya
  • 2,279
  • you didn't like my answer to your previous question then? :-) – David Carlisle Jul 05 '23 at 08:36
  • @DavidCarlisle Sure I did, but I want to understand my misconception. I'll accept your answer as soon as I find time to understand why that works ;). Btw., how did you guess the two are related? ():-) – Bubaya Jul 05 '23 at 08:39

2 Answers2

4

I'd do the scantokens on the outside rather than inside the def:

enter image description here

\documentclass{article}
\let\ea\expandafter
\newtoks\S
\S\ea{\detokenize{\textbf{f}}}

\scantokens\ea{\ea\def\ea\T\ea{\the\S}}%
\show\T %\T=macro: \textbf {f}.


\begin{document}
f \T
\end{document}
David Carlisle
  • 757,742
3

The \noexpand approach needs an \edef - \scantokens is 'tricky' (basically buggy). So something like:

\everyeof{\noexpand}%
\edef\T{\noexpand\scantokens{\noexpand\unexpanded{\detokenize{\textbf{f}}}}}
\edef\R{\T}
\show\R
\everyeof{}
\documentclass{article}
\begin{document}
\R
\end{document}

As you might have non-expandable content inside the rescanned tokens, I've wrapped them all in \unexpanded.

(I've somewhat shortened the setup to make it in my opinion clearer).

Joseph Wright
  • 259,911
  • 34
  • 706
  • 1,036
  • What does the \edef do if its content starts with a \noexpand? I guess the two \noexpand in the \edef keep \scantokens and \unexpanded as is until \edef\R{\T}. But what do I need the \unexpanded for? And why is \detokenize expanded already after \edef\T, although it occurs inside \unexpanded? Supposedly, because the \unexpanded is \noexpanded? Probably this all makes sense to me. Except I don't understand why \scantokens did not work with \expandafter. – Bubaya Jul 05 '23 at 08:56
  • @Bubaya \noexpand makes a token temporarily equal to \relax, so in an \edef it does nothing but is unchanged. This is standard TeX programming. The \unexpanded is the as \textbf (or similar) are not safe in an \edef. I guess you could use \protected@edef in LaTeX, but there are still edge cases so this seemed easier: inside \unexpanded, no expansion occurs (it's like a toks but without assignment). – Joseph Wright Jul 05 '23 at 10:00
  • @Bubaya Like I said, \scantokens is essentially buggy - there are ways it works, and ways it doesn't, and you have to pick the correct approach. That's why it's regarded by many people as a bit of a risky primitive (and why LuaTeX has \scantextokens which fixes these issues). – Joseph Wright Jul 05 '23 at 10:02