How to do \scantokens inside \edef without triggering "Runaway definition"

Question

I'm trying to find a way to use the e-TeX \scantokens command inside [the equivalent of] an \edef (well, actually an \xdef, but presumably this does not make any real difference since that's just the \global version of \edef). The reason I am doing this is that it seems to be needed to answer this question about hyperref and pdf outlines.

The problem seems to be that \scantokens doesn't actually expand to the tokens scanned; it actually expands to nothing, but arranges for the next tokens to be read from a pseudo-file named " ", which is then treated almost exactly like a real file, including the check for "runaway" conditions. (See page 10 of the e-TeX manual.)

There is one problem with this explanation, though: why would \input work, as it evidently does in the other question?

Anyway, here's a fairly minimal example:

%&eplain

% make \scantokens pseudo-files show up with name " " in the output
\tracingscantokens=1

\edef\foo{\scantokens{Test}}

Which produces the output

Running `TeX' on `scantokens-in-edef-minimal' with ``pdftex  -interaction=nonstopmode "\input" "scantokens-in-edef-minimal"''
This is pdfTeX, Version 3.1415926-1.40.11 (MiKTeX 2.9)
entering extended mode
(e:\home\tex\scantokens-in-edef-minimal.tex ( )
Runaway definition?
->Test 
! File ended while scanning definition of \foo.
<inserted text> 
                }
l.6 \edef\foo{\scantokens{Test}
                               }
! Too many }'s.
l.6 \edef\foo{\scantokens{Test}}

)
! Emergency stop.
<*> \input scantokens-in-edef-minimal

!  ==> Fatal error occurred, no output PDF file produced!
Transcript written on scantokens-in-edef-minimal.log.

TeX exited abnormally with code 1 at Thu Jan 27 12:24:57

Is it perhaps possible to somehow do the expansion with macros and then do a \def instead of the \edef, so the expansion would be finished before the definition began, and the "runaway definition" condition avoided?

Answer 1

In short, yes. Try

\edef\foo{\scantokens{Test\noexpand}}

or

\everyeof={\noexpand}
\edef\foo{\scantokens{text}}

in which the \noexpand ‘hides’ the EOF marker by turning it into \relax.

You might like to take a look at the definition of expl3's \tl_rescan:nn and \tl_set_rescan:Nnn, both wrappers for \scantokens based on the ideas in Heiko Oberdiek's catchfile package.

Then you can write

\tl_set_rescan:Nnn \foo { «catcode setup» } { «rescanned tokens» }

or

\tl_rescan:nn { «catcode setup» } { «tokens to re-scan (in a group)» }

The latter cannot be used inside an \edef but can otherwise come in handy when you're processing a macro argument that has already been tokenised. In practise I find the set_rescan version a little more convenient.