5

I meet the following issue when trying to use \scantokens in section titles, together with the hyperref package (whithout this package, all is fine).

\protect\input... can appear in a section title. But the same construction with \scantokens instead fails ("Second"): it appears that hyperref does not care about protection when creating a pdf string. Using \unexpanded{\protect\scantokens{...}} prevent expansion ("Fifth"), but if we leave out \protect, it crashes again ("Third"). Using two \unexpanded also works ("Fourth").

\documentclass{article}
\usepackage{hyperref}
\begin{document}
\tableofcontents
\section{\protect\input{data}}
% \section{\protect\scantokens{Second}}
% \section{\unexpanded{\scantokens{Third}}}
\section{\unexpanded{\unexpanded{\scantokens{Fourth}}}}
\section{\unexpanded{\protect\scantokens{Fifth}}}
\end{document}

(The file data only contains First (or whatever).) The example above compiles, but removing one of the % will make it fail.

Why do we need two levels of unexpansion/protection? Is there a neater way to do this?

Context: I have written a package (cprotect.dtx) that allows verbatim in the argument of most commands, by writing the argument to a file and \inputting it. Now, it would be nicer to use \scantokens rather than writing dozens of auxiliary files.

Bruno Le Floch
  • 44,937
  • Have you tried looking at the \jobname.toc and \jobname.out files? (Note that the latter is used for PDF Outline stuff, so if you aren't trying to make a PDF it probably won't exist...) – SamB Jan 24 '11 at 21:18
  • @SamB: I think I partially tracked down the problem: \pdfstringdef expands its argument, but does not set \protect to \noexpand\protect\noexpand as most other LaTeX cases of expanded definitions. To see this, we can run TeX on the minimal example --- \documentclass{article} \usepackage{hyperref} \begin{document} \def\AAA{\show\BBB} \pdfstringdef{\foo}{\protect \AAA} \end{document} --- Note that the error is \BBB not defined, meaning that \show was expanded and not executed. – Bruno Le Floch Jan 25 '11 at 11:10

2 Answers2

2

My other answer was the wrong approach. It can't work, because it is essentially impossible to delay expansion across a \pdfstringdef.

Instead, what we have to do is to stop [e-]TeX from signaling a Runaway definition / ! File ended while scanning definition ... at the end the \scantokens; which it does because \scantokens considers the generated tokens to come from a file named " ", and the string argument to \pdfstringdef ends up in the "replacement text" for an \xdef (cleverly disguised as a \Hy@temp), as seen in the following excerpt from \pdfstringdef (see hyperref.dtx / hyperref.pdf for the full source):

      \let\Hy@temp\xdef
      \let\def\HyPsd@DefCommand
      \let\gdef\HyPsd@DefCommand
      \let\edef\HyPsd@DefCommand
      \let\xdef\HyPsd@DefCommand
      \let\futurelet\HyPsd@LetCommand
      \let\let\HyPsd@LetCommand
      \Hy@temp#1{#2}%

(Evidently, it never occurred to Knuth that someone might want to read in a file during the expansion of\edef/\xdef; I didn't even find this exact error message in the index to "TeX: The Program"!)

I thought this was going to be a lot trickier, but they tell me it's simply a matter of inserting a \noexpand before the right brace in the parameter to \scantokens, and they seem to be right. For example, the following generates a PDF with a working outline without triggering any warnings:

\documentclass{article}
\usepackage{hyperref}

\begin{document}

\section{\scantokens{Test\noexpand}}

\end{document}

Caveat

Unfortunately this might not help you very much if you need \catcode reassignments, since those are stomach commands, and the stomach is not available in \pdfstringdef.

If that turns out to be a problem, you might try using \detokenize instead, since that is hard-coded to produce character tokens of categories 12 (for most characters) and 10 (for character 32, the ASCII space character) -- I couldn't help but notice this in the e-TeX manual (see page 14) when I was scouring it for a way to avoid the "Runaway definition" issue.

Community
  • 1
SamB
  • 2,837
  • Thank you. In my case, \cprotect\section{...} reads the argument of \section before feeding it to it with the correct catcodes (this allows verbatim in the argument). Currently, I write to a file, and input it. This requires many auxiliary files (one per \cprotect). I am trying to use \scantokens instead. All this is to say that no, I don't need catcode assignments, and I can put \noexpand at the end of the file/scantokened piece. There are other issues, though (table of contents), that will require one well structured auxiliary file. More work than I want to do right now. – Bruno Le Floch Jan 28 '11 at 21:05
  • @Bruno—I think, but I might be wrong, that \everyeof={\noexpand} would be a better idea that appending the \noexpand manually. – Will Robertson Jan 29 '11 at 02:40
  • @Will: doing \everyeof={\noexpand} locally is not an option, because I need to use \scantokens in an expansion context (e.g., \pdfstring). And I'm afraid that doing it globally might screw up other packages/user \input (?). Also, I already had to append a control word to gobble the extra new line at the end of the file (\scantokens{a}b gives a b), so changing it to \noexpand is easy (of course, I am playing around with non-printable characters to be robust against (most) catcode changes). – Bruno Le Floch Jan 29 '11 at 08:28
  • @Bruno—Sounds sensible. I'm pretty sure you can fix the extra newline problem with \endlinechar=-1 while \scantokens is operating (as done in the expl3 implementation). – Will Robertson Jan 29 '11 at 11:12
  • @Will: I had not tried that, and I think that it would work. But I need expandability. – Bruno Le Floch Jan 29 '11 at 11:21
  • @Bruno: well, surely you could append \noexpand to \everyeof locally to the \cprotect -- which is not to say, by any means, that this wouldn't cause just as much trouble for other packages if they happened to be used in the protected text; merely that the "stomach" would have to be available at that time, since you wouldn't be able to mess with \catcode otherwise... – SamB Jan 29 '11 at 20:00
  • @SamB: it seems cleaner to append ^^E^^L, where ^^E is given catcode escape, and ^^L letter, and define \^^L to be \noexpand: that gobbles the space, and takes care of the end-of-file. As long as I work with files, there is still a risk that \catcode\^gets changed, but with scantokens, it should work. --- I might end up using http://tex.stackexchange.com/questions/9781/are-we-inside-an-edef to fall back towards a detokenized version in an expanding context, and to the right\scantokens` in a non-expanding context. – Bruno Le Floch Jan 30 '11 at 01:04
1

Forget about the following: it won't help; \pdfstringdef will strip any unrecognized control sequences from the expansion -- we MUST expand at that time. Now all we've got to do is figure out why that doesn't work already...

I would suggest adding the appropriate redefinition of \protect to the \pdfstringdefPreHook hook with \pdfstringdefDisableCommands. Probably this would go something like the following:

\pdfstringdefDisableCommands{%
  \def\protect{\noexpand\protect\noexpand}%
}

SamB
  • 2,837
  • (\{ -> {) --- Your solution unfortunately prints hyperref's warnings in the pdf file: \documentclass{article} \usepackage{hyperref} \begin{document} \pdfstringdefDisableCommands{\def\protect{\noexpand\protect\noexpand}} \section{Test} \end{document} – Bruno Le Floch Jan 25 '11 at 17:12
  • @Bruno: Okay, yeah, this is a dead end -- it's not going to work with the approach employed by \pdfstringdef. – SamB Jan 25 '11 at 23:02
  • what worries me is that I am using that for a general purpose package, where I can't easily have a special case for "sections using hyperref". The best is probably to stick with \input, or use plenty of \unexpanded and \protect. But is there a bound on the number that would be needed? – Bruno Le Floch Jan 26 '11 at 01:12
  • @Bruno: that really can't work; any unrecognized control sequences will be discarded before the string is written to the .out file. It must be arranged that your user-level macros be expandable during a \pdfstringdef. I still don't understand why \scantokens inside a section heading is screwing up the .aux... but maybe if we arrange for \protect to be used except in \pdfstringdef, it will work? – SamB Jan 26 '11 at 17:46
  • Note: I'm leaving this answer here, despite its being a dead-end, because I think it could be instructive for others to see why it is a dead end. (For posterity: my other answer explains why \scantokens was failing.) – SamB Jan 29 '11 at 20:08