Scantokens and Let

Question

The package inputenc makes it possible, for example, to write ° to print a degree symbol. Simplifying a bit, it obtains that by making it an active character (catcode 13) and defining the corresponding macro. If its category is changed to other (catcode 12) and, then, changed back with \scantokens, it keeps printing the degree symbol. Since I need to print a string, a character at a time, spaces included, my first idea was of using \let:

\documentclass{article}
\usepackage[ansinew]{inputenc}
\usepackage[T1]{fontenc}

\begin{document}
\setlength\parindent{0pt}

A°@ 1

{\catcode`\°12 \xdef\str{A°@ 1}}
\str

\scantokens\expandafter{\str\empty}

\def\aaa{\afterassignment\bbb\let\ccc= }
\def\bbb{%
  \ifx\ccc\nil
     END
     \let\next\relax
  \else
     [\scantokens\expandafter{\ccc\empty}]\let\next\aaa
  \fi
  \next
}
\expandafter\aaa\str\nil

\end{document}

whose output is:

Unfortunately, as jfbu pointed out, \scantokens does nothing on the \let character and so it prints [ř] instead of [°]. To make it apparent, consider the following:

\documentclass{article}
\usepackage[ansinew]{inputenc}
\usepackage[T1]{fontenc}

\begin{document}
\setlength\parindent{0pt}

{\catcode`\°12 \gdef\ddd{°} \global\let\eee°}

[\ddd] - [\scantokens\expandafter{\ddd\empty}]

[\eee] - [\scantokens\expandafter{\eee\empty}]

\end{document}

whose output is:

Of course, in the first example, one possible solution would be to substitute the \scantokens in square brackets with \ifx\ccc\ddd°\else\ccc\fi, having taken care to add {\catcode`\°12 \global\let\ddd°} at the beginning. The problem with this approach is that it is not very 'scalable'.

My real goal here is to parse some (CP-1252 encoded) files, which can contain any characters, not just °. To put this in context and for the sake of simplicity, let us just suppose I am coding a hex viewer in (La)TeX (which, by the way, would not be such a bad idea).

So, for the textual part, I first 'load' the file with something like \edef\fff#1{\pdfunescapehex{\pdffiledump length \pdffilesize{#1}{#1}}}, which gives all character tokens with catcode 12, except the space (that has catcode 10); then I scan such a token list and act as needed, printing it or else.

So, is there a way to scan a token list, including spaces, without using \let? Assigning catcode 12 also to spaces would suffice, can I do it? How? Alternatively, is there a way of changing the catcode after a \let, which does not involve a lot of conditional expressions? If not, what would be the best (compact and/or elegant) way to do it?

Are you taking into account that ° is not a single token? — Manuel, Nov 23 '17 at 20:21
@Manuel it is a single token in ansinew (cp1252input encoding. This is notutf8`. — , Nov 23 '17 at 21:48
while Manuel's comment isn't strictly accurate note your code here os very fragile, no new documents should be using [ansinew] (which doesn't refer to any standard encoding) if the encoding of the text changes (for example as posted above it is in UTF-8) then the code will fail as ° may be multiple tokens. — David Carlisle, Nov 23 '17 at 21:54
Thank you all for replying so quickly. I am trying to parse some CP-1252 encoded files (which is just an 8-bit extension of ASCII), that is why I am not using UTF-8 here; the code above is just a minimal example. — LeaningTower, Nov 25 '17 at 00:53
The \edef in \edef\fff#1{... \pdffiledump ... {#1}{#1}} can not work. Either you use \def with #1 or you use \edef with for example \jobname.tex in place of #1. — , Nov 27 '17 at 07:37

score 6 · Answer 1 · 2017-11-27T09:52:56.370

When you do the \let\ccc= it will "let" \ccc stand for successive tokens, and they all happen to be non-expandable actually in your \str macro contents. So the \expandafter{\ccc does nothing. And then the \scantokens does nothing either as it sees \ccc, it does not see the ° token. By the way, you don't need \xdef for your \str definition, as there is nothing to expand as you have assigned \catcode12 to ° (in cp1252 code page, char code is 176 and this gives the ř in T1 font encoding.)

After update to original question. But it still isn't quite clear to me what is exactly the context. Here is how one can parse char per char, which is easy as everything but spaces is catcode 12. What I would do is to define macros \@namedef{mymeaning<char>} for special things (where <char> is one catcode 12 token, and perhaps use \ifdefined, and the others \scantokens can be used generally. Here I use \meaning and I separated backslash and percent because \scantokens on the char is no good then.

This is 8bit encoding ONLY method. I illustrate with iso-latin-1. Besides there are no \par token due to the \pdffiledump method.

And the file uses unix line endings.

% -*- coding:iso-latin-1; -*-
\documentclass{article}
\usepackage[latin1]{inputenc}

\def\ParsePerWord #1 {\ifx\ignorespaces#1\ignorespaces
                         <SPACE>\par
                         \expandafter\ParsePerWord
                      \else
                         \if\relax#1\else
                            \ParsePerChar #1\relax
                            \expandafter\expandafter\expandafter\ParsePerWord
                         \fi
                      \fi}
\def\ParsePerChar #1{\if\relax#1\bigskip % end of "word"
                     \else
                        \ifnum`#1=10 % unix end of line
                          <EOL>\par
                        \else
                          \ifnum`#1=92 % backslash
                             <BACKSLASH>\par
                          \else
                             \ifnum`#1=37 % percent
                                 <PERCENT>\par
                             \else
                                 \scantokens{\meaning #1}\par
                             \fi
                          \fi
                        \fi
                        \expandafter\ParsePerChar
                      \fi}

\begin{document}

éà°§ù"£{$\Omega$}\def\x#1{&}

\edef\fff{\pdfunescapehex{\pdffiledump offset 1198 length 100{\jobname.tex}}}

\show\fff

\ttfamily

\makeatletter
\@firstofone{\@firstofone{\expandafter\ParsePerWord\fff} \relax} %

\end{document}

In the above, some spaces (those delimiting a "word") are rendered via a \bigskip and others (spaces at start of lines for example) by <SPACE>\par.

It might be better to do the following

\def\ParsePerWord #1 {\ifx\ignorespaces#1\ignorespaces
                         <SPACE>\par
                         \expandafter\ParsePerWord
                      \else
                         \if\relax#1%
                            <END OF DUMP>\par % <<-- CHANGED !
                         \else
                            \ParsePerChar #1\relax
                            <SPACE>\par %       <<-- CHANGED !
                            \expandafter\expandafter\expandafter\ParsePerWord
                         \fi
                      \fi}
\def\ParsePerChar #1{\if\relax#1% end of "word" <<-- CHANGED !
                     \else
                     .... etc ....

which prints <SPACE>\par for all space characters from input. I also added an <END OF DUMP>\par (and removed the \bigskip).

But there is the problem that the last <SPACE> does not come from original, it is the one we added at end of \fff contents. There are various ways to get rid of it, but I don't know if the parsing loop is supposed to be expandable (so far, it is, it could be used in an \edef, if \par is let to \relax or prefixed by \noexpand to avoid special meanings it can have in a LaTeX document depending on location) so I leave it standing for now.

For example, here is output if \fff is set-up to contain the entire file.

This last <SPACE> does not come from the file contents but from our macro.

Remark: "ParsePerWord" is slightly misleading, because "words" may contain end-of-lines. To be closer to what text editors call words, we should presumably first "ParsePerLine" using not a space but a ^^J delimiter.

So I gather the problem is using \let to scan my string. Therefore, is there a (possibly elegant) way to make it print a degree symbol in square brackets instead of ř? I know I could have used \gdef in place of \xdef. — LeaningTower, Nov 25 '17 at 00:54
with \let things become complicated, of course you could make a giant nested conditional with 256 \ifx to identify original token. Now it is not clear what is your context. In your mwe, I would simply define \str with the space being assigned catcode 12, then I would grab token by token (in the sense of #1) to build a macro with contents [token1][token2 possibly ascii code 32]....etc..., then I would apply a unique \scantokens at the end to those contents (thus, with \expandafter once). I assume you don't have braces ? — , Nov 25 '17 at 07:53
I do have braces, but with catcode 12. I have expanded my question, including also some of your remarks, in the hope of making it clearer for everybody and to provide some context. — LeaningTower, Nov 27 '17 at 02:28
do I understand correctly that you want to parse cp1252 encoded non-TeX related files, from inside a TeX file, which may be or not cp1252 encoded itself ? because that point is still not clear to me. In my answers, the TeX file and the parsed file (which I took to be the .tex file itself) are encoded the same, so a \scantokens will most of the time "print" in PDF output the expected output. If the encodings differ, we have to do additional gymnastics. Then also, why use \pdfunescapehex, it might be simplerwith the hex encoded chars, we can define "cp1252-aware" macros from 2 hex letters. — , Nov 27 '17 at 10:48

Scantokens and Let

1 Answers1