Basics of parsing

Question

I'm trying to master the art of parsing text one character at a time. The following is what I've discovered so far. Is this as good as it gets? I would really prefer not to do all those \edefs as they waste RAM.

\documentclass{article}
\tracingmacros=1

\newcommand{\scanA}[1]% #1 = text to scan
{\futurelet\token\scanB#1 % does nothing for me
 \scanC#1 % splits off first token
 \edef\next{#1\relax}%
 \expandafter\scanD\next% refined version
 \expandafter\scanD\next% okay, scaning underway
 \expandafter\scanD\next
 \expandafter\scanE\next{} % end
}

\def\scanB #1 {\noindent scanB:\par token \token \par\#1 #1 \par}

\def\scanC #1#2 {\noindent scanC:\par\#1 #1 \par\#2 #2 \par}

\def\scanD #1#2\relax{\noindent scanD:\par\#1 #1 \par\#2 #2 \par
 \edef\next{#2\relax}}

\def\scanE #1#2\relax{\noindent scanE:\par\#1 #1 \par\#2 \meaning #2 
 \ifx#2\relax \relax\par The end.\par\fi}

\begin{document}
\scanA{test}
\end{document}

---

Lessons Learned: (Okay, I should have known some of this already, but didn't.)

\documentclass{article}
\tracingmacros=1

\def\scanA#1{(#1)}% copies 1 token
\def\scanB#1 {(#1)}% copies 1 word

\newcommand{\scan}[1]{\scanC#1\END}% loop until \END
\def\scanC#1{\ifx#1\END\else(#1)\expandafter\scanC\fi}% expand \fi before \scanC

\newcommand{\scanwords}[1]{\let\between=\empty\scanD#1 \END}
\def\scanD#1 {\ifx#1\END\else\between
  \let\between=\wordfill% insert \wordfill between words
  \scanC#1\END% scan letters of word
  \expandafter\scanD\fi% expand \fi before \scanD
}
\def\END{almost anything}
\def\wordfill{( )}% \def once, \let repeatedly

\begin{document}
\Huge
\scanA two words

\scanB two words

\scan{t e s t}% ignores spaces

\scanwords{two words}
\end{document}

All of the answers are better than my first attempt, but I need to accept one to "close" the question. Since my goal was to LEARN how to scan better...

---

Here is another variant I've been playing with. It stores the text as a token list.

\documentclass{article}
\usepackage{lipsum}

\def\END{\END}

\newtoks\mytoks

\def\parse{\futurelet\next\special}% some tokens are ignored
\def\special{% \space and \bgroup
  \expandafter\ifx\space\next\relax \mytoks=\expandafter{\the\mytoks\space}\fi%
  \ifx\bgroup\next\relax \expandafter\copygroup\else
    \expandafter\normal\fi}
\def\copygroup#1{\mytoks=\expandafter{\the\mytoks{#1}}\parse}%
\def\normal#1{%
  \ifx\END#1\relax
    \the\mytoks% end of environment
  \else 
    \mytoks=\expandafter{\the\mytoks#1}%
  \expandafter\parse\fi}

\begin{document}
\parse
\begin{center}
enviroment test
\end{center}
\noindent\hbox{parse} test
\END
\end{document}

Answer 1

The accepted TeX solution here includes several problems. One of them is that whole argument is read at each step, no only one token. The second is that the recursive loop in the accepted code generates the nested \if...\fi construction which is very limited in TeX.

So, I show here the common scanner declared by TeX primitives without the problems described above. The scanning of the spaces is allowed, but braces are not allowed (for simplicity).

\def\scan#1{\scanA#1\end}
\def\scanA{\futurelet\next\scanB}
\def\scanB{\expandafter\ifx\space\next \expandafter\scanC \else \expandafter\scanE \fi}
\def\scanC{\afterassignment\scanD \let\next= }
\def\scanD{\scanE{ }}
\def\scanE#1{\ifx\end#1\else
   (#1)% <- The processing over one token is here
   \expandafter \scanA \fi
}

\scan{abcdef ghijkl mno}

\bye

Edit: If you leave the space behavior unchanged (i.e. they are ignored), then the code is much more simple:

\def\scan#1{\scanA#1\end}
\def\scanA#1{\ifx\end#1\else
   (#1)% <- The processing over one token is here
   \expandafter \scanA \fi
}

Answer 2

Scanning one token at a time requires at least distinguishing whether the scanned token is a space or a left brace. This is because you can't remove the scanned token with a one parameter macro in those cases.

First of all, let's see what \futurelet does; your \futurelet\token\scanB tells TeX to look at what token follows \scanB, without removing it, then making a \let\token=<scanned token> assignment and finally “seeing” \scanB, which should make decisions based on the value of \token.

For terminating the scanning, you have to place some special token at the end; this token is frequently a “quark”, say

\def\quark{\quark}

so \scanB can do \ifx\token\quark and, in this case, stop the recursion. Let's put into play what we have till now:

\makeatletter
\def\scan@quark{\scan@quark}% if we find it in bad places, we'll know!
\newcommand\scan[1]{\futurelet\@let@token\scan@aux@i#1\scan@quark}
\def\scan@aux@i{%
  \ifx\@let@token\scan@quark
    \expandafter\@gobbletwo
  \else
    \expandafter\@firstofone
  \fi
  {\scan@aux@ii}%
}

The macro \scan@aux@ii should now go on with other tests. I used \@gobbletwo in the “true” case so to gobble \scan@aux@ii and \scan@quark.

If instead you want just to split the input at a certain token, a better approach is using delimited arguments: you can find several examples on the site. With expl3 it's quite easy, because there are built in functions that do the job.

So, say you have an input such as \word{abc^def^ghi} that you want to print with alternating colors. Here's an implementation:

\documentclass{article}
\usepackage{xparse,xcolor}

\ExplSyntaxOn
\NewDocumentCommand{\word}{m}
 {
  \kormylo_word:n { #1 }
 }

\seq_new:N \l_kormylo_word_fragment_seq
\bool_new:N \l_kormylo_second_color_bool

\cs_new_protected:Npn \kormylo_word:n #1
 {
  \kormylo_change_color:
  \seq_set_split:Nnn \l_kormylo_word_fragment_seq { ^ } { #1 }
  \seq_use:Nn \l_kormylo_word_fragment_seq { \kormylo_change_color: }
 }

\cs_new_protected:Npn \kormylo_change_color:
 {
  \bool_if:NTF \l_kormylo_second_color_bool
   { \color{second} \bool_set_false:N \l_kormylo_second_color_bool }
   { \color{first} \bool_set_true:N \l_kormylo_second_color_bool }
 }
\ExplSyntaxOff

\colorlet{first}{black}
\colorlet{second}{red}

\begin{document}

\word{su^per^cal^i^frag^i^lis^tic^ex^pi^al^i^do^cious}

\end{document}

Note that you can use spaces around the separator token for better input, such spaces will be disregarded.

The macros could be extended to allow spaces in the input: just split at spaces and do a mapping.

\documentclass{article}
\usepackage{xparse,xcolor}

\ExplSyntaxOn
\NewDocumentCommand{\words}{m}
 {
  \kormylo_words:n { #1 }
 }

\seq_new:N \l_kormylo_word_seq
\seq_new:N \l_kormylo_word_fragment_seq
\bool_new:N \l_kormylo_second_color_bool

\cs_new_protected:Npn \kormylo_words:n #1
 {
  \seq_set_split:Nnn \l_kormylo_word_seq { ~ } { #1 }
  \seq_map_inline:Nn \l_kormylo_word_seq
   {
    \kormylo_word:n { ##1 }
    \c_space_tl
   }
 }

\cs_new_protected:Npn \kormylo_word:n #1
 {
  \kormylo_change_color:
  \seq_set_split:Nnn \l_kormylo_word_fragment_seq { ^ } { #1 }
  \seq_use:Nn \l_kormylo_word_fragment_seq { \kormylo_change_color: }
 }

\cs_new_protected:Npn \kormylo_change_color:
 {
  \bool_if:NTF \l_kormylo_second_color_bool
   { \color{second} \bool_set_false:N \l_kormylo_second_color_bool }
   { \color{first} \bool_set_true:N \l_kormylo_second_color_bool }
 }
\ExplSyntaxOff

\colorlet{first}{black}
\colorlet{second}{red}

\begin{document}

\words{su^per^cal^i^frag^i^lis^tic^ex^pi^al^i^do^cious syl^la^ble con^cate^na^tion}

\end{document}

Answer 3

For the first answers, I assume that you want to scan pure text, without any groups commands etc. and without any spaces.

This is the (mainly) TeX solution.

\documentclass{minimal}
\begin{document}
\def\END{}
\def\ENDEND{}
\newcommand*\scan[1]{\scani #1\END\ENDEND}
\def\scani#1#2\ENDEND{%
  \ifx\END#1%
  \else%
    (#1)%
    \scani#2\ENDEND%
  \fi
}
\scan{test}
\end{document}

And here is the LaTeX3 version.

\documentclass{minimal}
\usepackage{expl3}
\begin{document}
\ExplSyntaxOn
\newcommand*\scan[1]
  {
    \tl_map_inline:nn {#1} { (##1) }
  }
\ExplSyntaxOff
\scan{test}
\end{document}

Both will output

(t)(e)(s)(t)

Dealing with spaces, it somewhat tricky. I have a TeX solution here (from Usenet times), but do not understand it myself.

For LaTeX3, here are solutions that can cope with spaces: LaTeX3: tl_map with spaces

Or you use my version, which is

\documentclass{minimal}
\usepackage{expl3}
\begin{document}
\ExplSyntaxOn
\newcommand*\scan[1]
  {
    \__scanloop: #1 \q_recursion_stop
  }
\cs_new:Nn \__scanloop:
  {
    \peek_meaning_remove:NTF \q_recursion_stop
      {}
      {
    \peek_charcode_remove:NTF \c_space_token
      { 
        (~)
        \__scanloop:
      }
    % else
      {
        \__scanloop_aux:
      }}
  }
\cs_new:Npn \__scanloop_aux: #1
  {
    ( #1 )
    \__scanloop:
  }
\ExplSyntaxOff
\scan{test with spaces}
\end{document}

which will output

(t)(e)(s)(t)( )(w)(i)(t)(h)( )(s)(p)(a)(c)(e)(s)

Answer 4

This is not an answer to the original scanning question. It is an 'optimization' of egreg's solution for the coloring problem. It saves 17 lines of code by use of l3regex.

\documentclass{minimal}
\usepackage{xparse,l3regex,xcolor}

\ExplSyntaxOn
\NewDocumentCommand{\words}{m}
  {
    \tl_set:Nn \l_tmpa_tl {#1}
    \regex_replace_all:nnN { \cU\^ } { \c{kormylo_change_color:} } \l_tmpa_tl  % replace ^
    \regex_replace_all:nnN { \s } { \cS_ \c{kormylo_change_color:} } \l_tmpa_tl  % replace spaces
    \kormylo_change_color:
    \tl_use:N \l_tmpa_tl
  }
\bool_new:N \l_kormylo_second_color_bool
\cs_new_protected:Npn \kormylo_change_color:
  {
    \bool_if:NTF \l_kormylo_second_color_bool
      { \color{second} \bool_set_false:N \l_kormylo_second_color_bool }
      { \color{first} \bool_set_true:N \l_kormylo_second_color_bool }
  }
\ExplSyntaxOff

\colorlet{first}{black}
\colorlet{second}{red}

\begin{document}

\words{su^per^cal^i^frag^i^lis^tic^ex^pi^al^i^do^cious syl^la^ble con^cate^na^tion}

\end{document}

Answer 5

Right now, it can scan up to a paragraph, but with an added layer of macro, I can scan multiple paragraphs.

\documentclass{article}
\usepackage{xcolor}
\def\mycolor{1}
\newcommand\myparse[1]{%
  \myparsehelpA#1 \relax\relax}
\def\myparsehelpA#1 #2\relax{%
  \myparsehelpB#1\relax\relax%
  \ifx\relax#2\else\ \myparsehelpA#2\relax\fi
}
\def\myparsehelpB#1#2\relax{%
  \ifcat^#1%
    \if1\mycolor\def\mycolor{2}\color{red}%
    \else\def\mycolor{1}\color{black}%
    \fi%
  \else#1\fi%
  \ifx\relax#2\else
    \myparsehelpB#2\relax%
  \fi
}
\begin{document}
\myparse{Th^is^ is ^a te^st of the e^merge^ncy broad^cast sy^stem}
\end{document}

The \myparse macro can handle some macros in its argument, but only if they take no arguments themselves. For example, \myparse{Th^is^ is ^a te^st of the e^merge^ncy broad^cast \itshape sy^stem; \bfseries th^is is \mdseries on^ly a test.} yields