How do I implement a string buffer in TeX?

Question

I've recently learned how to create a macro to iterate over a token sequence using \afterassignment and \let in TeX.

Currently, I'm trying to apply this to a tokenization macro (where "tokenization" means splitting a character sequence around a delimiting character/ character-sequence); that is, an algorithm stringTokenize which does the following:

1. Iterate over character in the string
2. if character is not the delimiter (for example, ,): add character to buffer; if character is the delimiter, expand buffer and do something with it (that is, pass to a given macro) and clear buffer.

This is not as simple as I thought however, because apparently you can't expand a character which is \let to a command into another command (in order to create a "buffer" structure), for example,

\edef\buffer{\buffer\character}

will (after n iterations) expand to

{\character\character(...)character}

which is n times the character last \let to \character.

How do I do this?

---

Here's what I have so far:

\documentclass{minimal}

\makeatletter

\def\structures@stringIterate#1#2{%

    \newcount\@@index%

    \def\@@stop{§}
    \def\@@next{\afterassignment\@@each\let\@@current= }
    \def\@@each{%
      \ifx\@@current\@@stop%
         \let\@@next=\relax
      \else%
        \advance\@@index1\relax
         #2{\the\@@index}{\@@current}\let\@@next=\@@next
      \fi%
      \@@next % macro expansion must be delayed until the condition has been evaluated
    }

    \@@next#1\@@stop

}

\def\structures@stringTokenize#1#2#3{%

    \newcount\@@index
    \@@index 0\relax

    \def\@@buffer{}

    \def\@@test##1##2{%
        \ifx#2##2
            \advance\@@index 1\relax
            #3{\the\@@index}{\@@buffer}
            \def\@@buffer{} % reset the buffer
        \else
            \edef\@@buffer{\@@buffer##2} % gather character
        \fi
    }

    \structures@stringIterate{#1}{\@@test}

}

\makeatother

\begin{document}

\makeatletter

    \def\@@print#1#2{token (#1): ``#2''\par}
    \structures@stringTokenize{Humpty Dumpty sat on a wall, Humpty Dumpty had a great fall, All the King's horses and all the King's men, Couldn't put Humpty together again.}{,}{\@@print}

\makeatother

\end{document}

Answer 1

The usual approach to grabbing tokens with a delimiter in TeX would be something like

\catcode`\@=11 %

\def\grabargs#1#2#3{%
  \def\grabargs@aux##1#2{%
    \ifx\relax##1\relax%
    \else
      #3{##1}\par
      \expandafter\grabargs@aux
    \fi
  }%
  \grabargs@aux#1#2\relax#2
}
\def\print#1{`{\tt #1}'}
\catcode`\@=12 %

\grabargs
  {Humpty Dumpty sat on a wall, Humpty Dumpty had a great fall, All the King's horses and all the King's men, Couldn't put Humpty together again.}
  {,}
  {\print}

\bye

where depending how complex things need to be we can use a more fancy approach to stopping the loop.

What you can't do is use \let here (at least in any particularly easy way): this creates implicit tokens which as you've found can be expanded in, for example, \edef. What you might be looking to do is pick up space tokens: they are tricky to grab as a normal argument (#1), so you might use \futurelet in something like

\catcode`\@=11 %

\newcount\mycount
\def\grabargs#1#2#3{%
  \def\grabbed@tokens{}%
  \mycount\z@
  \def\grabargs@auxii{%
    \ifx\grab@token#2%
      \grabargs@auxiv
      \def\grabbed@tokens{}%
      \expandafter\grabargs@auxv
    \else
      \ifx\grab@token\@sptoken
        \expandafter\expandafter\expandafter\grabargs@space
      \else
        \ifx\grab@token\relax
        \else
          \expandafter\expandafter\expandafter\expandafter\expandafter
            \expandafter\expandafter\grabargs@auxiii
        \fi
      \fi
    \fi
  }%
  \def\grabargs@auxiv{%
    #3{\the\mycount}\grabbed@tokens
    \mycount\z@
  }%
  \grabargs@auxi#1#2\relax
}
\def\grabargs@auxi{%
  \futurelet\grab@token\grabargs@auxii
}
\def\grabargs@auxiii#1{%
  \edef\grabbed@tokens{%
    \grabbed@tokens
    #1%
  }%
  \advance\mycount\@ne
  \grabargs@auxi
}
\def\grabargs@auxv#1{\grabargs@auxi}
\expandafter\def\expandafter\grabargs@space\space{%
  \grabargs@auxiii\space
}
\def\:{\let\@sptoken= } \:  %
\def\print#1#2{Tokens: #1 `{\tt #2}'\par}
\catcode`\@=12 %

\grabargs
  {Humpty Dumpty sat on a wall, Humpty Dumpty had a great fall, All the King's horses and all the King's men, Couldn't put Humpty together again.}
  {,}
  {\print}

\bye

The idea here is that \futurelet is very similar to your \afterassignment\@@each\let\@@current= except it doesn't remove the token from the input stream. We grab a token using \futurelet then can examine it. If it's the marker, we can print the current grabbed tokens (and count), then loop. If it's the end marker, we stop, and if it's a 'normal' token we can grab the argument and store. If it's a space, we do a special version of the 'grab' macro which will handle a space. What I've not added in the above is code to deal with brace groups: that can be done but the approach depends on what you want to do about such things.

---

Note that in both of the above implementations we are storing tokens, which could mean issues with category code 6 tokens (usually #) and how you want to manage line breaks, etc. This can all be sorted out, but without a tighter spec I'm leaving it alone. Also, note that it's easy to get caught out by \futurelet in an \halign: see for example Where do I find \futurelet's nasty behaviour documented?.