Methods for replacing character-tokens of catcode 12 in strings by other tokens?

Question

Assume you have a non delimited macro argument which forms a string/a sequence of arbitrary length of character tokens of category code 12.

And all instances of some characters in that string shall be replaced by other sequences of tokens.

E.g., all instances of A₁₂ shall be replaced by the control word token \foo, and all instances of B₁₂ shall be replaced by \bar\bar, and all instances of C₁₂ shall be replaced by \baz\baz\baz, and all instances of D₁₂ shall be replaced by the control word token \foobar, and all instances of E₁₂ shall be replaced by \foobaz\foobaz.

What could be an efficient way of implementing an (expandable) mechanism for performing such character-replacements?

What could be an efficient way of implementing an (expandable) mechanism for performing such character-replacements in case you wish both the characters and their replacements not to be hardcoded but to be proviedable via macro-arguments?

"Filtering" characters by delimited arguments in tail-recursive loop?
"Filtering" characters by \if-/\ifx-comparison-cascade in tail-recursive loop?
\uppercase/\lowercase?

Are all characters guaranteed to be catcode 12 or there might be spaces and/or braced tokens? — Phelype Oleinik, Jan 27 '20 at 02:37
It would be interesting to know how you get the string of category code 12 characters and a small use case. — egreg, Jan 27 '20 at 15:24

score 4 · Answer 1 · answered Jan 27 '20 at 03:09

Basically the same code I used in https://tex.stackexchange.com/a/521595/134574, but with the extra catcode-12 request and the input-to-output mapping given when you call the macro. Suppose the input token list is (expanded) \detokenize{AxBxC{CxBxA}}{AxBxC} (that is, the first two chunks are catcode 12 and the last one is catcode 11), the replacement is {A}{\foo} {B}{\bar} {C}{\baz}, then the output will look like:

This method will preserve all input tokens (spaces, grouped tokens, etc.) and will just replace the requested tokens (with the exception that pairs of catcode-1 and catcode-2 tokens are normalised to { and }). If you can guarantee anything about the input token list (for example, that it doesn't have spaces and grouped tokens) then the code becomes a lot simpler.

You use the macro like this:

\parse{AxBxC{CxBxA}{AxBxC}}{%
  {A}{\foo}
  {B}{\bar}
  {C}{\baz}
}

Here's the code:

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\NewExpandableDocumentCommand \parse {+m+m}
  { \fan_parse:nn {#1} {#2} }
\cs_new:Npn \fan_parse:nn #1 #2
  { \exp_args:No \exp_not:o { \__fan_parse:nn {#1} {#2} } }
\cs_new:Npn \__fan_parse:nn #1 #2
  {
    \exp:w
      \group_align_safe_begin:
        \__fan_parse_loop:w #1
          \q_recursion_tail \q_recursion_stop
        \__fan_replacement:n {#2}
        \__fan_result:n { }
  }
\cs_new:Npn \__fan_end:w
    \__fan_replacement:n #1
    \__fan_result:n #2
  {
      \group_align_safe_end:
    \exp_end:
    #2
  }
\cs_new:Npn \__fan_parse_loop:w #1 \q_recursion_stop
  {
    \tl_if_head_is_N_type:nTF {#1}
      { \__fan_N_type:N }
      {
        \tl_if_head_is_group:nTF {#1}
          { \__fan_group:nw }
          { \__fan_space:w }
      }
    #1 \q_recursion_stop
  }
\cs_new:Npn \__fan_N_type:N #1
  {
    \quark_if_recursion_tail_stop_do:Nn #1
      { \__fan_end:w }
    \__fan_parse_specials:Nw #1
  }
\cs_new:Npn \__fan_group:nw #1 #2 \__fan_replacement:n #3
  {
    \exp_args:NNo \exp_args:No \__fan_group:n
      { \__fan_parse:nn {#1} {#3} }
    #2 \__fan_replacement:n {#3}
  }
\cs_new:Npn \__fan_group:n #1 { \__fan_add_result:nw { {#1} } }
\exp_last_unbraced:NNo
\cs_new:Npn \__fan_space:w \c_space_tl { \__fan_add_result:nw { ~ } }
\cs_new:Npn \__fan_add_result:nw #1 #2 \q_recursion_stop
    \__fan_replacement:n #3
    \__fan_result:n #4
  {
    \__fan_parse_loop:w #2 \q_recursion_stop
    \__fan_replacement:n {#3}
    \__fan_result:n {#4 #1}
  }
% The macro that does the replacement
\cs_new:Npn \__fan_parse_specials:Nw #1 #2 \__fan_replacement:n #3
  {
    \exp_args:Nf \__fan_add_result:nw
      {
        \token_if_eq_catcode:NNTF \c_catcode_other_token #1
          {
            \exp_after:wN \exp_after:wN \exp_after:wN \exp_stop_f:
            \str_case:nnF {#1} {#3}
              {#1}
          }
          {#1}
      }
    #2 \__fan_replacement:n {#3}
  }
\ExplSyntaxOff
\begin{document}
\edef\tmp{{\detokenize{AxBxC{CxBxA}}{AxBxC}}}
\ttfamily\expandafter\parse\tmp{%
  {A}{\foo}
  {B}{\bar}
  {C}{\baz}
}
\end{document}

If you want to replace everything, not just catcode-12 tokens, then you need to remove the \token_if_eq_catcode:NNTF from \__fan_parse_specials:Nw:

\cs_new:Npn \__fan_parse_specials:Nw #1 #2 \__fan_replacement:n #3
  {
    \exp_args:Nf \__fan_add_result:nw
      {
        \exp_after:wN \exp_after:wN \exp_after:wN \exp_stop_f:
        \str_case:nnF {#1} {#3}
          {#1}
      }
    #2 \__fan_replacement:n {#3}
  }

Steven B. Segletes · Answer 2 · 2020-01-27T12:29:18.333

No problem for tokcycle.

\documentclass{article}
\usepackage{tokcycle}
\usepackage[T1]{fontenc}
\def\foo{Fxyz}
\def\bar{Bpdq}
\def\baz{Babc}
\def\foobar{FB123}
\def\foobaz{FB789}
\edef\theinput{\detokenize{XYZABCDEFG}}

\tokcycleenvironment\stringsub
{\expandafter\ifx\detokenize{A}##1\addcytoks{\foo}\else
 \expandafter\ifx\detokenize{B}##1\addcytoks{\bar}\else
 \expandafter\ifx\detokenize{C}##1\addcytoks{\baz}\else
 \expandafter\ifx\detokenize{D}##1\addcytoks{\foobar}\else
 \expandafter\ifx\detokenize{E}##1\addcytoks{\foobaz}\else
 \addcytoks{##1}\fi\fi\fi\fi\fi}
{\processtoks{##1}}
{\addcytoks{##1}}
{\addcytoks{##1}}
\begin{document}
\theinput

\expandafter\stringsub\theinput\endstringsub

\detokenize\expandafter{\the\cytoks}
\end{document}

While the OP did not ask for the case where the input stream contains spaces, groups, and/or macros, this also poses no problem. Example:

\edef\theinput{\detokenize{XY} \noexpand\textit{\detokenize{ZABC}}\detokenize{DE}
  \detokenize{FG}}

Result:

Ulrich Diez · Answer 3 · 2020-01-28T13:25:29.070

To me the question seems

to be an academic one.
interesting.

Be aware that applying \string/\meaning/\detokenize (from the ε-TeX extensions) and the like does not necessarily yield explicit category-code-12(other)-character-tokens only: Spaces (character code 32 in the TeX engine's internal character encoding) delivered by these control sequences will always be of category code 10(space).

I think the issue can be divided into the following topics:

Iterating on lists of given tokens and hereby collecting the tokens that form the result by means of methods only that are suitable for pure-expansion-contexts.

In your scenario, which is about sequences of catcode-12-character-tokens only, curly braces (explicit character tokens of category code 1 or 2) and spaces (explicit character tokens of character code 32 and category code 10) and the like don't need to be taken into consideration.
Methods suitable for pure-expansion-contexts for finding out whether a token is an element of a set of tokens, and if so, which one it is. In your scenario the set of tokens would be A₁₂, B₁₂, C₁₂, D₁₂, E₁₂, and you need to find out if an element of the string is also an element of that set, and if so, which one.

In case explicit catcode-12-character-tokens can be taken for granted, "filtering" can be done by means of \if-comparison for checking whether character-codes are the same.

In case explicit catcode-12-character-tokens cannot be taken for granted while "expandibility" is desired, "filtering" gets more complicated because combining all kinds of \if..-comparison alone might not be sufficient for distinguishing explicit character tokens from implicit character tokens:

E.g., in traditional TeX it is not possible to distinguish by such methods only that both are suitable for pure-expansion-contexts and are not based on delimited arguments between a catcode-13-character-token which via \let is made equal to its catcode-12-pendant and that catcode-12-pendant.

E.g., with
```
\catcode`\:=13
\expandafter\let\expandafter:\expandafter=\string:
```
it might be difficult to distinguish :₁₃ from :₁₂ by methods only that both are suitable for pure-expansion-contexts and are not based on delimited arguments.

In case \escapechar is not positive, you might face some more problems:

E.g., with
```
\escapechar=-1
\let\:=:
```
it might be difficult in traditional TeX to distinguish between : and \: only by "expandible methods" that are not based on delimited arguments.

Be that as it may. Let's stick to the preconditions provided by you and assume that the string in question (probably due to some sophisticated pre-processing) contains only explicit character tokens of category code 12.

In case the set of catcode-12-character-tokens whose instances are to be cranked out of the string is always the same, you can instead of having TeX apply whatsoever \if/\ifx-comparison have TeX do the "filtering" by means of delimited arguments.

As long as, e.g., it is ensured that the control symbol token \! is not provided as a component of the argument (as is the case with a set of catcode-12-character-tokens), you can, e.g., do the filtering with something like this:

\begingroup
\catcode`\A=12 %
\catcode`\B=12 %
\catcode`\C=12 %
\catcode`\D=12 %
\catcode`\E=12 %
\def\firstofone#1{#1}%
\firstofone{%
  \endgroup
  \long\def\abcdeFork#1{%
     \abcdeSelect
     \!#1\!A\!B\!C\!D\!E\!{}% Case: #1 is empty
     \!\!#1\!B\!C\!D\!E\!{\foo}% Case: #1 = A of catcode 12
     \!\!A\!#1\!C\!D\!E\!{\bar\bar}% Case: #1 = B of catcode 12
     \!\!A\!B\!#1\!D\!E\!{\baz\baz\baz}% Case: #1 = C of catcode 12
     \!\!A\!B\!C\!#1\!E\!{\foobar}% Case: #1 = D of catcode 12
     \!\!A\!B\!C\!D\!#1\!{\foobaz\foobaz}% Case: #1 = E of catcode 12
     \!\!A\!B\!C\!D\!E\!{#1}% Case: #1 is something else.
     \!\!\!\!%
  }%
  \long\def\abcdeSelect#1\!\!A\!B\!C\!D\!E\!#2#3\!\!\!\!{#2}%
}%

% Some dummy-definitions - be aware that \bar usually is already defined in TeX/LaTeX
% and the already existing definition gets overridden here:
\def\space{ }%
\def\foo{\string\foo}
\def\bar{\string\bar}
\def\baz{\string\baz}
\def\foobar{\string\foobar}
\def\foobaz{\string\foobaz}

\begingroup
\tt\frenchspacing

\string\abcdeFork\string{\string} yields: \abcdeFork{}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space A\string} yields:
\expandafter\abcdeFork\expandafter{\string A}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space B\string} yields:
\expandafter\abcdeFork\expandafter{\string B}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space C\string} yields:
\expandafter\abcdeFork\expandafter{\string C}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space D\string} yields:
\expandafter\abcdeFork\expandafter{\string D}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space E\string} yields:
\expandafter\abcdeFork\expandafter{\string E}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space X\string} yields:
\expandafter\abcdeFork\expandafter{\string X}

\endgroup

\bye

In situations where it is not ensured that \! is not provided as a component of the argument, you can combine this with a check for \!.

The gist of the check is:

Append \! to the argument. Then have TeX gobble everything to the first instance of \!. If the result is an empty argument, then no \! was present, otherwise at least one \! was present which indicates that the argument neither is just A₁₂ nor is just B₁₂ nor is just C₁₂ nor is just D₁₂ nor is just E₁₂.

You need sub-routines for checking whether an argument is empty and for gobbling everything till \!.

In the example below checking for the emptiness of an argument is done by the macro \CheckWhetherNull.

\CheckWhetherNull is explained in detail in my answer to the question "Expandable test for an empty token list—methods, performance, and robustness". In that answer it is not named \CheckWhetherNull but it is named \CheckWhetherEmpty.

\long\def\firstoftwo#1#2{#1}
\long\def\secondoftwo#1#2{#2}
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \CheckWhetherNull{<Argument which is to be checked>}%
%%                    {<Tokens to be delivered in case that argument
%%                      which is to be checked is empty>}%
%%                    {<Tokens to be delivered in case that argument
%%                      which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\long\def\CheckWhetherNull#1{%
  \romannumeral0\expandafter\secondoftwo\string{\expandafter
  \secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \secondoftwo\string}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\firstoftwo\expandafter{} \secondoftwo}%
  {\firstoftwo\expandafter{} \firstoftwo}%
}%
%%-----------------------------------------------------------------------------
\long\def\GobbleToExclam#1\!{}%
%%-----------------------------------------------------------------------------
%% Check whether argument does not contain \! (unless nested in braces and
%% thus not disturbing)
%%.............................................................................
%% \checkwhethernoexclam{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked does not contain \!>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked does contain \!>}%
%%
\long\def\checkwhethernoexclam#1{%
  \expandafter\CheckWhetherNull\expandafter{\GobbleToExclam#1\!}%
}%
%
\begingroup
\catcode`\A=12 %
\catcode`\B=12 %
\catcode`\C=12 %
\catcode`\D=12 %
\catcode`\E=12 %
\def\firstofone#1{#1}%
\firstofone{%
  \endgroup
  \long\def\abcdeFork#1{%
     \checkwhethernoexclam{#1}{%
       \abcdeSelect
       \!#1\!A\!B\!C\!D\!E\!{}% Case: #1 is empty
       \!\!#1\!B\!C\!D\!E\!{\foo}% Case: #1 = A of catcode 12
       \!\!A\!#1\!C\!D\!E\!{\bar\bar}% Case: #1 = B of catcode 12
       \!\!A\!B\!#1\!D\!E\!{\baz\baz\baz}% Case: #1 = C of catcode 12
       \!\!A\!B\!C\!#1\!E\!{\foobar}% Case: #1 = D of catcode 12
       \!\!A\!B\!C\!D\!#1\!{\foobaz\foobaz}% Case: #1 = E of catcode 12
       \!\!A\!B\!C\!D\!E\!{#1}% Case: #1 is something else without \! 
       \!\!\!\!%
    }{#1}% Case: #1 is something else with \! 
  }%
  \long\def\abcdeSelect#1\!\!A\!B\!C\!D\!E\!#2#3\!\!\!\!{#2}%
}%

% Some dummy-definitions - be aware that \bar usually is already defined in TeX/LaTeX
% and the already existing definition gets overridden here:
\def\space{ }%
\def\foo{\string\foo}
\def\bar{\string\bar}
\def\baz{\string\baz}
\def\foobar{\string\foobar}
\def\foobaz{\string\foobaz}
\def\!{\string\!}

\begingroup
\tt\frenchspacing

\string\abcdeFork\string{\string} yields: \abcdeFork{}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space A\string} yields:
\expandafter\abcdeFork\expandafter{\string A}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space B\string} yields:
\expandafter\abcdeFork\expandafter{\string B}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space C\string} yields:
\expandafter\abcdeFork\expandafter{\string C}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space D\string} yields:
\expandafter\abcdeFork\expandafter{\string D}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space E\string} yields:
\expandafter\abcdeFork\expandafter{\string E}

\string\expandafter\string\abcdeFork\string\expandafter\string{\string\string\space X\string} yields:
\expandafter\abcdeFork\expandafter{\string X}

\string\abcdeFork\string{\string\!\string} yields: \abcdeFork{\!}

\endgroup

\bye

Using this forking-/filtering-technique you can create a tail-recursive loop for iterating on your list of catcode-12-character-tokens and passing each character to \abcdeFork.

As you gave the precondition that there can only occur (explicit?) catcode-12-character-tokens in the string, the string can be taken for a list of explicit (catcode-12-)character-tokens, and the loop does not need to take space-tokens and curly braces/explicit character-tokens of category code 1 or 2 into consideration, and can be structured as follows:

Have a tail-recursive macro which processes two arguments:

The list of remaining catcode-12-character-tokens.
The tokens forming the result collected so far.

In case the list of remaining catcode-12-character-tokens is empty, terminate the loop by delivering the tokens forming the result collected so far.

In case the list of remaining catcode-12-character-tokens is not empty, call the tail-recursive macro again, with arguments modified as follows:

The first element is removed from the list of remaining catcode-12-character-tokens.
The result of extracting the first element from the list of remaining catcode-12-character-tokens and processing it by \abcdeFork is attached to the tokens forming the result collected so far.

This could look like this:

\long\def\firstoftwo#1#2{#1}
\long\def\secondoftwo#1#2{#2}
\long\def\PassFirstToSecond#1#2{#2{#1}}
\long\def\exchange#1#2{#2#1}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \CheckWhetherNull{<Argument which is to be checked>}%
%%                    {<Tokens to be delivered in case that argument
%%                      which is to be checked is empty>}%
%%                    {<Tokens to be delivered in case that argument
%%                      which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\long\def\CheckWhetherNull#1{%
  \romannumeral0\expandafter\secondoftwo\string{\expandafter
  \secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \secondoftwo\string}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\firstoftwo\expandafter{} \secondoftwo}%
  {\firstoftwo\expandafter{} \firstoftwo}%
}%
%%-----------------------------------------------------------------------------
\long\def\GobbleToExclam#1\!{}%
%%-----------------------------------------------------------------------------
%% Check whether argument does not contain \! (unless nested in braces and
%% thus not disturbing)
%%.............................................................................
%% \checkwhethernoexclam{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked does not contain \!>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked does contain \!>}%
%%
\long\def\checkwhethernoexclam#1{%
  \expandafter\CheckWhetherNull\expandafter{\GobbleToExclam#1\!}%
}%
%
\begingroup
\catcode`\A=12 %
\catcode`\B=12 %
\catcode`\C=12 %
\catcode`\D=12 %
\catcode`\E=12 %
\def\firstofone#1{#1}%
\firstofone{%
  \endgroup
  \long\def\abcdeFork#1{%
     \romannumeral0\checkwhethernoexclam{ #1}{%
       \abcdeSelect
       \!#1\!A\!B\!C\!D\!E\!{ }% Case: #1 is empty
       \!\!#1\!B\!C\!D\!E\!{ \foo}% Case: #1 = A of catcode 12
       \!\!A\!#1\!C\!D\!E\!{ \bar\bar}% Case: #1 = B of catcode 12
       \!\!A\!B\!#1\!D\!E\!{ \baz\baz\baz}% Case: #1 = C of catcode 12
       \!\!A\!B\!C\!#1\!E\!{ \foobar}% Case: #1 = D of catcode 12
       \!\!A\!B\!C\!D\!#1\!{ \foobaz\foobaz}% Case: #1 = E of catcode 12
       \!\!A\!B\!C\!D\!E\!{ #1}% Case: #1 is something else without \! 
       \!\!\!\!%
    }{ #1}% Case: #1 is something else with \! 
  }%
  \long\def\abcdeSelect#1\!\!A\!B\!C\!D\!E\!#2#3\!\!\!\!{#2}%
}%
%%-----------------------------------------------------------------------------
%% Extract first non-delimited argument from list:
%%
%%   \romannumeral\ExtractFirstArgLoop{<List of non-delimited arguments>\SelDOm} 
%%   yields first element of <List of non-delimited arguments>.
%%
%%   <List of non-delimited arguments> must not be empty.
%%   <List of non-delimited arguments> may contain the token \SelDOm.
%%
%%   \romannumeral\ExtractFirstArgLoop{uvwxy\SelDOm} yields  {u}
%%   \romannumeral\ExtractFirstArgLoop{{uv}wxy\SelDOm} yields  {uv}
%%-----------------------------------------------------------------------------
\long\def\RemoveTillSelDOm#1#2\SelDOm{{#1}}%
\long\def\ExtractFirstArgLoop#1{%
  \expandafter\CheckWhetherNull\expandafter{\firstoftwo{}#1}%
  {0 #1}%
  {\expandafter\ExtractFirstArgLoop\expandafter{\RemoveTillSelDOm#1}}%
}%
%%-----------------------------------------------------------------------------
%% The replacement-routine:
%%-----------------------------------------------------------------------------
\long\def\abcdeReplace#1{\romannumeral0\abcdeReplaceloop{}{#1}}%
\long\def\abcdeReplaceloop#1#2{%
  % #1 = tokens forming the result collected so far
  % #2 = list of remaining catcode-12-character-tokens
  \CheckWhetherNull{#2}{ #1}{%
    \expandafter\PassFirstToSecond\expandafter{\firstoftwo{}#2}{%
      \expandafter\abcdeReplaceloop\expandafter{%
        \romannumeral0%
        \expandafter\exchange\expandafter{%
          \romannumeral0\exchange{ }{%
            \expandafter\expandafter\expandafter\expandafter
            \expandafter\expandafter\expandafter
          }%
          \expandafter\abcdeFork\romannumeral\ExtractFirstArgLoop{#2\SelDOm}%
        }{ #1}%
      }%
    }%
  }%
}%

% Some dummy-definitions - be aware that \bar usually is already defined in TeX/LaTeX
% and the already existing definition gets overridden here:
\def\space{ }%
\def\foo{\string\foo}
\def\bar{\string\bar}
\def\baz{\string\baz}
\def\foobar{\string\foobar}
\def\foobaz{\string\foobaz}
%
% Now let's define a macro holding a test-string:  
\edef\teststring{%
   \string s\string o\string m\string e\string t\string h%
   \string i\string n\string g%
   \string 1\string A\string 2\string E\string 3\string E%
   \string 4\string B\string 5\string D\string 6\string D%
   \string 7\string C\string 8\string C\string 9\string A%
   \string s\string o\string m\string e\string t\string h%
   \string i\string n\string g%
}

\begingroup
\font\myfont=cmtt8 at 7pt
\myfont
\frenchspacing

\string\teststring\space yields:

\teststring

\string\expandafter\string\abcdeReplace\string\expandafter\string{\string\teststring\string}
yields:

\expandafter\abcdeReplace\expandafter{\teststring}

\endgroup

\bye

In case of

having the replacement-mechanism expandable and
the set of catcode-12-character-tokens whose instances are to be cranked out of the string not always being the same but being "dynamically specifiable" via macro-arguments which denote how characters shall get replaced by token-sequences,

things might end up with nesting tail recursive loops as iteration is needed both on the list of catcode-12-character-tokens that forms the string and on the list of replacement-directives.

E.g., a nested tail-recursive loop where explicit category-code-12-character-tokens get "filtered via \if-comparison" is feasible as in this case \if-comparison for checking whether character codes are the same seems sufficient.

Syntax of the routine \Replace in the example below is:

\Replace{%
  &langle;Catcode-12-Character-token-string where characters are to be replaced&rangle;
}%%
  {{&langle;Character 1&rangle;}{&langle;Replacement of Character 1&rangle;}}%
  {{&langle;Character 2&rangle;}{&langle;Replacement of Character 2&rangle;}}%
  ...
  {{&langle;Character K&rangle;}{&langle;Replacement of Character K&rangle;}}%
}%

\long\def\firstoftwo#1#2{#1}
\long\def\secondoftwo#1#2{#2}
\long\def\PassFirstToSecond#1#2{#2{#1}}
\long\def\exchange#1#2{#2#1}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \CheckWhetherNull{<Argument which is to be checked>}%
%%                    {<Tokens to be delivered in case that argument
%%                      which is to be checked is empty>}%
%%                    {<Tokens to be delivered in case that argument
%%                      which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\long\def\CheckWhetherNull#1{%
  \romannumeral0\expandafter\secondoftwo\string{\expandafter
  \secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \secondoftwo\string}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\firstoftwo\expandafter{} \secondoftwo}%
  {\firstoftwo\expandafter{} \firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Extract first non-delimited argument from list:
%%
%%   \romannumeral\ExtractFirstArgLoop{<List of non-delimited arguments>\SelDOm} 
%%   yields first element of <List of non-delimited arguments>.
%%
%%   <List of non-delimited arguments> must not be empty.
%%   <List of non-delimited arguments> may contain the token \SelDOm.
%%
%%   \romannumeral\ExtractFirstArgLoop{uvwxy\SelDOm} yields  {u}
%%   \romannumeral\ExtractFirstArgLoop{{uv}wxy\SelDOm} yields  {uv}
%%-----------------------------------------------------------------------------
\long\def\RemoveTillSelDOm#1#2\SelDOm{{#1}}%
\long\def\ExtractFirstArgLoop#1{%
  \expandafter\CheckWhetherNull\expandafter{\firstoftwo{}#1}%
  {0 #1}%
  {\expandafter\ExtractFirstArgLoop\expandafter{\RemoveTillSelDOm#1}}%
}%
%%-----------------------------------------------------------------------------
%% The replacement-routine:
%%-----------------------------------------------------------------------------
\long\def\Replace{\romannumeral0\Replaceloop{}}%
\long\def\Replaceloop#1#2#3{%
  % #1 = tokens forming the result collected so far
  % #2 = list of remaining catcode-12-character-tokens
  % #3 = list of replacements
  \CheckWhetherNull{#2}{ #1}{%
    \PassFirstToSecond{#3}{%
      \expandafter\PassFirstToSecond\expandafter{\firstoftwo{}#2}{%
        \expandafter\Replaceloop\expandafter{%
          \romannumeral0%
          \expandafter\exchange\expandafter{%
            \romannumeral0%
            \expandafter\InnerReplaceloop\romannumeral\ExtractFirstArgLoop{#2\SelDOm}{#3}%
          }{ #1}%
        }%
      }%
    }%
  }%
}%
\long\def\InnerReplaceloop#1#2{%
  % #1 = catcode-12-token
  % #2 = remaining list of replacements
  \expandafter\CheckWhetherNull\expandafter{\firstoftwo#2{}.}{ #1}{%
    \if
    \expandafter\expandafter
    \expandafter            \firstoftwo
    \expandafter\firstoftwo
    \romannumeral\ExtractFirstArgLoop{#2\SelDOm}{}#1%
      \expandafter\firstoftwo
    \else
      \expandafter\secondoftwo
    \fi
    {%
      \exchange{ }{%
        \expandafter\expandafter
        \expandafter            \expandafter
        \expandafter\expandafter
        \expandafter
      }%
      \expandafter\expandafter
      \expandafter            \secondoftwo
      \expandafter\firstoftwo
      \romannumeral\ExtractFirstArgLoop{#2\SelDOm}{}%
    }{%
      \expandafter\PassFirstToSecond\expandafter{%
        \firstoftwo{}#2%
      }{\InnerReplaceloop{#1}}%
    }%
  }%
}%
%-------------------------------------------------------------------------------------
% Some dummy-definitions - be aware that \bar usually is already defined in TeX/LaTeX
% and the already existing definition gets overridden here:
\def\space{ }%
\def\foo{\string\foo}
\def\bar{\string\bar}
\def\baz{\string\baz}
\def\foobar{\string\foobar}
\def\foobaz{\string\foobaz}
%
% Now let's define a macro holding a test-string:  
\edef\teststring{%
   \string s\string o\string m\string e\string t\string h%
   \string i\string n\string g%
   \string 1\string A\string 2\string E\string 3\string E%
   \string 4\string B\string 5\string D\string 6\string D%
   \string 7\string C\string 8\string C\string 9\string A%
   \string s\string o\string m\string e\string t\string h%
   \string i\string n\string g%
}%

\begingroup
\font\myfont=cmtt8 at 7pt
\myfont
\baselineskip=8.2pt
\parindent=0pt
\frenchspacing

\string\teststring\space yields:
\medskip

\teststring
\medskip

\string\expandafter\string\Replace\string\expandafter\string{\string\teststring\string}\string{\hfil\break
\null\space\space\string{\string{A\string}\string{\string\foo\string}\string}\hfil\break
\null\space\space\string{\string{B\string}\string{\string\bar\string\bar\string}\string}\hfil\break
\null\space\space\string{\string{C\string}\string{\string\baz\string\baz\string\baz\string}\string}\hfil\break
\null\space\space\string{\string{D\string}\string{\string\foobar\string}\string}\hfil\break
\null\space\space\string{\string{E\string}\string{\string\foobaz\string\foobaz\string}\string}\hfil\break
\string}\hfil\break
yields:
\medskip

\expandafter\Replace\expandafter{\teststring}{%
  {{A}{\foo}}%
  {{B}{\bar\bar}}%
  {{C}{\baz\baz\baz}}%
  {{D}{\foobar}}%
  {{E}{\foobaz\foobaz}}%
}

\endgroup

\bye

Methods for replacing character-tokens of catcode 12 in strings by other tokens?

3 Answers3