In LaTeX, what is the most efficient way to search for an optional token in given input?

Question

My question is inspired by this comment by Joseph Wright.

Let's say we have input strings with varying character combinations, such as:

123ee4978124e340862e7352e442we34ts8034luio29752o93eweqwt27sfs59395239sfe384928%

ai*4et8!sur*ysnvo546wet*ywugh)oeigh*(wahfiwnegi45oehs*klgv'niwu75*3tsjkg;ninuw%

9.368e-03, 0.3846e-045, 238476.9734, 1,00,000.000e+0, -3.8746e+006, 23.0000874%

The enormous elephant was enranged by the sight of his savage captors carrying their ropes%
, axes and huge iron-barred cages.%

Let us also assume that our optional token is e. That is, we would like to 'look for' every instance of e in any/all input string/number elements, in order to perform some formatting/replacing operations. For instance, let's assume that we'd like to replace every instance of e with a blank space!

Given this problem, what would be the most efficient TeX/ LaTeX -based method to search for such an optional token from the given input?

Note: In this question, I am not including any further operations the user may or may not wish to perform once every instance of the optional token has been 'looked for'. However, if for demonstration purposes, some sort of post-search operation is necessary, then it would be a bonus!

Gentle suggestions:

When the experience with TeX/ LaTeX -based parsing is concerned, I am an 'entry-level TeX user', at best. Therefore, please consider the following:

• The approach should be sufficiently generic.
• Kindly provide TeX/ LaTeX -based solutions. LuaTeX/ XeTeX -based solutions are not preferable, as I'd rather not get confused with new syntax.
• Prospective solutions must consider the limited knowledge, understanding and abilities of an 'entry-level TeX/ LaTeX user' and therefore, in-line comments and brief explanations regarding the code behaviour would be most helpful!
• Updated: As far as possible, kindly provide solutions that do not use any additional packages dedicated to string-parsing (e.g. xstring). For reasons, see the update given below.

Many thanks in advance! :)

UPDATE: Perhaps I'm breaking some kind of rule here, by posting this Q-update after @HeikoOberdiek has posted an answer, nonetheless, here it is.

This update isn't intended to disregard @Heiko's answer in any way, because I understand the reasons behind his suggestions. However, his suggestion that beginners should just stick to the xstring package doesn't really work for me, since -

1. I'm already familiar with xstring & its \StrSubstitute macro; and
2. As a beginner, I am more keen on learning some basic TeX/LaTeX parsing techniques from more experienced users, so that I can improve my TeX abilities.

Therefore, I've added an extra suggestion to the above list via this update. I hope that's OK.

Answer 1

For a "entry-level TeX/LaTeX user", the most efficient way is, IMHO, to use a dedicated package such as xstring. This package works for both plain TeX and LaTeX. String replacements can be done with macro \StrSubstitute. The amount of work would be the reading of the documentation.

For the experienced user, the best method heavily depends on the circumstances:

• Which kind of tokens the string consists of? Spaces and groups can be tricky.
• Is an expandable implementation necessary or are assignments allowed? An expandable implementation has a broader usages, but at the cost of efficiency and complexity.
• How is the end of the string is specified?
• What is the search string? One token, several tokens?
• How should the result look like?
• ...

There are several methods, their usefulness depends on the answers, e.g.:

• \futurelet based parsing by inspecting the next token. This is not expandable, but efficient. Runtime complexity of O(n) can be possible.

• Parsing by macro parameter text. This allows expandable implementations. But usually not fast. Also groups with curly braces are causing much trouble.

• The TeX engine pdfTeX provides \pdfmatch, but this is not generic TeX. However, it is quite powerful, because regular expressions are supported. But the string is sanitized to characters with catcode 12 (or 10 for the space).

  Regular expressions are also supported by l3regex of the LaTeX3 project.

• Category code trickery. The replacement of e with a space is very easy:

    \catcode`\e=10 % category code of space
  

  But it should only be used inside a group. Otherwise it would be a little difficult to write \end{document}.

  Also active characters can be helpful.

• ...