Token counter - strange behaviour of math mode

Question

Description

I'm trying to define a macro, say \CountToken, that analyzes a selected text, counts the number of occurrences of a given token in this text and stores the number of occurrences in a counter. The token can also be in the form \<some_macro_name>, where \<some_macro_name> has the function of a separator. For my purposes, only such tokens are important that are not hidden inside a group.

The syntax assumed is

\CountToken[<selected_token>,\<counter_name>]
  <text>
\finish

The result should be as follows:

• create the counter \<counter_name>,
• count the number of occurrences of <selected_token> within <text> that is delimited by \finish,
• store the number of occurrences in \<counter_name>.

Example:

\CountToken[i,\cnti]
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur adipiscing elit.
\finish

Here, \the\cnti gives 6.

Own attempt (MWE)

Here is my minimal working example.

\documentclass{article}
\def\finish{}
\newcount\tmp
\def\CountToken[#1,#2]{%
  \def\TestedToken{#1}
  \tmp=0
  \newcount#2
  \let\TotalOccurrence#2
  \let\next\TestNext\next}
\long\def\TestNext#1{%
  \def\CurrentToken{#1}%
  \ifx#1\finish
    \def\next{\TotalOccurrence=\the\tmp\relax}
  \else
    \ifx\CurrentToken\TestedToken
      \advance\tmp by 1
    \fi
  \fi
  \next}
\begin{document}
%-----------------------
\CountToken[i,\cnti]
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur adipiscing elit.
\finish
%-----------------------
number of i's: \the\cnti
%-----------------------
\CountToken[f,\cntf]
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur adipiscing elit.
\finish
%-----------------------
number of f's: \the\cntf
%-----------------------
\CountToken[\something,\cntsomething]
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur\something adipiscing elit.\something
\finish
%-----------------------
number of \verb|\something|'s: \the\cntsomething
\end{document}

Everything seems to be running fine. However, changing the value of the fraction to 22/3 generates a compilation error. Experimentally, I found that this happens if the numerator of the fraction starts with two same digits. I know that, e.g., writing \frac{{22}}3 helps, but this is not a serious way for me.

Request

I would like to know several things.

1. How can I improve my definition of \CountToken such that everything is running fine? (Please, only LaTeX solutions.)
2. Could you describe the origin of the error described above?
3. Are there any further safe ways to count the selected token in a text snippet containing general mathematical stuff?

Answer 1

The problem is not math mode but any braced expression where the first two tokens are the same. Take your example of "{22}":

At some point, you looked at everything before {22}. Then the argument #1 of \TestNext becomes 22 and therefore \ifx#1\finish becomes \ifx22\finish.

Here \ifx 22 is true, so \finish is expanded, leading to the error.

You can fix this by comparing \CurrentToken instead of using #1 directly. You also need a helper macro storing your terminator:

\documentclass{article}

\def\finish{}
\def\finishmarker{\finish}
\newcount\tmp

\def\CountToken[#1,#2]{%
  \def\TestedToken{#1}%
  \tmp=0
  \newcount#2%
  \let\TotalOccurrence#2%
  \let\next\TestNext\next}

\long\def\TestNext#1{%
  \def\CurrentToken{#1}%
  \ifx\CurrentToken\finishmarker
    \def\next{\TotalOccurrence=\the\tmp\relax}%
  \else
    \ifx\CurrentToken\TestedToken
      \advance\tmp by 1
    \fi
  \fi
  \next}

\begin{document}

%-----------------------
\CountToken[i,\cnti]
  Lorem ipsum dolor $\frac{22}{3}$ sit amet,
  consectetur adipiscing elit.
\finish
%-----------------------
number of i's: \the\cnti



%-----------------------
\CountToken[f,\cntf]
  Lorem ipsum dolor $\frac{22}{3}$ sit amet,
  consectetur adipiscing elit.
\finish
%-----------------------
number of f's: \the\cntf


%-----------------------
\CountToken[\something,\cntsomething]
  Lorem ipsum dolor $\frac{22}{3}$ sit amet,
  consectetur\something adipiscing elit.\something
\finish
%-----------------------
number of \verb|\something|'s: \the\cntsomething


\end{document}

Another problem with the code is that it only counts "outer" tokens, meaning that tokens "hidden" inside of {} are ignored.

Answer 2

You do \def\CurrentToken{#1} but then do \ifx#1\finish instead of \ifx\CurrentToken\finish. Wait! This wouldn't yield true if we're at the end!

Yes, if you do \def\finish{\finish}. Of course you have to be careful that \finish doesn't end up in some place where it's expanded.

The \tmp counter is redundant: you have \TotalOccurrence available, use it.

\documentclass{article}

\def\finish{\finish}

\def\CountToken[#1,#2]{%
  \def\TestedToken{#1}%
  \ifdefined#2#2=0 \else\newcount#2\fi
  \let\TotalOccurrence#2%
  \let\next\TestNext\next
}

\long\def\TestNext#1{%
  \def\CurrentToken{#1}%
  \ifx\CurrentToken\finish
    \let\next\relax
  \else
    \ifx\CurrentToken\TestedToken
      \advance\TotalOccurrence by 1
    \fi
  \fi
  \next
}

\begin{document}

%-----------------------
\CountToken[i,\cnti]
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur adipiscing elit.
\finish
%-----------------------
number of i's: \the\cnti



%-----------------------
\CountToken[f,\cntf]
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur adipiscing elit.
\finish
%-----------------------
number of f's: \the\cntf


%-----------------------
\CountToken[\something,\cntsomething]
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur\something adipiscing elit.\something
\finish
%-----------------------
number of \verb|\something|'s: \the\cntsomething

\end{document}

Just for completeness, here's an expl3 version:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\NewDocumentCommand{\CountToken}{mm+m}
 {% #1 = token to check, #2 = counter, #3 = text to count in
  \int_zero_new:N #2
  \tl_map_inline:nn { #3 }
   {
    \tl_if_eq:nnT { #1 } { ##1 } { \int_incr:N #2 }
   }
 }
\ExplSyntaxOff

\begin{document}

%-----------------------
\CountToken{i}{\cnti}{
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur adipiscing elit.
}
%-----------------------
number of i's: \the\cnti



%-----------------------
\CountToken{f}{\cntf}{
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur adipiscing elit.
}
%-----------------------
number of f's: \the\cntf


%-----------------------
\CountToken{\something}{\cntsomething}{
  Lorem ipsum dolor $\frac{21}{3}$ sit amet,
  consectetur\something adipiscing elit.\something
}
%-----------------------
number of \verb|\something|'s: \the\cntsomething

\end{document}

A similar problem was solved in https://tex.stackexchange.com/a/525246/4427 (also counting tokens or sets of tokens inside braces).

Let's look at the code in more detail.

The macro \CountToken is defined to have three mandatory arguments, the last of which can contain also \par tokens (because of the + prefix).

The command first sets the counter to zero (argument #2) or allocates a new one if not yet existing (the effect is similar to \ifdefined#2#2=0 \else\newcount#2\fi).

Next we map the third argument: a token list can be seen as a list of items (spaces between items are ignored, though), where braced groups count as a single item. Each item is passed to the code specified as the second argument, in this case

\tl_if_eq:nnT { #1 } { ##1 } { \int_incr:N #2 }

Here ##1 stands for the current item in the mapping. The T stands for True: the third argument is only executed if the two token lists given as the first two arguments are equal.

It's essentially the same code as above, but the details of building the loop and checking equality are already provided by standard functions.