Define an expandable function for comparing a token list to a string in LaTeX3

Question

In LaTeX3, I could define a function for comparing a token list to a string

\cs_generate_variant:Nn \tl_if_eq:nnTF { V }
\cs_new_nopar:Npn \module_compare:n #1
  {
    \tl_if_eq:VnTF \g_some_tl { #1 } { 1 } { 0 }
  }

But now I want to make this function exhaustively expandable. Are there any other similar functions other than \tl_if_eq:nnTF that I could use?

---

Edit: to give more context to my problem, I added the following unworking example. (Note that I need to use \ifnum for some reasons.)

\documentclass{article}
\usepackage{expl3}
\ExplSyntaxOn
\tl_set:Nn \g_some_tl {abc}
\cs_generate_variant:Nn \tl_if_eq:nnTF { V }
\cs_new_nopar:Npn \mycompare #1
  {
    \tl_if_eq:VnTF \g_some_tl { #1 } { 1 } { 0 }
  }
\ExplSyntaxOff
\begin{document}
\def\mytest#1{\ifnum \mycompare{#1} > 0 do some\else do other\fi}
\mytest{uvw}
\mytest{abc}
\end{document}

According to Joseph Wright's comments, replacing \tl_if_eq:VnTF with \str_if_eq:VnTF solves my problem.

Answer 1

Token list comparison cannot be expandable, because the only way TeX has for comparing them is to make them the replacement text of macros.

However, all the engines (except for Knuth TeX) implement an expandable “string comparison” that basically works with \detokenize (details are a bit more involved).

With \usepackage{pdftexcmds} you have available \pdf@strcmp (it is called \pdfstrcmp by pdftex, \strcmp by XeTeX and it's emulated with a Lua script in LuaTeX). The code

\pdf@strcmp{<string-A>}{<string-B>}

returns zero if the strings are equal (after expansion, here the details become involved), 1 or –1 otherwise.

This is also available with expl3 as \str_if_eq:nn(TF) (the parentheses mean that one or both the specifiers can be omitted) or \str_if_eq_x:nn(TF). The former doesn't perform expansion on its first two arguments, the latter does.

In your case you could do

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\DeclareExpandableDocumentCommand{\mytest}{m}
 {
  \zh_mytest:n { #1 }
 }

\cs_new:Npn \zh_mytest:n #1
 {
  \str_if_eq:VnTF \g_zh_fixed_tl { #1 } { 1 } { 0 }
 }
\cs_generate_variant:Nn \str_if_eq:nnTF { V }

\tl_gclear_new:N \g_zh_fixed_tl
\tl_gset:Nn \g_zh_fixed_tl { abc }
\ExplSyntaxOff

\begin{document}
\edef\test{\ifnum\mytest{x}>0 Equal\else Unequal\fi}

\texttt{\meaning\test}

\edef\test{\ifnum\mytest{abc}>0 Equal\else Unequal\fi}

\texttt{\meaning\test}
\end{document}

The usage of \edef is just to show that the macro is fully expandable. However, mixing in this way expl3 code and old style code is not really recommended.

I'd prefer something like

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\DeclareExpandableDocumentCommand{\mytestTF}{mmm}
 {
  \zh_mytest:nnn { #1 } { #2 } { #3 }
 }

\cs_new:Npn \zh_mytest:nnn #1 #2 #3
 {
  \str_if_eq:VnTF \g_zh_fixed_tl { #1 } { #2 } { #3 }
 }
\cs_generate_variant:Nn \str_if_eq:nnTF { V }

\tl_gclear_new:N \g_zh_fixed_tl
\tl_gset:Nn \g_zh_fixed_tl { abc }
\ExplSyntaxOff

\begin{document}
\edef\test{\mytestTF{x}{Equal}{Unequal}}

\texttt{\meaning\test}

\edef\test{\mytestTF{abc}{Equal}{Unequal}}

\texttt{\meaning\test}
\end{document}

Answer 2

You can use the etl package. It can expandably compare token lists. It has a few limits though:

• it ignores the character code of group begin and group end tokens (normally { and }, but one could also do \catcode`\(=1 \catcode`\)=2 and then use (something) and etl couldn't differentiate that from {something})

• it can't tell apart an active character which was let to the same character with another category, so after

  \lccode`\~=`\:
  \lowercase{\let~:}
  \catcode`\:=13
  

  And now the package couldn't detect whether : is the active token or one of category 12.

• spaces are normalised (but this happens at an early stage of TeX's parsing)

• as Ulrich Diez points out in the comments (and his answer) another issue is if the \escapechar is negative. To give a concise summary what is distinguished by etl: Tokens which have the same meaning and string representation are considered equal.

See the following:

\documentclass[]{article}
\usepackage{etl}
\ExplSyntaxOn
\tl_const:Nn \c_my_cmp_tl { some~ {complex}~ token~ \list }
\cs_generate_variant:Nn \etl_if_eq:nnTF { V }
\NewExpandableDocumentCommand \mycompare { m }
  {
    \etl_if_eq:VnTF \c_my_cmp_tl {#1} { 1 } { 0 }
  }
\ExplSyntaxOff
\begin{document}
% completely wrong
\mycompare{Something wrong}
% missing braces
\mycompare{some complex token list}
% missing braces
\mycompare{some complex token \list}
% last not a macro
\mycompare{some {complex} token list}
% last not a macro
\edef\tmp{\noexpand\mycompare{some {complex} token \string\list}}
\tmp
% this is the only correct one
\mycompare{some {complex} token \list}
% macro parameters are no problem
\mycompare{## macro#1 parameters}
% proving it's fully expandable
\edef\tmp{\mycompare{wrong}}\texttt{\meaning\tmp}
% proving it's fully expandable
\edef\tmp{\mycompare{some {complex} token \list}}\texttt{\meaning\tmp}
\end{document}

Answer 3

I think an expandable test entirely in TeX, not using Lua, that is reliable to 100% is not feasible—at least not feasible with unicode-TeX-engines like XeTeX or LuaTeX where 1114112 different character-codes are possible.

(I think handling explicit character-tokens of category 1/2 is feasible if you are willing to have a bunch of code just for doing some brace-hacking.)

The hard problems are:

• Finding an expandable way of, e.g., distinguishing an explicit non-active character token from its active pendant when the active-pendant is let equal to the non-active character token in question.
  E.g.,

  \begingroup\catcode`\Z=13\def\temp{\endgroup\letZ= }\temp Z
  

  How to expandably distinguish active-Z from letter-Z under these conditions without using macros that process delimited arguments?
  (In this scenario applying \string or \meaning either to active-Z or to letter-Z yields the same; comparing active-Z to letter-Z via \ifx/\if/\ifcat yields the true-branch. What else do we have for distinguishing tokens expandably?)
• Finding an expandable way of, e.g., distinguishing a one-letter-control-sequence whose name equals an explicit character token which it is let equal to while \escapechar is negative.
  E.g.,

  \let\!=!\escapechar=-1
  

  How to expandably distinguish \! from other-! under these conditions without using macros that process delimited arguments?
  (In this scenario applying \string or \meaning either to other-! or to \! yields the same; comparing other-! to \! via \ifx/\if/\ifcat yields the true-branch. What else do we have for distinguishing tokens expandably?)

I introduced the restriction of not using macros that process delimited arguments.

For each possible character-code you could write a macro for detecting by means of delimited arguments whether a token of that character-code is an active character of that character-code/is a one-letter-control-sequence whose name equals the character with the character-code in question/is something else.

But on a unicode-machine like xetex or luatex 1114112 character-codes from 0 to 1114111 are possible, thus 1114112 such macros would be needed. (Defining such macros "on the fly" is not an option as defining contradicts expandability.)

On a traditional 8-bit-engine where only 256 character-codes are possible, one could nowadays probably define 256 macros.

---

Outline of a mechanism for cranking out active-character and—in case of \escapechar being negative—one-letter-control-sequence via many macros that process delimited arguments—only applicable with 8-bit-engines if applicable at all:

Assuming that you wish to examine a macro-argument where you already know that it is

• either an explicit character token not of category 1 or 2
• or an active character token let equal to a pendant not of category 1 or 2
• or a single-letter-control-sequence let equal to a character-token not of category 1 or 2 whose character-code corresponds to the character forming the name of the single-letter-control-sequence while \esapechar is negative,

i.e., in any case something not of category 1/2 whose stringification yields a single character-token of category 10 or 12

, you can expandably crank out active characters and (in the edge case of \escapechar being negative) single-letter-control-sequences expandably by means of something like

\long\def\tokenfork#1<token1><token2>#2#3\SEP{#2}%
\long\def\forktoken#1{%
  \tokenfork
  #1<token2>{tokens in case #1 = <token1>}%
  <token1>#1{tokens in case #1 = <token2>}%
  <token1><token2>{<tokens in case #1 is s.th. else}%
  \SEP
}

where <token 1> is the active character and <token 2> is the single-letter-control-sequence.

Instead of \tokenfork and \forktoken you choose macro-names wherein the character occurs, e.g. \Afork and \forkA for cranking out active-A and \A via \csname fork\string#1\endcsname{#1} whereby #1 is something whose character-code is A and whose stringification yields the single letter A:

\long\def\firstofthree#1#2#3{#1}%
\long\def\secondofthree#1#2#3{#2}%
\long\def\thirdofthree#1#2#3{#3}%
\long\def\forkdefiner#1{\begingroup\lccode`\X=#1 \lccode`\~=#1 \lowercase{\let~\relax\definefork{X}{~}}}%
\long\def\definefork#1#2{%
  \expandafter\let\csname #1fork\endcsname\relax
  \expandafter\let\csname fork#1\endcsname\relax
  \expandafter\let\csname#1\endcsname\relax
  \expandafter\defineforkB\csname #1fork\expandafter\endcsname
                          \csname fork#1\expandafter\endcsname
                          \csname#1\endcsname{#2}%
}%
\long\def\defineforkB#1#2#3#4{%
  \long\gdef#1##1#3#4##2##3\SEP{##2}%
  \long\gdef#2##1{#1##1#4{\thirdofthree}#3##1{\secondofthree}#3#4{\firstofthree}\SEP}%
  \endgroup
}%
% In a loop define two macros forming forking mechanism for the character-codes 0..255
% that are possible in traditional 8-bit engines:
\newcount\scratchy
\scratchy=-1
\loop
\ifnum\scratchy<255 %
  \advance\scratchy by 1 %
  \forkdefiner{\scratchy}%
\repeat
% Now test with letter-A, one-letter-\A, active-A
\message{%
  ^^J%
  \csname fork\string A\endcsname{A}{neither active char nor one-letter-cs}%
                                    {active char}%
                                    {one-letter-cs}%
}%
{\escapechar=-1 \message{%
  ^^J%
  \csname fork\string\A\endcsname{\A}{neither active char nor one-letter-cs}%
                                     {active char}%
                                     {one-letter-cs}%
}}%
{\catcode`\A=13 \message{%
  ^^J%
  \csname fork\string A\endcsname{A}{neither active char nor one-letter-cs}%
                                    {active char}%
                                    {one-letter-cs}%
}}%
% Now test with other-!, one-letter-\!, active-!
\message{%
  ^^J%
  \csname fork\string!\endcsname{!}{neither active char nor one-letter-cs}%
                                   {active char}%
                                   {one-letter-cs}%
}%
{\escapechar=-1 \message{%
  ^^J%
  \csname fork\string\!\endcsname{\!}{neither active char nor one-letter-cs}%
                                     {active char}%
                                     {one-letter-cs}%
}}%
{\catcode`\!=13 \message{%
  ^^J%
  \csname fork\string!\endcsname{!}{neither active char nor one-letter-cs}%
                                   {active char}%
                                   {one-letter-cs}%
}}%
\bye

Output:

$ pdftex test.tex
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=pdftex)
 restricted \write18 enabled.
entering extended mode
(./test.tex 
neither active char nor one-letter-cs 
one-letter-cs 
active char
neither active char nor one-letter-cs 
one-letter-cs 
active char )
No pages of output.
Transcript written on test.log.

---

When examining tokens, the nameless control-sequence, producible via \csname\endcsname or via ending a line of .tex-input with an escape-char (backslash) while \endlinechar is negative, might also need special treatment via delimited arguments because applying \string to it yields <escapechar>csname<escapechar>endcsname. You get the same stringification for a control-sequence-token whose name is csname<escapechar>endcsname, producible via \csname csname\string\endcsname\endcsname. There is the very very edge case of these two different tokens having assigned the same meaning so that examining meaning and \string-representation alone is not sufficient for distinguishing them.
There are more weird things, e.g., frozen-\relax (which cannot be redefined) versus normal \relax, ...