Using string to process a macro name using listofitems seems to fail

Question

I am trying to process the name of a command using listofitems, but when I am using \string to convert that command it does not behave the way I think it would behave.

Here is a code to illustrate:

\documentclass[varwidth]{standalone}
\usepackage{xstring}
\usepackage{listofitems}
\makeatletter\newcommand{\currentname}{}
% Process ABC@DEF
\newcommand{\makelistA}[1]{
    \setsepchar{@}%
    \renewcommand{\currentname}{#1}%
    \readlist\currentlist{\currentname}%
    \currentname = [\currentlist[1],\currentlist[2]]\newline%
}
\newcommand{\runA}{\makelistA{ABC@DEF}}
% Process the command \ABC@DEF
\newcommand{\ABC@DEF}{}
\newcommand{\makelistB}[1]{
    \setsepchar{@}%
    \renewcommand{\currentname}{\StrGobbleLeft{\string#1}{1}}%
    \readlist\currentlist{\currentname}%
    \currentname = [\currentlist[1],\currentlist[2]]\newline%
}
\newcommand{\runB}{\makelistB{\ABC@DEF}}
\makeatother
\begin{document}
\runA % Writes ABC@DEF = [ABC,DEF] as expected
\runB % Writes ABC@DEF = [ABC@DEF,ABC@DEF] and generates errors
\end{document}

The version that process the command fails, and I cannot understand why.

Any explanation? And how to make it work?

\string is only used to write macros without expanding them, usually to the aux file. I believe what you want is \detokenize. — John Kormylo, Oct 28 '22 at 01:36

score 2 · Answer 1 · answered Oct 28 '22 at 09:06

There are two issues with the code. The first is that storing the output of xstring macros should not be done with \renewcommand. This stores the whole code of \StrGobbleLeft in the command, while you only want the result. With xstring this is done with an optional final argument:

\StrGobbleLeft{\string#1}{1}[\currentname]

The second issue is a catcode problem. You use \makeatletter, which sets the category code of @ to 11 (i.e., letters, see What are category codes?). However, \string (and \detokenize as well) set the category code of @ to 12 (i.e., other). Now listofitems cannot split the list because it is looking for a separator with catcode 11 while you provide a separator with catcode 12, and token comparison is catcode-aware.

The solution is easy: don't make at letter, but make it other.

\documentclass[varwidth]{standalone}
\usepackage{xstring}
\usepackage{listofitems}
\makeatletter
\newcommand{\currentname}{}
% Process ABC@DEF
\newcommand{\makelistA}[1]{
    \setsepchar{@}%
    \renewcommand{\currentname}{#1}%
    \readlist\currentlist{\currentname}%
    \currentname = [\currentlist[1],\currentlist[2]]\newline%
}
\newcommand{\runA}{\makelistA{ABC@DEF}}
% Process the command \ABC@DEF
\newcommand{\ABC@DEF}{}
\makeatother
\newcommand{\makelistB}[1]{
    \setsepchar{@}%
    \StrGobbleLeft{\string#1}{1}[\currentname]
    \readlist\currentlist{\currentname}%
    \currentname = [\currentlist[1],\currentlist[2]]\newline%
}
\makeatletter
\newcommand{\runB}{\makelistB{\ABC@DEF}}
\makeatother
\begin{document}
\runA % Writes ABC@DEF = [ABC,DEF] as expected
\runB % Same output as above
\end{document}

Result:

Note that the catcode change must be done outside of \newcommand{\makelistB} to make the change effective during the definition of the command instead of during execution, which is too late as the @ symbols are already tokenized at that point.

A possible improvement to listofitems would be do make the separator comparison catcode-independent, similar to xstring which offers starred variants like \IfStrEq* that first normalizes catcodes and then perform the comparison.

score 2 · Accepted Answer · answered Oct 28 '22 at 09:12

There are a few problems. First and foremost,

\renewcommand{\currentname}{\StrGobbleLeft{\string#1}{1}}

is never going to work, because \StrGobbleLeft is not expandable. You want

\StrGobbleLeft{\string#1}{1}[\currentname]

which would correctly store ABC@DEF in \currentname. However, listofitems won't find @, because you're defining the command in a context where @ has category code 11, but in the generated string it has category code 12.

You may want

\newcommand{\makelistB}[1]{%
    \expandafter\setsepchar\expandafter{\string @}%
    \StrGobbleLeft{\string#1}{1}[\currentname]%
    \readlist\currentlist{\currentname}%
    \currentname = [\currentlist[1],\currentlist[2]]\newline
}

so the separator @ certainly has category code 12.

Here's the expl3 code that's not confused by category codes.

\documentclass{article}
\ExplSyntaxOn
\NewDocumentCommand{\makelist}{m}
 {
  \tl_set:Nx \l_tmpa_tl { \cs_to_str:N #1 }
  \regex_split:nVN { @ } \l_tmpa_tl \l_tmpa_seq
  [\seq_use:Nn \l_tmpa_seq {,}]
 }
\cs_generate_variant:Nn \regex_split:nnN { nV }
\ExplSyntaxOff
\begin{document}
\makeatletter % pretend we're in a package file
\newcommand{\ABC@DEF}
\makelist{\ABC@DEF}
\makeatother
\end{document}

The output, as you can check, is

[ABC,DEF]

Using string to process a macro name using listofitems seems to fail

2 Answers2