Why does this regexpatch command only work once, not twice?

Question

Based on an early version of How can I add a single curly brace to a macro?, I have tried the following to add, one by one, inner commands to some outer command. (The reason I am doing this is that I do not know when the last element has been reached, to every step of the sequence must be valid code.) But while the first replacement works, the second does not, and I don't really see a difference:

\documentclass{article}
\usepackage{regexpatch}
\newcommand{\addinnertoouter}[1]{
    \regexpatchcmd{\test}
    { \c{outer} \{ (.*) \} \Z}
    { \c{outer} \cB\{ \1 \c{inner} \cB\{ #1 \cE\} \cE\} }
    {}{error}
}
\begin{document}
\ttfamily

\def\test{\outer{\inner{abc}}}
\meaning\test

\addinnertoouter{def}
\meaning\test

\addinnertoouter{ghi}
\meaning\test

\end{document}

Do you know about \tl_build_... described in interface3? — egreg, Sep 02 '19 at 10:42
@egreg I did not, thanks! One problem that I have is that I don't know when the sequence will be finished, so calling the required tl_build_end command will be difficult. (I might be able to do it before using the command.) — bers, Sep 02 '19 at 10:51

Phelype Oleinik · Accepted Answer · 2019-09-02T01:42:25.440

Using l3regex directly instead of \regexpatchcmd it works:

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\NewDocumentCommand \addinnertoouter { m }
  {
    \regex_replace_once:nnNTF
      { \c{outercmd} \{ (.*) \} \Z}
      { \c{outercmd} \cB\{ \1 \c{inner} \cB\{ #1 \cE\} \cE\} }
      \test
      { }
      { error }
  }
\ExplSyntaxOff
\begin{document}
\ttfamily

\def\test{\outercmd{\inner{abc}}}
\meaning\test

\addinnertoouter{def}
\meaning\test

\addinnertoouter{ghi}
\meaning\test
\end{document}

(\outer is a (very special) TeX primitive, so it's better if you use something else.)

The issue is that l3regex apparently makes the added def tokens catcode 12 (looks like a bug to me; I'll summon a wizard to check), and when \regexpatchcmd tries to retokenize them they become catcode 11, and \regexpatchcmd thinks (correctly) that the macro cannot be patched. If you don't mind that the inserted tokens be catcode 12 (in this case it probably won't matter), then \regex_replace_once:nnNTF will suit your needs.

Beware that the argument of your \addinnertoouter is still a regular expression, so if you do \addinnertoouter{\textit{jkl}} the output will not be what you expect. You can use the \u feature of l3regex to add an arbitrary token list (without catcode changes, so it will also work with \regexpatchcmd):

\documentclass{article}
\usepackage{regexpatch}
\ExplSyntaxOn
\tl_new:N \l_bers_tmpa_tl
\NewDocumentCommand \addinnertoouter { m }
  {
    \tl_set:Nn \l_bers_tmpa_tl {#1}
    \regexpatchcmd
      \test
      { \c{outercmd} \{ (.*) \} \Z}
      { \c{outercmd} \cB\{ \1 \c{inner} \cB\{ \u{l_bers_tmpa_tl} \cE\} \cE\} }
      { }
      { error }
  }
\ExplSyntaxOff
\begin{document}
\ttfamily

\def\test{\outercmd{\inner{abc}}}
\meaning\test

\addinnertoouter{def}
\meaning\test

\addinnertoouter{ghi}
\meaning\test

\addinnertoouter{\textit{jkl}}
\meaning\test
\end{document}

score 7 · Answer 2 · answered Sep 02 '19 at 01:06

7

Phelype Oleinik already described the problem with catcode changes in \regexpatchcmd, so I'm skipping the explanation part here ... The problem can also be fixed by replacing the line

{ \c{outer} \cB\{ \1 \c{inner} \cB\{ #1 \cE\} \cE\} }

by

{ \c{outer} \cB\{ \1 \c{inner} \cB\{ \cL(#1) \cE\} \cE\} }

in your original code, correctly giving

answered Sep 02 '19 at 01:06

siracusa

13,411

Thanks! Again, this is exactly what I need. Still, I have accepted the other answer as it was earlier, gives an explanation and a working solution. I do like the elegance of this answer, though - \cL is basically what I have been looking for. For all others searching, that stuff is explained in the l3regex package documentation. – bers Sep 02 '19 at 07:35
Hmmm - this stops working when I replace def by d1f, probably because \cL means "letters". Is there a more general catcode that I could use? – bers Sep 02 '19 at 08:30
@bers \cL d\cO 1\cL f will do; \cO in front of 1 is redundant, because digits have category code 12 by default, but clarity might be preferable. – egreg Sep 02 '19 at 10:32
@egreg I see. But that requires knowing what the parameters are - I get these from a different macro. What I did now is change the catcode of the digits. – bers Sep 02 '19 at 10:52
1

@bers You can do \def\temp{<the tokens>} and use \u{temp} for the replacement text. This will keep the original category codes. – egreg Sep 02 '19 at 11:11
1

@bers My answer only works for letter characters. The only general solution I can see is the one using the \u command as described by the other answers, because the regex replacement text must produce exactly the same catcodes for each token as active for the current catcode regime. – siracusa Sep 02 '19 at 22:10

score 2 · Answer 3 · answered Sep 02 '19 at 11:13

You can use the \u feature of l3regex:

\documentclass{article}
\usepackage{regexpatch}
\newcommand{\addinnertoouter}[1]{%
  \def\berstemp{#1}%
  \regexpatchcmd{\test}
    { \c{outer} \{ (.*) \} \Z}
    { \c{outer} \cB\{ \1 \c{inner} \cB\{ \u{berstemp} \cE\} \cE\} }
    {}{error}
}
\begin{document}
\ttfamily

\def\test{\outer{\inner{abc}}}
\meaning\test

\addinnertoouter{def}
\meaning\test

\addinnertoouter{ghi}
\meaning\test

\end{document}

Ah, that's the trick! I had tried \u{#1}... – bers Sep 02 '19 at 12:02 — bers, Sep 02 '19 at 12:02

Why does this regexpatch command only work once, not twice?

3 Answers3