Regular expressions ... how to also return the sequence splitting characters?

Question

So, we have this great regex code by User Andrew as an answer to my OP "Automatically put certain inputs (e.g. punctuation marks) outside of the environment/command":

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\seq_new:N \l_word_seq % define a new seqence
\NewDocumentCommand\IterateOverPunctutation{ m m }{
  % apply "function" #2 to the "words" in #1 between the punctuation characters
  \regex_split:nnN { [\.\,\;\:\s]+ } { #1 } \l_word_seq% split the sequence
  \seq_map_function:NN \l_word_seq {#2}% apply #2 to each word in \l_word_seq
}
\ExplSyntaxOff

\begin{document}

\newcommand\Any[1]{``\textbf{#1}''\space}% a dummy \Any command

\IterateOverPunctutation{A, B: C. D}\Any

\end{document}

Which neatly produces:

And so it goes from:

A, B: C. D

to

“A” “B” “C” “D”

But my follow-up question now is ... how to go from (this is just an example ... see Andrew's code or linked OP for more info):

A, B: C. D

to e.g.

“A”, “B”: “C”. “D”

That is, how to still also output/repeat the separators (in this case , and . and . respectively)?

Steven B. Segletes · Accepted Answer · 2017-10-05T18:42:26.510

4

The listofitems package can parse the input, either using nested levels of parsing, or (as is done here) with multiple (co-equal) parsing separators. The macro \setsepchar{.||,||:} sets the coequal parsing characters ., ,, and :. No final separator (i.e., after the "D") is assumed, but can be changed if I misinterpreted the conditions.

While (in my MWE) \mylist[<i>] gives the i'th data field, \mylistsep[<i>] gives the separator following the i'th data field.

\documentclass{article}
\usepackage{listofitems}
\newcommand\punctiterate[1]{%
  \setsepchar{.||,||:}%
  \readlist*\mylist{#1}%
  \foreachitem\i\in\mylist{\ifnum\icnt>1\relax\mylistsep[\the\numexpr\icnt-1] \fi``\i''}
}
\begin{document}
\punctiterate{A, B: C. D}
\end{document}

ADDENDUM

The OP would like to have \pars embedded in the input. To achieve this goal, I use a 2-level parsing, with \par as the 2nd-level parsing separator. If a \par is present in any 1st-level item, then the 2nd-level list length is 2, otherwise it is 1. I can use this fact to intercept the presence of a \par and act accordingly.

[UPDATE: Christian has updated the listofitems package to V1.4, available at https://ctan.org/tex-archive/macros/generic/listofitems. With the update, \par is once again available for use as a list separator.]

\documentclass{article}
\usepackage{listofitems}
\newcommand\punctiterate[1]{%
  \setsepchar{.||,||:/\par}%
  \readlist*\mylist{#1}%
  \foreachitem\i\in\mylist{%
    \ifnum\listlen\mylist[\icnt]>1\relax\par``\mylist[\icnt,2]''\else%
      ``\i''\fi\ifnum\icnt<\mylistlen\relax\mylistsep[\icnt] \fi%
  }%
}
\begin{document}
\punctiterate{A, B: C. \par D}
\end{document}

edited Oct 05 '17 at 18:42

answered Oct 02 '17 at 12:20

Steven B. Segletes

237,551

Somehow the code jams if there is a \par e.g in \punctiterate{A, B: C. \par D}... Do you see any solution there please? – O0123 Oct 02 '17 at 12:58
Another question: your answer shows it is possible to surround each item with curly quotes ... but is it also possible to insert other commands around each item? For example \textbf{...}? – O0123 Oct 02 '17 at 14:31
@VincentMiaEdieVerheyen How am I to interpret the \par? Like a letter, like punctuation, or like something else. That is to say, what do you expect the output to look like if there is a \par in the input? – Steven B. Segletes Oct 02 '17 at 14:35
@VincentMiaEdieVerheyen Using \textbf is no problem...just change \`\i''to\textbf{\i}` – Steven B. Segletes Oct 02 '17 at 14:36
I mean the regular definition and use of Tex's \par, to close a paragraph. You are right in that \textbf is not a problem, my apologies ... I am having problems with another function\tooltip{\i}, but that is beyond the scope of this question. – O0123 Oct 02 '17 at 14:45
@VincentMiaEdieVerheyen I understand you mean the regular definition of \par, but I would like an explanation of how you want the output to look, when \par is in the input. – Steven B. Segletes Oct 02 '17 at 14:46
I guess to make the output look just as you would expect from a \par. :) To keep Tex able from interpreting it as a \par and outputting as such. So nothing different from what it would mean outside your code. – O0123 Oct 02 '17 at 15:24
@VincentMiaEdieVerheyen Please see my addendum. – Steven B. Segletes Oct 02 '17 at 15:27
Thank you. Please inform us once the listofitems-package has been updated to v1.4. – O0123 Oct 03 '17 at 01:34
@VincentMiaEdieVerheyen The listofitems package has been updated and is available at https://ctan.org/tex-archive/macros/generic/listofitems. – Steven B. Segletes Oct 05 '17 at 18:42

score 2 · Answer 2 · answered Feb 10 '19 at 08:02

You want to grab the separators in \regex_split:nnN. This is precisely what groups are for: just add parentheses to your pattern and whatever these parentheses match will show up in the resulting sequence. Then when mapping through the sequence we need to keep track of odd versus even items because they will be words and separators respectively. You may want to apply further regex processing on the separators to distinguish spaces and punctuation, and so on. It may be useful to use a regex with more than one group (pair of parentheses) to separately grab the two, something like ([\.\,\;\:])([\s\c{par}]+). Then the loop through the sequence would have to test \int_mod:nn { \l_word_int } { 3 } of course.

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\seq_new:N \l_word_seq % define a new sequence
\int_new:N \l_word_int % counts words to distinguish words and separators
\NewDocumentCommand\IterateOverPunctutation{ m m m }{
  % apply "function" #2 to the "words" in #1 between the punctuation characters
  \regex_split:nnN { ([\.\,\;\:\s]+) } { #1 } \l_word_seq% split the sequence
  \int_zero:N \l_word_int
  \seq_map_inline:Nn \l_word_seq
    {
      \int_incr:N \l_word_int
      \int_if_odd:nTF { \l_word_int }
        { #2 {##1} } % apply #2 to each word in \l_word_seq
        { #3 {##1} } % apply #3 to each separator in \l_word_seq
    }
}
\ExplSyntaxOff

\begin{document}

\newcommand\Any[1]{``\textbf{#1}''}% a dummy \Any command
\newcommand\Thing[1]{\underline{#1}}% a dummy \Thing command

\IterateOverPunctutation{A, B: C. D}\Any\Thing

\end{document}

I didn't use \seq_indexed_map_inline:Nn because it has only been added 1 year ago and not everyone may have it. With it you can avoid needing the int variable.

Regular expressions ... how to also return the sequence splitting characters?

2 Answers2

Linked