Iterating through a list and modifying elements one at a time to construct new list

Question

Long time reader, first time poster.

I have a question about creating a macro that iterates through a list. I've read through some questions posted by other users, but all of those have something to do with using LaTeX in a complicated way. My question's about using plain TeX to iterate through a list and construct a new list by modifying each element in the old list. I'm currently self-learning TeX and I think programming some basic, but versatile and robust macros will help me get a better sense of how the guts of TeX work. Anyways, enough background, onto my question.

This is my code so far:

I defined the iterator recursively. It reads one argument at a time, determines if the argument is a comma or a stop marker (of my own definition, called \myStop), moves on if it's a comma, stops if it's the stop marker, and otherwise replaces the element by itself and the string (or token list) "+ 1".

\edef\myStop{)}
\def\Iterator#1{
        \ifx\myStop#1 %do nothing {end of list}
            \else %
                \ifx,#1 \expandafter\IteratorIntermediateOne % currently 2 if-statements deep
                    \else #1 + 1, \expandafter\IteratorIntermediateOne %currently 2 if-statements deep
            \fi%
        \fi%
}

I've had to define an intermediate Iterator called IteratorIntermediateOne to house the command \expandafter\Iterator because I'm not currently aware of a way to group terms after an \expandafter command in a way that would be equivalent to something like \expandafter{\expandafter\Iterator}\fi\fi. So I guess that's my first question: Is there a way to define nested \expandafter commands?

Now that everything's in context, here's all my code:

\edef\MyList{1,2,3}
\edef\myStop{)}
\def\IteratorIntermediateOne{\expandafter\Iterator}
\def\Iterator#1{%
        \ifx\myStop#1 %do nothing {end of list}
            \else %
                \ifx,#1 \expandafter\IteratorIntermediateOne % currently 2 if-statements deep
                    \else #1 + 1, \expandafter\IteratorIntermediateOne %currently 2 if-statements deep
            \fi%
        \fi%
}
\edef\newList{\expandafter\Iterator\MyList\myStop}

Disclaimer: I'm aware of the extra comma this places after the last element. I am also aware there aren't enough cases to detect poorly constructed lists. I don't know enough about TeX to even begin to imagine how to safeguard the macro against improperly constructed token lists, so apologies if the more experienced of you out can't help but laugh at this code.

Okay, my other question is this: Is there a more efficient way to define a macro that calls itself? Is it better practice to define an iterator macro separately and call it using TeX's built-in \loop command? If so, can someone walk me through how I would do that, because I'm having trouble making sense of the loop call in the context of TeX's eye's, mouth, gullet, and stomach processes. Does the loop expand the macro, reach \repeat pass it on to the stomach and return back to the macro? I can't find a good explanation anywhere.

Thanks for all your help!

Answer 1

The intention seems to be to add 1 to each item I would code it more like this (assuming etex)

\edef\MyList{1,2,3,25,456,2}
\def\Iterator#1{\expandafter\xiterator#1\stopiteration,}
\def\xiterator#1,{\the\numexpr#1+1\relax,\xiterator}
\def\stopiteration#1\relax#2\xiterator{#1\relax}
\message{\Iterator\MyList}
\bye

which makes the message

2,3,4,26,457,3 

Answer 2

\input listofitems
\def\MyList{1,2,3,25,456,2}
\def\processlist#1{%
  \readlist\myterms\MyList
  \foreachitem\z\in\myterms{%
    \ifnum\zcnt=1\else,\fi
    \the\numexpr\z+1\relax
  }%
}

\processlist\Mylist
\bye

If you actually need to save the updated list, we can do so in the \mytoks token list:

\input listofitems
\newtoks\mytoks
\def\MyList{1,2,3,25,456,2}
\def\processlist#1{%
  \mytoks{}%
  \readlist\myterms\MyList
  \foreachitem\z\in\myterms{%
    \ifnum\zcnt=1\else\mytoks\expandafter{\the\mytoks,}\fi
    \mytoks\expandafter\expandafter\expandafter{%
      \expandafter\the\expandafter\mytoks\the\numexpr\z+1\relax}
  }%
}
\processlist\Mylist
List is \the\mytoks
\bye

Answer 3

You said:

...so apologies if the more experienced of you out can't help but laugh at this code.

When I started learning TeX, I had the feeling that it was a very steep learning curve.

From time to time I was frustrated. ;-)

I do not believe that people like you, who are taking on this learning curve, are in a situation where it is appropriate to laugh at their attempts at macro programming/TeX-programming.

I think any attempt to achieve or learn something good in a way which both is also based on reason and in itself is no misdeed, basically deserves respect.

If you have questions about how the code from my examples below works, don't hesitate to ask them. It is then useful to describe how you think the code works and where you get stuck with the understanding. In my experience, this makes it easier for the respondents to find out exactly what information (e.g. about how TeX primitives work and which of the "side effects" briefly hinted at in the back chapters of the TeXbook are used for programming tricks) is still missing for understanding.

---

Assuming that the entries in your comma-list are not surrounded by spaces, and that the \relax-primitive does not occur within the comma-list, and that \numexpr from the ε-TeX-extensions is available, you can probably do something like this:

\long\def\gobble#1{}%
\long\def\firstofone#1{#1}%
\def\Iterator#1#2,{%
  % #1 - element-separator to prepend; empty in the 1st iteration;
  %      comma in consecutive iterations
  % #2 - either current element of old list or the \relax that was
  %      appended for denoting the end of the list
  \ifx\relax#2\expandafter\gobble\else\expandafter\firstofone\fi
  {%
    #1\number\numexpr#2+1\relax\Iterator{,}%
  }%
}%
\def\MyList{1,2,3}
\edef\newList{\expandafter\Iterator\expandafter{\expandafter}\MyList,\relax,}
\begingroup
\tt
\string\MyList: \meaning\MyList
\string\newList: \meaning\newList
\endgroup
\bye

The gist with this example is:

Within the \edef-definition-text of \newList the TeX-primitive \expandafter is used for expanding \MyList. Also the sequence ,\relax, is appended.

This way at the time of defining \newList \edef-driven expansion of the definition-text of \newList at some stage yields a sequence \Iterator{}Comma,sparated,items,from,\MyList,\relax,.

So \relax, marks the end of the list.

Now—still driven by \edef-expansion—\Iterator (recursively) picks a non-delimited-argument #1 (which in the first iteration is empty and in consecutive iterations holds a comma, i.e., which holds the separator to prepend to the item of the new list) and a comma-delimited-argument #2 which either holds the next item coming from \myList's comma-list or holds the end-marker \relax , and in any case places—nested in curly braces— a token-sequence denoting the next iteration, formed by

• the undelimited-argument #1, i.e., the separator that must precede the next item of the new list,
• the expression \number\numexpr#2+1\relax for adding 1 to the value represented by the comma-delimited-argument and this way forming the next item of the new list,
• a call to itself for processing the next item remaining from the expansion of \myList, this time providing a comma within the undelimited argument, denoting that next time the next item of the new list is to be preceded by a comma.

Via \ifx\relax#2 it is checked whether the end of the comma-list/the \relax appended to the list in the beginning of \edef-expansion is reached. If so, the token-sequence nested in curly braces denoting the next iteration is "gobbled/removed" via \gobble and thus not carried out, which terminates iteration/tail-recursion. If not so, the surrounding curly braces are removed from that sequence by applying \firstofone whereafter that sequence gets processed.

The undelimited argument #1 of \Iterator, which holds the separator to prepend to the item of the new list, only in the first iteration is empty. In each consecutive iteration it holds a comma which in that consecutive iteration's previous iteration was provided by the \Iterator-macro itself as a part of the token-sequence which then formed the next iteration. This way (only) the first item of the new list is not preceded by a comma.

---

If you don't have ε-TeX-extensions' \numexpr available I can offer a routine for incrementing non-negative integer numbers. (In "real life" you might be interested in the packages intcalc and bigintcalc.)

%------------------------------------------------------------------------------
% Expandable incrementing of non-negative integer number formed by a sequence
% of explicit catcode-12-character-tokens from the set {0,1,2,3,4,5,6,7,8,9}
%..............................................................................
% \Increment{<non-negative integer number k as sequence of explicit
%              catcode-12-character-tokens from the set 0123456789>}
% ->
% <natural number (k+1) as sequence of explicit catcode-12-character-tokens
%  from the set 0123456789>
% In expansion-contexts the result is delivered after two expansion-steps/is 
% obtained by "hitting" \Increment with \expandafter twice.
%------------------------------------------------------------------------------
\def\Increment#1{%
  \romannumeral0%
  \IncrementReverse{\IncrementFork{}}{\relax}{}#1\relax
}%
\def\IncrementReverse#1#2#3#4{%
  % #1 - tokens to prepend to reversed list
  % #2 - tokens to append to reversed list
  % #3 - reversed list constructed so far
  % #4 - current element of not-reversed list
  \ifx\relax#4%
    \expandafter\firstoftwo
  \else
    \expandafter\secondoftwo
  \fi
  {#1#3#2}{\IncrementReverse{#1}{#2}{#4#3}}%
}%
\def\IncrementSelect#10123456789\relax#2#3!!{#2}%
\def\IncrementFork#1#2{%
  % #1 - digits incremented so far
  % #2 - current digit to increment or end-marker \relax
  \IncrementSelect
  #2123456789\relax{\IncrementReverse{ }{}{}#11}%
  0#223456789\relax{\IncrementReverse{ }{}{}#12}%
  01#23456789\relax{\IncrementReverse{ }{}{}#13}%
  012#2456789\relax{\IncrementReverse{ }{}{}#14}%
  0123#256789\relax{\IncrementReverse{ }{}{}#15}%
  01234#26789\relax{\IncrementReverse{ }{}{}#16}%
  012345#2789\relax{\IncrementReverse{ }{}{}#17}%
  0123456#289\relax{\IncrementReverse{ }{}{}#18}%
  01234567#29\relax{\IncrementReverse{ }{}{}#19}%
  012345678#2\relax{\IncrementFork{#10}}%
  0123456789#2{\IncrementReverse{ }{}{}#11\relax}%
  0123456789\relax{\IncrementReverse{ }{}{}#11#2}%
  !!%
}%
%%-----------------------------------------------------------------------------
\long\def\firstoftwo#1#2{#1}%
\long\def\secondoftwo#1#2{#2}%
\def\Iterator#1#2,{%
  % #1 - element-separator to prepend
  % #2 - current element of old list
  \ifx\relax#2\expandafter\firstoftwo\else\expandafter\secondoftwo\fi
  {}{%
    #1\Increment{#2}\Iterator{,}%
  }%
}%
\def\MyList{1,2,3}
\edef\newList{\expandafter\Iterator\expandafter{\expandafter}\MyList,\relax,}
\begingroup
\tt
\string\MyList: \meaning\MyList
\string\newList: \meaning\newList
\endgroup
\bye

If you wish a routine which does without \edef, you can, e.g., use the \romannumeral0-expansion- and argument-exchanging-technique—the gist of \romannumeral0-expansion is:

• TeX expands expandable tokens while gathering tokens that belong to the ⟨number⟩-quantity that is to be represented in roman numerals.
• If the first token which TeX finds while gathering the ⟨number⟩-quantity is a digit, e.g., 0, then the process of gathering tokens that belong to the ⟨number⟩-quantity turns into a process of gathering more digits or something that is not a digit and therefore terminates the process of gathering. Expandable tokens get expanded while gathering digits. A space-token terminating a digit-sequence terminates the process of gathering more digits and gets silently discarded.
• If the number gathered is not positive, TeX will silently swallow the tokens forming the ⟨number⟩-quantity without delivering any token in return.

This implies that \romannumeral can be used for tricking TeX into doing a lot of expansion- and argument-exchanging-work as long as it is ensured that in the end a non-positive number is found.

%------------------------------------------------------------------------------
% Expandable incrementing of non-negative integer number formed by a sequence
% of explicit catcode-12-character-tokens from the set {0,1,2,3,4,5,6,7,8,9}
%..............................................................................
% \Increment{<non-negative integer number k as sequence of explicit
%              catcode-12-character-tokens from the set 0123456789>}
% ->
% <natural number (k+1) as sequence of explicit catcode-12-character-tokens
%  from the set 0123456789>
% In expansion-contexts the result is delivered after two expansion-steps/is 
% obtained by "hitting" \Increment with \expandafter twice.
%------------------------------------------------------------------------------
\def\Increment#1{%
  \romannumeral0%
  \IncrementReverse{\IncrementFork{}}{\relax}{}#1\relax
}%
\def\IncrementReverse#1#2#3#4{%
  % #1 - tokens to prepend to reversed list
  % #2 - tokens to append to reversed list
  % #3 - reversed list constructed so far
  % #4 - current element of not-reversed list
  \ifx\relax#4%
    \expandafter\firstoftwo
  \else
    \expandafter\secondoftwo
  \fi
  {#1#3#2}{\IncrementReverse{#1}{#2}{#4#3}}%
}%
\def\IncrementSelect#10123456789\relax#2#3!!{#2}%
\def\IncrementFork#1#2{%
  % #1 - digits incremented so far
  % #2 - current digit to increment or end-marker \relax
  \IncrementSelect
  #2123456789\relax{\IncrementReverse{ }{}{}#11}%
  0#223456789\relax{\IncrementReverse{ }{}{}#12}%
  01#23456789\relax{\IncrementReverse{ }{}{}#13}%
  012#2456789\relax{\IncrementReverse{ }{}{}#14}%
  0123#256789\relax{\IncrementReverse{ }{}{}#15}%
  01234#26789\relax{\IncrementReverse{ }{}{}#16}%
  012345#2789\relax{\IncrementReverse{ }{}{}#17}%
  0123456#289\relax{\IncrementReverse{ }{}{}#18}%
  01234567#29\relax{\IncrementReverse{ }{}{}#19}%
  012345678#2\relax{\IncrementFork{#10}}%
  0123456789#2{\IncrementReverse{ }{}{}#11\relax}%
  0123456789\relax{\IncrementReverse{ }{}{}#11#2}%
  !!%
}%
%%-----------------------------------------------------------------------------
\long\def\firstoftwo#1#2{#1}%
\long\def\secondoftwo#1#2{#2}%
\long\def\exchange#1#2{#2#1}%
\def\Iterator#1,#2\relax#3#4{%
  % #1 - current element of old list
  % #2 - remaining elements of old list
  % #3 - element-separator to prepend
  % #4 - new list constructed so far
  \ifx\relax#1\expandafter\firstoftwo\else\expandafter\secondoftwo\fi
  { #4}{%
    \expandafter\exchange
    \expandafter{%
    \expandafter{%
    \romannumeral0%
    \expandafter\expandafter\expandafter\exchange
    \expandafter\expandafter\expandafter{%
    \Increment{#1}}{ #4#3}}}{\Iterator#2\relax{,}}%
  }%
}%
\def\MyList{0,1,2,3}
\expandafter\def
\expandafter\newList
\expandafter{%
\romannumeral0\expandafter\Iterator\MyList,{\relax},\relax{}{}}%
\begingroup
\tt
\string\MyList: \meaning\MyList
\string\newList: \meaning\newList
\endgroup
\bye

Answer 4

Since you're new, you can start with expl3.

\documentclass{article}
\usepackage{xparse,xfp}
\ExplSyntaxOn
\NewDocumentCommand{\generatelist}{mmm}
 {% #1=output, #2=input, #3=iterator
  \harry_list_generate:nnn { #1 } { #2 } { #3 }
 }
% variables
\clist_new:N \l__harry_list_input_clist
\clist_new:N \l__harry_list_output_clist
% the main function
\cs_new_protected:Nn \harry_list_generate:nnn
 {
  % if the input is a single token, assume it is a control sequence
  \tl_if_single:nTF { #2 }
   { \clist_set_eq:NN \l__harry_list_input_clist #2 }
   { \clist_set:Nn \l__harry_list_input_clist { #2 } }
  % now \l__harry_list_input_clist contains the input
% clear the output list
  \clist_clear:N \l__harry_list_output_clist
% map the input list applying the iterator to each item
  \clist_map_inline:Nn \l__harry_list_input_clist
   {
    \clist_put_right:Nx \l__harry_list_output_clist { #3 { ##1 } }
   }
% make the output list
  \clist_set_eq:NN #1 \l__harry_list_output_clist
 }
\ExplSyntaxOff
% two example iterators
\newcommand{\addone}[1]{\inteval{#1+1}}
\newcommand{\addhyphens}[1]{-#1-}
% a control sequence expanding to a list
\newcommand{\List}{1,2,3,41}
\generatelist{\ListA}{\List}{\addone}
\generatelist{\ListB}{1,2,3}{\addhyphens}
\show\ListA
\show\ListB

This outputs

> \ListA=macro:
->2,3,4,42.
l.50 \show\ListA
?
> \ListB=macro:
->-1-,-2-,-3-.
l.51 \show\ListB