Most efficient way to reverse a token list (tail recursion)

Question

Goal: reverse a list of characters, going from \def\mylist{abcdefgh} to \def\mylist{hgfedcba}. This is easy using a marker which does not appear in the list, such as \relax:

\def\mylist{abcdefgh}
\def\reverse #1%
  {\edef #1{\expandafter \reverseloop #1\relax \marker }}
\def\reverseloop #1#2\marker
  {\ifx#1\relax\reverseend\fi \reverseloop #2\marker #1}
\def\reverseend #1\marker #2{}
\reverse\mylist
\show\mylist

So far, so good. Unfortunately, this wastes a large amount of memory, and trying to apply the same function when \mylist has a few thousand characters already blows up. Indeed, each call to \reverseloop reads the whole token list as its #2 argument, and this is not flushed from TeX's memory via tail recursion, because TeX never reaches the end of the replacement text of \reverseloop, or rather, only reaches it at the very end, once all the \reverseloop macros have been expanded. You can see this from the call trace in

\def\fiveup{\edef\mylist{\mylist\mylist\mylist\mylist\mylist}}
\fiveup \fiveup \fiveup \fiveup
\tracingall
\reverse\mylist

Thus, the whole process consumes a memory proportional to the square of the number of characters, reaching millions, typical size of TeX's main memory. How can I implement such a reversal using only a linear amount of memory?

It should easily scale up to 100000 characters, albeit maybe be a bit slow there: of course we cannot avoid a quadratic time. I don't care too much about expandability.

Answer 1

\def\firstoftwo#1#2{#1}
\def\secondoftwo#1#2{#2}

\def\rev#1#2\revA#3\revB{%
  \if\relax\detokenize{#2}\relax
    \expandafter\firstoftwo
  \else
    \expandafter\secondoftwo
  \fi{#1#3}{\rev#2\revA#1#3\revB}}

\edef\x{\rev abcde\revA\revB}\show\x

A string with 10000 characters is reversed in about 20 seconds on my machine, without clobbering the memory.

For your list I get

8.16 real         5.16 user         0.05 sys

(just because I had to react to \show)

In #3 there is the "reversed-so-far" string; at each step of the recursion I put in front of it the first token in the remaining string, which is #1#2. When #2 is empty, the recursion ends.

The "linear" reversing should be obtained by

\catcode`\@=11
\def\reverse#1{\count@=\z@\def\temp{}
  \expandafter\doreverse#1\doreverse
  \loop\ifnum\count@>\z@
    \edef\temp{\temp\csname @@\romannumeral\count@\endcsname}%
    \advance\count@\m@ne
  \repeat
  \expandafter\def\expandafter#1\expandafter{\temp}%
}
\def\doreverse#1{%
  \unless\ifx#1\doreverse
    \advance\count@\@ne
    \expandafter\def\csname @@\romannumeral\count@\endcsname{#1}%
    \expandafter\doreverse
  \fi}
\catcode`\@=12

which is limited only by available memory, using the space for control sequences.

With \def\mylist{<string>}, \reverse\mylist defines successively \i, \ii and so on to the tokens forming the list and at the end stores them back in reverse order in \temp to which \mylist is then made equivalent. So after

\def\mylist{abcdefgh}
\reverse\mylist

\mylist will expand to hgfedcba. It doesn't work as is for braced groups, but the modification in that case should be trivial.

I've reversed a 40000 character long string in 42 seconds. TeX refuses to do a 100000 character long string, because it exhausts the pool size. (I removed \begingroup and \endgroup as it makes run away of save size.)

Answer 2

I didn't check the memory, but a lua solution would be:

\def\StrRev#1{\directlua{tex.print(string.reverse('#1'))}}

abcdefgh\par
\StrRev{abcdefgh}

Which prints:

I measured the running time of two string lengths. On my machine:

Using \nullfont

100 000 chars : 0.21 s
1 000 000 chars : 1.24 s

With font

100 000 chars : 0.74 s
1 000 000 chars : 6.21 s

Answer 3

Here is an expandable solution that preserves outer braces and order of entries:

\documentclass{article}
\makeatletter
\let\xp\expandafter
\@ifdefinable\leftbracechar\relax
\edef\leftbracechar{\xp\@gobble\string\{}
\newcommand*\ifstrsame[2]{%
  \ifnum\pdfstrcmp{#1}{#2}=\z@\xp\@firstoftwo\else
    \xp\@secondoftwo\fi
}
\newcommand*\ifleftbrace[1]{%
  \ifstrsame{\detokenize\xp{\@gobble#1.}}{}\@secondoftwo{%
    \ifstrsame{\xp\@car\detokenize{#1}\@nil}\leftbracechar
      \@firstoftwo\@secondoftwo
  }%
}
\begingroup
\catcode`\&=3
\gdef\preservebracereverse#1{\pr@reverse{}&#1\@nnil}
\gdef\pr@reverse#1#2\@nnil{%
  \pr@rev@rse{#1}{\xp\ifleftbrace\xp{\@gobble#2}}#2\@nnil
}
\gdef\pr@rev@rse#1#2&#3{%
  \xp\ifx\@car#3\@nil\@nnil\xp\@firstoftwo\else\xp\@secondoftwo\fi
  {\unexpanded{#1}}
  {#2{\pr@reverse{{#3}#1}}{\pr@reverse{#3#1}}&}%
}
\endgroup
\makeatother

% Example:
\edef\x{\preservebracereverse{{ax}bcd{ey}}}
\show\x

\begin{document}

\end{document} 

Answer 4

Warning, this would take approximately 7 minutes to run, if you expand the list \XXX rather than \ABC as shown in the minimal below:

\documentclass{article}
\usepackage{lipsum}
\edef\ABC{So far, so good. Unfortunately, this wastes a large amount of memory, and trying to apply the same function when mylist has a few thousand characters already blows up. Indeed, each call to reverseloop reads the whole token list as its 2 argument, and this is not flushed from TeX's memory via tail recursion, because TeX never reaches the end of the replacement text of reverseloop, or rather, only reaches it at the very end, once all the reverseloop macros have been expanded. You can see this from the call trace in.So far, so good. Unfortunately, this wastes a large amount of memory, and trying to apply the same function when mylist has a few thousand characters already blows up. Indeed, each call to reverseloop reads the whole token list as its 2 argument, and this is not flushed from TeX's memory via tail recursion, because TeX never reaches the end of the replacement text of reverseloop, or rather, only reaches it at the very end, once all the reverseloop macros have been expanded. You can see this from the call trace in.\par}
\edef\X{\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC\ABC}
\edef\XX{\X \X \X \X}
\edef\XXX{\XX\XX}
\makeatletter
\begin{document}
\makeatletter
\let\stack\@empty
\def\add@element#1{%
  \def\element{#1}%
  \push@element
}
\def\push@element{%
   \xdef\stack{\element\space \stack}
}
\newcounter{cnt}
\expandafter\@tfor\expandafter\next\expandafter:\expandafter=\ABC \do{%
  \add@element{\next}
  \stepcounter{cnt}
}
\stack

\thecnt

\end{document}

It reverses 116552 characters over 60 pages. I used \edef to store the list. I iterate over the list using LaTeX \@tfor and then pushed it onto a stack. When the stack is expanded it prints the list in reverse.

A real TeXnical solution for the list as presented would be to put the letters in a box one letter wide and then split the box in a loop.

The best solution would be to sort the list as you capture the letters, i.e, before you insert them in the list.

Answer 5

_{I don't know what OP might have came up with in the 11 years, but there doesn't seem to be anything "interesting" in source3.pdf, in particular \tl_reverse_items:n takes O(n²) time. Not sure about the memory, and there isn't a \str_reverse_items:n.}

---

With \expanded primitive available it would be possible in O(n log n). (divide and conquer.)

Otherwise, I think best case (with f-type expansion or similar) is O(n √n).

The idea is that, we

• count the number of items,
• reverse first √n items (takes O(n) time),
• throw it after the remaining n-√n items (takes O(n) time)
• then continue to reverse the remaining n-√n items.

Each time √n items are reversed, the time taken is O(n), so total time complexity is O(n √n).

---

I try implementing it. The implementation takes 2.83s to reverse 524288 characters on LuaLaTeX. Looks as expected.

The implementation only handles non-space string characters however, although the convert-everything-to-space part can also be implemented in O(n √n) using the same idea described above, I would not expect a large difference.

For comparison \tl_reverse_items:n takes 1.74s to reverse 8192 items.

(although the comparison is unfair against \tl_reverse_items:n since it does not grab 8 items at a time & have to return the braces)

---

I think this is optimal in this condition, as even the simple task of expandably collecting n undelimited items in the input stream and put them in a group seems to require quadratic time in n without \expanded (but takes linear time with)

Time complexity O(n log n) is easy.
(because everything (except avoiding hash halving) \expanded can do, \edef can do unexpandably in similar time complexity. At least in this particular case where it doesn't need to nest, so \edef can replace the role of \expanded in the divide-and-conquer approach)

---

(the implementation is not very convenient to post because it depends on a bunch of unpublished libraries etc.)

---

Alternatively, there is a solution that takes O(kn) time and O(k × n^(1/k)) tok registers (or hash table entries, assuming TeX hashing takes O(1))

Below I'll describe the solution that takes O(n) (see note below) time and O(√n) tok registers. Generalizing (e.g. to a solution that takes O(n) time and O(∛n) tok registers) is not difficult.

• split the string into √n parts, store each into a tok register. This can be done in linear time.
• reverse each part (whose size is √n) in linear time in the part size, using √n tok registers.
• concatenate the parts into the result in reverse order. This can also be done in linear time.

(I try implementing this one and it takes 1.45s to reverse 524288 characters. Marginally faster than the above I guess, although the implementation could be optimized a bit e.g. chunking 8 characters at once in the latter half.)

\tl_build_put_left implementation, despite having time complexity O(n log₅ n) instead of O(n), is surprisingly fast at 1.74s.

---

In retrospect, there's a problem with this approach, namely that a number in range 1..n takes log₁₀(n) decimal digits to represent, so actual time complexity is O(n log n).

In my algorithm, both the string split part and the concatenate part are affected by this.

So, workaround...

• first, for the string split part, I can't come up with any way better than to define √n control sequences to count the steps. Each control sequence would need to have csname length at least log₁₀(√n) = O(log n).
• for the concatenating part, similarly define √n control sequences each \toksdef'ed to the corresponding toks register, then define √n control sequences each expand to the following control sequence plus the \toksdef'ed control sequence, so the full expansion of one of them would equal to the concatenation of all these √n toks registers.

Time complexity of these parts is O(√n log n), but fortunately they only need to be done once, so they're insignificant compared to O(n).

Another workaround (for the concatenating part only) is, to split into parts of size O(log n), then handle that with the naive algorithm, time complexity would be O(n log log n).
Applying that idea recursively, I guess it would become O(n log^* n) or something similar.

Final time complexity is still O(n), but memory usage becomes O(√n log n). Of course similar idea exists for O(∛n log n) etc., although I haven't worked out the details completely.

---

Another (rather weird in my opinion) workaround is to use the e-TeX extension \currentiflevel which can be incremented and decremented expandably in O(1).

I test the limit of this one, on LuaLaTeX it seems to be unbounded, but on LaTeX, PDFLaTeX and XeLaTeX it runs to more than 2000000 until reports "TeX capacity exceeded", so in any case that's large enough for any purpose and in theoretical analysis we can assume it takes linear memory and only bounded above by the available memory. (unlike e.g. the \romannumeral nesting level, \expandafter nesting level or \dimexpr nesting level etc.)

Although if the value is already a large value (unlikely in practice, but possible in theory) it might be a little difficult, not sure... I can see a few ways

• repeatedly execute \fi until it reaches a small value, then repeatedly execute \ifcase\z@ until it reaches the original value (I think this cannot affect the behavior of normal program...? Except possibly changing the content in the log I was wrong, \currentiftype exists)

• hope that TeX implementation evaluates the internal number \numexpr\currentiflevel-\originalvalue\relax in O(1) time where ⟨internal number⟩ is expected, which is \the\toks\numexpr\currentiflevel-\originalvalue\relax in this case.

  (this is in theory possible, as at no point in time is O(log n) tokens generated, although I'm not sure how it's implemented in reality)

Answer 6

I am not sure about the tail recursion, but maybe:

\catcode`@=11
\input lambda.sty
\catcode`@=12
\Show\Reverse[a,b,c,d,e,f,g,h] % => [h,g,f,e,d,c,b,a]
\bye

could be of interest here (lambda.sty).

Answer 7

Here is a solution (in pdfTeX only) that runs in linear time but breaks above 400000 characters or so. I got the idea from the answer by user202729. Convert each character to four base 4 digits (we could also do base 16, or even up to base 40, but it's more annoying to code so many cases later on), then for each digit produce \iftrue or \iffalse\else or \ifcase0 or \ifdefined\rev@toif, which are all true. When the whole string has been read, TeX has in memory a whole lot of nested ifs (four times the length) and we can retrieve their type using \currentiftype! Close the conditionals along the way, and have as an end marker some other type of conditional. The code is not even too long. The same could be done with boxes or skips or similar, but I think that TeX goes (quickly) through a list when it adds another item to it, so the time would really be quadratic.

Anyway, the solution below is seemingly competitive with the times mentioned by user202729.

\catcode`@=11\relax
% The idea is to store the string as collections of 4 nested ifs, in TeX's memory for that.
% When unrolling, \currentiftype is among 15,16,17,18 so we must subtract 15(1+4+16+64)
\long\def\reverse#1{\if..\expandafter\rev@loop\detokenize{#1}\iffalse\fi\fi\rev@unroll\fi}
\def\rev@unroll{%
  \expandafter\ifnum\the\currentiftype=1 \rev@end\fi
  \rev@chr{\currentiftype\fi+4\currentiftype\fi+16\currentiftype\fi+64\currentiftype\fi-1275}%
  \rev@unroll}
\def\rev@end#1\rev@unroll{\fi}
% \rev@chr{<numexpr>} produces a catcode-12 char with that charcode in [0,255]
\def\rev@chr#1{\csname rev@chr-\the\numexpr#1\relax\endcsname}
\begingroup
  \catcode0=12\relax
  \lccode0=0\relax
  \def\rev@loop{
    \lowercase{\expandafter\gdef\csname rev@chr-\the\lccode0\endcsname{^^@}}
    \ifnum\lccode0<255
      \lccode0=\numexpr\lccode0+1\relax
      \rev@loop
    \fi
  }
  \rev@loop
\endgroup
\def\rev@loop#1{\iffalse#1\fi\expandafter\rev@A\number`#1;\rev@loop}
\def\rev@A#1;{\expandafter\rev@B\the\numexpr(#1+32)/64-1\expandafter;%
  \the\numexpr(#1+8)/16-1\expandafter;%
  \the\numexpr(#1+2)/4-1\expandafter;#1;}
\def\rev@B#1;#2;#3;#4;{\rev@toif#1%
  \expandafter\rev@toif\the\numexpr#2-4#1\relax
  \expandafter\rev@toif\the\numexpr#3-4#2\relax
  \expandafter\rev@toif\the\numexpr#4-4*#3\relax}
\def\rev@toif#1{\expandafter\rev@toif@aux\ifcase#1
  \empty\iftrue\fi
  \or \iffalse\else\fi
  \or \ifcase\z@\fi
  \or \ifdefined\rev@toif\fi
  \else \ERROR \ERROR ?\fi}
\def\rev@toif@aux#1#2#3{\expandafter #1\expandafter #2}
\catcode`@=12\relax
% ----- test the code
\ExplSyntaxOn
\RequirePackage{l3benchmark}
% with 400,4000,40000,400000 characters (it starts blowing up after that)
\edef\foo{\prg_replicate:nn{100}{^Y+W}}
\benchmark:n { \edef\foo{\expandafter\reverse\expandafter{\foo}} }
\edef\foo{\prg_replicate:nn{1000}{^Y+W}}
\benchmark:n { \edef\foo{\expandafter\reverse\expandafter{\foo}} }
\edef\foo{\prg_replicate:nn{10000}{^Y+W}}
\benchmark:n { \edef\foo{\expandafter\reverse\expandafter{\foo}} }
\edef\foo{\prg_replicate:nn{100000}{^Y+W}}
\benchmark:n { \edef\foo{\expandafter\reverse\expandafter{\foo}} }