21

This is a follow-up question to How does one append material to a token list?

The macros \prependto and \appendto work nicely to add content to token lists. How would one go about creating macros like \gobblefirst and \gobblelast that removes the first/last token from the list?

Assume the following cases:

  1. The tokens are separated by some fixed token (like \relax); or
  2. The tokens are not separated by any fixed token.

I assume a solution to 1 would be (far) easier than a solution to 2.

Community
  • 1
Werner
  • 603,163
  • Do you want to remove the very first/last token from a toks, or the first/last occurrence of a group of tokens from a toks? – Joseph Wright Apr 20 '12 at 07:35

4 Answers4

13

When you want to maintain token lists where something can be added or removed at either end, it's best to add a separator:

\catcode`@=11 % \makeatletter
\def\prependtolist#1#2{% #1=list name, #2=item to add
  \toks@={\listitem{#2}}%
  \toks@=\expandafter{\the\expandafter\toks@#1}%
  \edef#1{\the\toks@}
}
\def\appendtolist#1#2{%
  \expandafter\def\expandafter#1\expandafter{#1\listitem{#2}}%
}
\def\removetop#1#2{% #1=list,#2=macro to store the removed item
  \expandafter\removetopaux#1\removetop{#1}{#2}}
\def\removetopaux\listitem#1#2\removetop#3#4{%
  \def#2{#1}%
  \def#3{#2}%
}

\catcode`@=12

(there are better ways, this is just to give the flavor).

When the list has to be used, just set

\def\listitem#1{#1}

With LaTeX3 there are a bunch of facilities for easing the job. The "sequence" data type is the easiest:

\usepackage{expl3}
\seq_new:N \l_werner_my_seq
\seq_put_left:Nn \l_werner_my_seq {a} % analog of \prependtolist
\seq_put_right:Nn \l_werner_my_seq {a} % analog of \appendtolist
\seq_pop_left:NN \l_werner_my_seq \l_tmpa_tl % analog of \removetop
\seq_pop_right:NN \l_werner_my_seq \l_tmpa_tl % analog of \removebottom 

% How to deliver the contents of a sequence
\seq_map_function:NN \l_werner_my_seq \use:n

The last line shows how to deliver the contents of the list (the implicit separator is redefined to do nothing).

Working with token lists is possible too, but one can remove items rather than tokens, where an item is either a token which is not an explicit brace or a braced group.

\tl_new:N \l_werner_my_tl
\tl_put_left:Nn \l_werner_my_tl {a} % analog of \prependto
\tl_put_right:Nn \l_werner_my_tl {a} % analog of \appendto
\tl_set:Nx \l_werner_my_tl { \tl_tail:N \l_werner_my_tl } % your \gobblefirst

The \gobblelast macro can be obtained by reversing twice the token list:

\cs_generate_variant:Nn \tl_reverse:n {f}
\tl_set:Nx \l_werner_my_tl { \tl_reverse:f { \tl_tail:f { \tl_reverse:V \l_werner_my_tl } } }

Implementation of the "token list" approach:

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn
\NewDocumentCommand{\uselist}{ m }{ \tl_use:c { l_werner_#1_tl } }
\NewDocumentCommand{\createlist}{ m }{ \tl_new:c { l_werner_#1_tl } }
\NewDocumentCommand{\prependtolist}{ m m }
 {
  \werner_prepend:nn { #1 } { #2 }
 }
\NewDocumentCommand{\appendtolist}{ m m }
 {
  \werner_append:nn { #1 } { #2 }
 }
\NewDocumentCommand{\gobblefirst}{ m }
 {
  \werner_gobble_first:n { #1 }
 }
\NewDocumentCommand{\gobblelast}{ m }
 {
  \werner_gobble_last:n { #1 }
 }

\cs_new_protected:Npn \werner_prepend:nn #1 #2
 {
  \tl_put_left:cn { l_werner_#1_tl } { #2 }
 }
\cs_new_protected:Npn \werner_append:nn #1 #2
 {
  \tl_put_right:cn { l_werner_#1_tl } { #2 }
 }
\cs_new_protected:Npn \werner_gobble_first:n #1
 {
  \tl_set:cx { l_werner_#1_tl } { \tl_tail:v { l_werner_#1_tl } }
 }
\cs_new_protected:Npn \werner_gobble_last:n #1
 {
  \tl_set:cx { l_werner_#1_tl } { \tl_reverse:v { l_werner_#1_tl } }
  \werner_gobble_first:n { #1 }
  \tl_set:cx { l_werner_#1_tl } { \tl_reverse:v { l_werner_#1_tl } }
 }
\cs_generate_variant:Nn \tl_reverse:n {v}
\NewDocumentCommand{\showlist}{m}{ \tl_show:c { l_werner_#1_tl } }

\ExplSyntaxOff

\createlist{my}
\appendtolist{my}{z}
\prependtolist{my}{{ab}c}
\appendtolist{my}{\texttt}

\showlist{my}    % {ab}cz\texttt
\gobblefirst{my}
\showlist{my}    % cz\texttt
\gobblelast{my}
\showlist{my}    % cz
\gobblelast{my}
\showlist{my}    % c

Instead of \showlist you can say \uselist{my} to deliver the list's content.

Implementation of the "sequence" approach

\documentclass{article}
\usepackage{xparse}
\ExplSyntaxOn

\NewDocumentCommand{\createlist}{ m }{ \seq_new:c { l_werner_#1_seq } }
\NewDocumentCommand{\uselist}{ m } { \seq_map_function:cN { l_werner_#1_seq } \use:n }
\NewDocumentCommand{\showlist}{ m } { \seq_show:c { l_werner_#1_seq } }
\NewDocumentCommand{\prependtolist}{ m m }{ \werner_prepend:nn { #1 } { #2 } }
\NewDocumentCommand{\appendtolist}{ m m }{ \werner_append:nn { #1 } { #2 } }
\NewDocumentCommand{\gobblefirst}{ m }{ \werner_gobble_first:n { #1 } }
\NewDocumentCommand{\gobblelast}{ m }{ \werner_gobble_last:n { #1 } }

\cs_new_protected:Npn \werner_prepend:nn #1 #2
 {
  \seq_put_left:cn { l_werner_#1_seq } { #2 }
 }
\cs_new_protected:Npn \werner_append:nn #1 #2
 {
  \seq_put_right:cn { l_werner_#1_seq } { #2 }
 }
\cs_new_protected:Npn \werner_gobble_first:n #1
 {
  \seq_pop_left:cN { l_werner_#1_seq } \l_tmpa_tl
 }
\cs_new_protected:Npn \werner_gobble_last:n #1
 {
  \seq_pop_right:cN { l_werner_#1_seq } \l_tmpa_tl
 }

\ExplSyntaxOff

The same input as before will do, with the difference that \gobblefirst will remove {ab}c and not {ab}.

egreg
  • 1,121,712
13

I think you need to give some conditions on what is allowed in your token list.

Token registers can contain arbitrary balanced list such as {ab}{cd} that is 8 tokens but presumably when you say "remove the first token" you mean get {cd} as if you just remove the { you get a list ab}{cd} which can't be stored in a token register. (Although if you really wanted that you could convert it to ab\egroup{cd} which is similar in some respects, and can be stored). however I'll assume you want to consider brace groups as tokens. Other interesting tokens are # and unmatched \if etc.

So removing from the start can be done with essentially no restrictions:

\newtoks\aa

\def\gobble#1{}

\def\gobblefirst{%
  \aa\expandafter\expandafter\expandafter{\expandafter\gobble\the\aa}}


\aa{a\foo#{this and that}\hhh \if \egroup  \bgroup}\showthe\aa

\gobblefirst\showthe\aa
\gobblefirst\showthe\aa
\gobblefirst\showthe\aa
\gobblefirst\showthe\aa
\gobblefirst\showthe\aa
\gobblefirst\showthe\aa
\gobblefirst\showthe\aa
\gobblefirst\showthe\aa


\bye

Produces (if run through plain TeX)

> a\foo ##{this and that}\hhh \if \egroup \bgroup .
l.10 ...that}\hhh \if \egroup  \bgroup}\showthe\aa

? 
> \foo ##{this and that}\hhh \if \egroup \bgroup .
l.12 \gobblefirst\showthe\aa

? 
> ##{this and that}\hhh \if \egroup \bgroup .
l.13 \gobblefirst\showthe\aa

? 
> {this and that}\hhh \if \egroup \bgroup .
l.14 \gobblefirst\showthe\aa

? 
> \hhh \if \egroup \bgroup .
l.15 \gobblefirst\showthe\aa

? 
> \if \egroup \bgroup .
l.16 \gobblefirst\showthe\aa

? 
> \egroup \bgroup .
l.17 \gobblefirst\showthe\aa

? 
> \bgroup .
l.18 \gobblefirst\showthe\aa

? 
> .
l.19 \gobblefirst\showthe\aa

I can't see any way of removing from the other end in the same generality, you always have to impose some restrictions a reserved "marker" token that may not appear in the list, or no brace groups or....

gobble last

This version has some restrictions, the main one being that the token \xlast may not be in the list, the other ones I leave as a trap for the unsuspecting user....

\def\xlast#1{%
\ifx\xlast#1%
 \else
 \bb\expandafter{\the\expandafter\bb\the\cc}%
 \cc{#1}%
 \expandafter\ifx\expandafter\xlast\gobble#1\xlast
 \else
 \expandafter\cc\expandafter{\expandafter{\the\cc}}%
 \fi
 \expandafter\xlast
\fi}

\newtoks\bb
\newtoks\cc
\def\gobblelasta{%
\bb{}%
\cc{}%
\expandafter\xlast\the\aa\xlast
\aa\bb}

\aa{a\foo#{this and that}\hhh  \fi \egroup  \bgroup}\showthe\aa
\gobblelasta\showthe\aa
\gobblelasta\showthe\aa
\gobblelasta\showthe\aa
\gobblelasta\showthe\aa
\gobblelasta\showthe\aa
\gobblelasta\showthe\aa
\gobblelasta\showthe\aa
\gobblelasta\showthe\aa

\bye

Produces the following from plain TeX:

> a\foo ##{this and that}\hhh \fi \egroup \bgroup .
l.46 ...hat}\hhh  \fi \egroup  \bgroup}\showthe\aa

? 
> a\foo ##{this and that}\hhh \fi \egroup .
l.47 \gobblelasta\showthe\aa

? 
> a\foo ##{this and that}\hhh \fi .
l.48 \gobblelasta\showthe\aa

? 
> a\foo ##{this and that}\hhh .
l.49 \gobblelasta\showthe\aa

? 
> a\foo ##{this and that}.
l.50 \gobblelasta\showthe\aa

? 
> a\foo ##.
l.51 \gobblelasta\showthe\aa

? 
> a\foo .
l.52 \gobblelasta\showthe\aa

? 
> a.
l.53 \gobblelasta\showthe\aa

? 
> .
l.54 \gobblelasta\showthe\aa
David Carlisle
  • 757,742
  • Ha! I just did a test on removing 100 tokens from the end of a 250 token list 1000 times and my version ran 50 times faster than yours :-) My version doesn't work for token lists longer than about 250 though because of grouping levels :-( – Stephan Lehmke Apr 21 '12 at 04:55
  • I just saw that your version seems to swallow spaces. This was masked by spaces being preserved inside the braces. If I remove the braces, spaces vanish. – Stephan Lehmke Apr 21 '12 at 06:11
  • 1
    @DavidCarlisle: Here is an unsuspecting user :). Please do we need the first \expandafter in \expandafter\cc\expandafter{\expandafter{\the\cc}}? Also, one outer brace is lost in {{this and that}}. Then any space between the closing and opening braces in {#} {{this and that}} is lost. – Ahmed Musa Apr 21 '12 at 10:18
8

Here are three versions of \gobblelast:

\makeatletter

\def\gobblelast@nil{gobblelast@nil}

\newtoks\gobblelast@toks

\let\gobblelast@obr={
\let\gobblelast@cbr=}
\def\gobblelast@mkspc#1%
{%
  \expandafter\futurelet\expandafter\gobblelast@spc\expandafter\@gobble
}
\futurelet\next\gobblelast@mkspc a b

\newcommand\gobblelast[2]
{%
  \global\@tempcnta#2\relax
  \global\advance\@tempcnta\@ne
  \global\gobblelast@toks={}%
  \begingroup
  \let\@gobblelast@iter\@@gobblelast@iter
  \expandafter\futurelet\expandafter\next\expandafter\gobblelast@iter
  \the#1\relax\gobblelast@nil
  #1=\expandafter{\the\gobblelast@toks}%
}

\newcommand\gobblelast@iter
{%
  \gobblelast@ifx{\next\gobblelast@nil}
  {\endgroup\@gobble}
  {%
    \gobblelast@ifx{\next\gobblelast@obr}
    {\gobblelast@readobr}
    {%
      \gobblelast@ifx{\next\gobblelast@cbr}
      {\gobblelast@readcbr}
      {%
        \gobblelast@ifx{\next\gobblelast@spc}
        {\gobblelast@readspc}
        {\gobblelast@readtoken}
      }%
    }%
  }%
}

\newcommand\gobblelast@ifx[1]
{%
  \ifx#1\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
}

\newcommand\gobblelast@readobr
{%
  \afterassignment\@gobblelast@readobr\let\next=%
}%

\newcommand\@gobblelast@readobr
{%
  \toks0\expandafter\expandafter\expandafter{\expandafter\string\expandafter{\iffalse}\fi}%
  \@gobblelast@iter
}

\newcommand\gobblelast@readcbr
{%
  \afterassignment\@gobblelast@readcbr\let\next=%
}%

\newcommand\@gobblelast@readcbr
{%
  \toks0\expandafter\expandafter\expandafter{\expandafter\iffalse\expandafter{\expandafter\fi\string}}%
  \@gobblelast@iter
}

\def\gobblelast@readspc%
{\afterassignment\@gobblelast@readspc\@tempcntb=`}

\def\@gobblelast@readspc%
{%
  \toks0{ }%
  \@gobblelast@iter
}

\def\gobblelast@readtoken#1%
{%
  \toks0{#1}%
  \@gobblelast@iter
}

\newcommand\@@gobblelast@iter
{%
  \begingroup
    \aftergroup\gobblelast@finish
    \futurelet\next\gobblelast@iter    
}

\newcommand\gobblelast@finish
{%
  \ifnum\@tempcnta>\z@
    \global\advance\@tempcnta\m@ne
   \else
    \global\gobblelast@toks=\expandafter{\the\toks\expandafter\z@\the\gobblelast@toks}%
  \fi
  \endgroup
}

\newcommand\weakgobblelast[2]
{%
  \global\@tempcnta#2\relax
  \global\advance\@tempcnta\@ne
  \global\gobblelast@toks={}%
  \begingroup
  \let\@gobblelast@iter\@weak@gobblelast@iter
  \expandafter\futurelet\expandafter\next\expandafter\weakgobblelast@iter
  \the#1\relax\gobblelast@nil
  #1=\expandafter{\the\gobblelast@toks}%
}

\newcommand\weakgobblelast@iter
{%
  \gobblelast@ifx{\next\gobblelast@nil}
  {\endgroup\@gobble}
  {%
    \gobblelast@ifx{\next\gobblelast@obr}
    {\gobblelast@readbr}
    {%
      \gobblelast@ifx{\next\gobblelast@spc}
      {\gobblelast@readspc}
      {\gobblelast@readtoken}
    }%
  }%
}

\newcommand\@weak@gobblelast@iter
{%
  \begingroup
    \aftergroup\gobblelast@finish
    \futurelet\next\weakgobblelast@iter    
}

\newcommand\gobblelast@readbr[1]
{%
  \toks0{{#1}}%
  \@gobblelast@iter
}%

\newcommand\voodoogobblelast[2]
{%
  \global\@tempcnta#2\relax
  \global\advance\@tempcnta\@ne
  \global\gobblelast@toks={}%
  \global\gobblelast@temptoks{}
  \begingroup
  \let\@gobblelast@iter\@voodoo@gobblelast@iter
  \expandafter\futurelet\expandafter\next\expandafter\voodoogobblelast@iter
  \the#1\relax\gobblelast@nil
  \edef\tmp{\expandafter\unexpanded\expandafter{\expandafter\unexpanded\expandafter{\the\gobblelast@temptoks}}\the\gobblelast@toks}
  #1=\expandafter{\tmp}%
}

\newcommand\voodoogobblelast@iter
{%
  \gobblelast@ifx{\next\gobblelast@nil}
  {\endgroup\@gobble}
  {%
    \gobblelast@ifx{\next\gobblelast@obr}
    {\vgobblelast@readobr}
    {%
      \gobblelast@ifx{\next\gobblelast@cbr}
      {\vgobblelast@readcbr}
      {%
        \gobblelast@ifx{\next\gobblelast@spc}
        {\gobblelast@readspc}
        {\gobblelast@readtoken}
      }%
    }%
  }%
}

\newcommand\@voodoo@gobblelast@iter
{%
  \begingroup
    \aftergroup\voodoogobblelast@finish
    \futurelet\next\voodoogobblelast@iter    
}


\newcommand\vgobblelast@readobr
{%
  \afterassignment\v@gobblelast@readobr\let\next=%
}%

\newcommand\gobblelast@vobr{{\iffalse}\fi}

\newcommand\v@gobblelast@readobr
{%
  \toks0{\gobblelast@vobr}%
  \@gobblelast@iter
}

\newcommand\vgobblelast@readcbr
{%
  \afterassignment\v@gobblelast@readcbr\let\next=%
}%

\newcommand\gobblelast@vcbr{\iffalse{\fi}}

\newcommand\v@gobblelast@readcbr
{%
  \toks0{\gobblelast@vcbr}%
  \@gobblelast@iter
}

\newtoks\gobblelast@temptoks


\newcommand\voodoogobblelast@finish
{%
  \ifnum\@tempcnta>\z@
    \global\advance\@tempcnta\m@ne
   \else
    \expandafter\ifx\expandafter\gobblelast@vobr\the\toks\z@
      \edef\tmp{\noexpand\gobblelast@vobr\expandafter\unexpanded\expandafter{\expandafter\unexpanded\expandafter{\the\gobblelast@temptoks}}\the\gobblelast@toks}
      \global\gobblelast@toks=\expandafter{\tmp}%
      \global\gobblelast@temptoks{}
     \else
      \expandafter\ifx\expandafter\gobblelast@vcbr\the\toks\z@
        \edef\tmp{\noexpand\gobblelast@vcbr\expandafter\unexpanded\expandafter{\expandafter\unexpanded\expandafter{\the\gobblelast@temptoks}}\the\gobblelast@toks}
        \global\gobblelast@toks=\expandafter{\tmp}%
        \global\gobblelast@temptoks{}
       \else
        \global\gobblelast@temptoks=\expandafter{\the\toks\expandafter\z@\the\gobblelast@temptoks}%
      \fi
    \fi
  \fi
  \endgroup
}

The first version, \gobblelast, will remove tokens one by one, but braces have to be treated specially. I haven't found a way to put them into a token list such that it becomes unbalanced, so they are converted to catcode 12.

Testing with 

\newtoks\mytoks

\mytoks={a\foo#{this and that} \hhh  \fi \egroup  \bgroup}

\gobblelast{\mytoks}{10}

\showthe\mytoks

will give

> a\foo ##{this and .
l.274     \showthe\mytoks

but the braces are not real braces any more.

If you are willing to accept the following recusive definition of weak token:

A weak token is

  1. Any single token which is not a brace or
  2. any list of weak tokens enclosed in braces.

then \weakgobblelast will preserve braces, but everything enclosed in braces will be counted as one token. Hence,

\newtoks\mytoks

\mytoks={a\foo#{this and that} \hhh  \fi \egroup  \bgroup}

\weakgobblelast{\mytoks}{6}

\showthe\mytoks

will give

> a\foo ##.
l.274     \showthe\mytoks

If you really really want do do voodoo magic with 'real' unbalanced brace groups, then \voodoogobblelast is for you. It will create a "safety package" of tokens which upon full expansion recreate the truncated token list including possibly unbalanced braces...

Handle with care! ;-)

The following 

\newtoks\mytoks

\mytoks={a\foo#{this and that} \hhh  \fi \egroup  \bgroup}

\voodoogobblelast{\mytoks}{10}

\showthe\mytoks

\mytoks{\newcommand\foo{\newcommand\bar[1]{\baz\space\quux}}}

\voodoogobblelast{\mytoks}{4}

\showthe\mytoks

\expandafter\edef\expandafter\next\expandafter{\the\mytoks}}}

\show\next

\next

\show\foo

will give

> \unexpanded {a\foo ##}\gobblelast@vobr \unexpanded {this and }.
l.274     \showthe\mytoks

> \unexpanded {\newcommand \foo }\gobblelast@vobr \unexpanded {\newcommand \bar [1]}\gobblelast@vobr \unexpanded {\baz }.
l.280     \showthe\mytoks

> \next=macro:
->\newcommand \foo {\newcommand \bar [1]{\baz }}.
l.284     \show\next

> \foo=\long macro:
->\newcommand \bar [1]{\baz }.
l.288     \show\foo

I have the subtle feeling I committed a couple of deadly sins making this, but anyway...

Stephan Lehmke
  • 24,361
  • 3
    I suppose it's ungentlemanly to point out that \string is cheating:-) – David Carlisle Apr 20 '12 at 12:49
  • I'm really bad in this kind of low-level hackery :-( It's about the 137th thing I tried and the first that did't lead to total desaster. If anyone can point me to a source to look for a better solution, I'd be grateful. – Stephan Lehmke Apr 20 '12 at 13:02
  • 1
    (it also fails if the last token is an undefined command) As noted in my answer (which I just updated to do gobblelast) I think you have to interpret "token" as including brace groups as token registers can not store unbalanced lists. (so same rules as macro parameters or latex \@tfor) – David Carlisle Apr 20 '12 at 13:44
  • @DavidCarlisle You're right, how embarassing ;-) Copy-paste error. I removed some superfluous \expandafters. – Stephan Lehmke Apr 20 '12 at 16:06
  • @DavidCarlisle Well last time I looked, a brace was a token ;-) So I'm giving a true alternative in the sense that single braces can be removed, but the rest are not 'real' braces any more until further notice :-) Maybe one could put them back with some \lccode trick? – Stephan Lehmke Apr 20 '12 at 17:32
8

I wanted spaces and braces preserved but I find Stephan Lehmke's solution too long.

\documentclass{article}
\makeatletter
\newtoks\giventoks
\newtoks\@temptokenb
\def\removelasttoken{%
  \let\origbgroup\bgroup
  \let\bgroup\@undefined
  \@temptokena{}\@temptokenb{}%
  \expandafter\removelasttoken@a\the\giventoks\removelasttoken
  \giventoks=\@temptokenb
  \let\bgroup\origbgroup
}
\def\removelasttoken@a{\futurelet\next\removelasttoken@b}
\def\removelasttoken@b{%
  \ifx\next\@sptoken
    \expandafter\removelasttoken@c
  \else
    \expandafter\removelasttoken@e
  \fi
}
\@namedef{removelasttoken@c} {\futurelet\next\removelasttoken@d}
\def\removelasttoken@d{%
  \ifx\next\removelasttoken
    \@temptokenb\expandafter{\the\expandafter\@temptokenb\the\@temptokena}%
    \expandafter\removelasttoken@e
  \else
    \@temptokenb\expandafter
      {\the\expandafter\@temptokenb\the\expandafter\@temptokena\space}%
    \@temptokena{}%
    \expandafter\removelasttoken@a
  \fi
}
\def\removelasttoken@e#1{%
  \ifx\removelasttoken#1\else
    \@temptokenb\expandafter{\the\expandafter\@temptokenb\the\@temptokena}%
    \@temptokena{#1}%
    \edef\reserved@a{%
      \@temptokena{\ifx\next\origbgroup
        {\the\@temptokena}\else\the\@temptokena\fi}%
    }%
    \reserved@a
    \expandafter\removelasttoken@a
  \fi
}
\makeatother
% Example:
\begin{document}
\def\showout{\typeout{\the\giventoks}}
\giventoks{a\foo{#} {{this and that}} \hhh \fi \ifx \egroup  \bgroup \undefined}
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\removelasttoken\showout
\end{document}

The example produces

a\foo {##} {{this and that}} \hhh \fi \ifx \egroup \bgroup 
a\foo {##} {{this and that}} \hhh \fi \ifx \egroup 
a\foo {##} {{this and that}} \hhh \fi \ifx 
a\foo {##} {{this and that}} \hhh \fi 
a\foo {##} {{this and that}} \hhh 
a\foo {##} {{this and that}}<space>
a\foo {##} {{this and that}}
a\foo {##}<space>
a\foo {##}
a\foo

With eTeX, we can also do

% \gobblelasttoken<cmd>
\def\gobblelasttoken#1{%
  \let\origbgroup\bgroup
  \let\bgroup\@undefined
  \@temptokena{}\@temptokenb{}%
  \expandafter\removelasttoken@a#1\gobblelasttoken
  \let\bgroup\origbgroup
  \edef#1{\the\@temptokenb}%
}

\removelasttoken@a and the other macros remain the same.

Example:

\begin{document}
\def\showout{\typeout{\unexpanded\expandafter{\giventoks}}}
\edef\giventoks{\unexpanded{a\foo{#} {{this and that}} \hhh \fi \ifx
  \egroup\bgroup \undefined}}
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\gobblelasttoken\giventoks\showout
\end{document}
Ahmed Musa
  • 11,742
  • That's certainly more elegant (but also less efficient for removing several tokens ;-) – Stephan Lehmke Apr 21 '12 at 12:47
  • At least you know how to properly deal with spaces, my blunders are really embarassing... Not to know there is \@sptoken already defined by LaTeX... :-( – Stephan Lehmke Apr 21 '12 at 12:53
  • @StephanLehmke: Please what do you mean by 'less efficient'? My only reason for previously grouping the entire loop was because of \toks0. So I removed it to avoid a local group. – Ahmed Musa Apr 21 '12 at 13:07
  • No offense meant. I just observed you are removing one token at a time, so for removing n tokens, the macro would have to be called n times. Looking back at the question, removing only one token was explicitly requested. I just inferred from "material" in the title that removing several tokens would be required. – Stephan Lehmke Apr 21 '12 at 13:14