Test whether token register is empty

Question

Is it possible to test whether a token register is empty without expanding it?

Answer 1

At time of writing the other TeX based answers on this page are flawed in that they hide a conditional \ifx inside a macro but still use \else/\fi at the "top" level. This will mean that things break unexpectedly when used inside other conditionals.

The LaTeX3 programming language expl3 contains a module for doing stuff with token registers:

\usepackage{expl3}
...
\ExplSyntaxOn
\toks_if_empty:NTF \mytoks {true} {false}
\ExplSyntaxOff

It essentially does internally what the other answers here are suggesting, but it uses expansion to grab its arguments so the branching is robust (and you don't have \fi lying around to get in your way).

Update: So what does this approach do that is superior to other methods? Consider the style of solution first offered in answer to this question:

\def\IfEmpty#1{%
  \edef\1{\the#1}
  \ifx\1\empty
}
...
\IfEmpty\foo T\else F\fi

This doesn't behave nicely when nested, because TeX scans ahead when discarding unfollowed branches of a conditional. Consider

\ifx\bar\baz
  \IfEmpty\foo T\else F\fi % <- uh oh
\else
  E
\fi

If \bar = \baz, then the second branch is discarded and the first branch is executed. So far so good. If \bar ≠ \baz, then the first branch is discarded by reading ahead until the first unmatched \else — and this is the one in the line labelled "uh oh" above. So you could collapse the expansion of the above snippet in this case to:

\iffalse\else F\fi % <- uh oh
\else
  E
\fi

and hence the cause of the ‘Extra \else’ error message in this case.

So this form for conditionals doesn't work so well. Next try. You can also write this style of code like this:

\def\IfEmpty#1#2#3{%
  \edef\1{\the#1}
  \ifx\1\empty
    #2%
  \else
    #3%
  \fi
}

This avoids the problems of nesting as in the previous trial solution, but it's prone to another problem: #2 and #3 have trailing material behind them, namely \else and \fi. This is a problem if you want to write something like

\def\processfoo#1{...something with #1...}
\IfEmpty\foo{\error}{\processfoo} {arg}

because the #1 passed to \processfoo will be \fi instead of the desired {arg}. The conditional in this case is better written as

\def\IfEmpty#1#2#3{%
  \edef\1{\the#1}
  \ifx\1\empty
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
  {#2}{#3}
}

so overcome this problem. This is how expl3 conditionals work, and it's why we're writing TF at the end of all their names to indicate "true" and "false" branches. (Or just T or just F if you only want one of them.)

Incidentally, there are expandable tests for checking for emptiness, which is why I suggest using the expl3 approach for this test. Expandability is not always required, of course, but code that is fully expandable tends to be more reliable and it's always nice to have for cases such as

\typeout{ \toks_if_empty:NT \foo {Warning:~\string\foo\space is~ empty} }

Answer 2

With LuaTeX you can do this:

\directlua{
if string.len(tex.toks["headline"]) > 0 then
    print("not emtpy")
end
}

But since I never use token list: don't use this code to build nuclear plants.

Answer 3

Ulrich Diez regularly posts on comp.text.tex some code along the following lines (slightly modified by me for toks):

\newcommand\@ifempty@toks[1]{%
  \ifcat\relax\detokenize\expandafter{\the#1}\relax
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
}

Or, without using e-TeX,

\newcommand\@ifempty@toks[1]{%
  \ifcase\iffalse{{{\fi\expandafter\@ifempty@@toks\the#1}1}1}0
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi}
\newcommand{\@ifempty@@toks}
  {\expandafter\@gobble\expandafter{\expandafter{%
        \ifcase`}\expandafter}\expandafter\fi\string}

EDIT: Ulrich Diez rightfully points out that redefining \relax can make the first test fail. It is slightly safer to use a character token like $ or X that has category code among 3,4,7,8,11.

Additionally, neither method works if the \toks contains outer macros, as in \outer\def\foo{} \newtoks\mytoks \mytoks=\expandafter{\noexpand\foo}.

Answer 4

Would something like this work?

\def\ifempty#1{%
\edef\ContentsOfList{\the#1}%
\edef\CompareTo{}%
\ifx\ContentsOfList\CompareTo}

You could then use it like this:

\toks0={\blah\bloh a b c}
\toks2={}

\ifempty{\toks0} empty\else nonempty\fi

\ifempty{\toks2} empty\else nonempty\fi

Answer 5

Similar to Jan Hlavacek's answer, but expandable:

\def\certainlynotintoks{\certainlynotintoks}
\def\iftoksempty#1{\expandafter\ifx\expandafter
    \certainlynotintoks\the#1 \certainlynotintoks}

\toks0{\undefined will not be expanded}
\toks2{}
\iftoksempty{\toks0} empty\else not empty\fi \par
\iftoksempty{\toks2} empty\else not empty\fi

This requires that \certainlynotintoks does not appear as the first token in the token register. Both this and Jan's answer do expand the token register, but only once. I don't believe there's a way to avoid that.

Edit: Taking Will's (absolutely correct) comment into consideration, change the definition of \iftoksempty to

\makeatletter
\def\iftoksempty#1{\expandafter\ifx\expandafter
    \certainlynotintoks\the#1 \certainlynotintoks \expandafter\@firstoftwo
    \else \expandafter\@secondoftwo\fi}

Then you can use \iftoksempty{\toks0}{empty}{not empty}.

Answer 6

A rather retro way of avoiding the problems Will talks about is to use flags. Define:

\newif\iftest 
\def\testtoksempty#1{\edef\1{\the#1}
  \ifx \1\empty \testtrue % from plain.tex: \def\empty{}
  \else\testfalse \fi}

Then use:

\testtoksempty\toksreg
\iftest toksreg is empty \else toksreg's got something \fi

I tend to avoid any kind of overlap between \if conditional and macro expansion because of my incomplete, fearful grasp of the issues Will nicely explained (and which I now understand a bit better).
This is clunky, ASM-like way to code, and the use of a fixed test flag means that Boolean expressions are going to be fiddly: you may be driven to multiple, nearly identical macros - \tessttoksempty, \tesssttoksempty, etc.
But it's robust and easy to debug.

Answer 7

With all "traditional" approaches presented so far and with expl3's \tl_if_empty:VTF the content of the token-register is produced via \the and examined by means of macros.

The circumstance of a macro-argument containing \outer-tokens triggers an error-message.

Therefore these "traditional" approaches and expl3's \tl_if_empty:VTF fail if the register to test for emptiness contains tokens that at the time when being produced by \the are defined in terms of \outer.

E.g., with

\newtoks\mytoks
\mytoks{\foo}%
\outer\def\foo{foo is outer}%

\the\mytoks will produce a token which is defined in terms of \outer and therefore cannot be a component of a macro-argument.

(However, interface3.pdf, the reference documentation for the expl3 programming environment, explicitly states in section "1.4 TeX concepts not supported by LaTeX3":

The TeX concept of an "\outer" macro is not supported at all by LaTeX3. As such, the functions provided here may break when used on top of LaTeX 2ε if \outer tokens are used in the arguments.

)

If working on a LuaLaTeX-engine, according to LuaTeX Reference Manual stable December 2021 Version 1.15, you can do something like this, where \outer-tokens are not a problem:

\errorcontextlines=10000
\makeatletter
%%=============================================================================
\@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=`\^^00}%
\@ifdefinable\gobbletoks{%
  \begingroup
  \escapechar=-1
  \expandafter\endgroup\expandafter\def\expandafter\gobbletoks\expandafter#\expandafter1\string\toks{}%
}%
\newcommand\toksnum[1]{\expandafter\gobbletoks\meaning#1}%
% Don't use tex.sprint or the like for sending control-sequence-tokens to TeX-level
% as tex.sprint("\macro") only works out as long as backslash has catcode 0.
\newcommand\CheckWhetherTokenRegisterEmpty[1]{%
  \romannumeral
  \directlua{%
    if string.len(tex.toks[\number\toksnum#1]) > 0 then
      token.put_next ( token.create("@secondoftwo") ) else token.put_next ( token.create("@firstoftwo") )
    end
    token.put_next ( token.create("expandafter"), token.create("UD@stopromannumeral") )
  }%
}%
%%=============================================================================
\makeatother
\newtoks\testtoks
\testtoks={}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%
\testtoks={ }
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%
\testtoks={something}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%
% This is not a problem:
\testtoks={\foobar}
\outer\def\foobar{foobar is outer}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%
\catcode\/=0 \catcode\=12
/message{%
  ^^J%
  /string/testtoks
  /CheckWhetherTokenRegisterEmpty/testtoks{ is}{ is not} empty%
}%
/catcode/\=0 \catcode//=12
\stop

If Lua-extensions are not available and if you can rely on the content of the token-register not containing \outer-tokens you can probably do something like this:

\errorcontextlines=10000
\makeatletter
%%=============================================================================
%% PARAPHERNALIA:
%% \UD@firstoftwo, \UD@secondoftwo,\UD@stopromannumeral, \UD@CheckWhetherNull,
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=`\^^00}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/g/comp.text.tex/c/kuOEIQIrElc/m/lUg37FmhA74J>
\newcommand\UD@CheckWhetherNull[1]{%
  \romannumeral\expandafter\UD@secondoftwo\string{\expandafter
  \UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@secondoftwo}{%
  \expandafter\UD@stopromannumeral\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% With eTeX do something like:
%% 
%% \newcommand\UD@CheckWhetherNull[1]{%
%%   \romannumeral\ifcat$\detokenize{#1}$%
%%     \expandafter\expandafter\expandafter\UD@stopromannumeral
%%     \expandafter\UD@firstoftwo
%%   \else
%%     \expandafter\expandafter\expandafter\UD@stopromannumeral
%%     \expandafter\UD@secondoftwo
%%   \fi
%% }%
%%-----------------------------------------------------------------------------
%% Check whether token-register is empty:
%%.............................................................................
\newcommand\CheckWhetherTokenRegisterEmpty[1]{%
  \romannumeral
  \expandafter\expandafter\expandafter\UD@secondoftwo
  \expandafter\UD@CheckWhetherNull
  \expandafter{\the#1}%
}%
%%=============================================================================
\makeatother
\newtoks\testtoks
\testtoks={}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%
\testtoks={ }
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%
\testtoks={something}
\message{%
  ^^J%
  \string\testtoks
  \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
}%
% This is a problem:
%
% \testtoks={\foobar}
% \outer\def\foobar{foobar is outer}
% \message{%
%   ^^J%
%   \string\testtoks
%   \CheckWhetherTokenRegisterEmpty\testtoks{ is}{ is not} empty%
% }%
\stop

---

Another interesting question might be the handling of the case of the argument of the routine for checking emptiness of a token-register not denoting a token-register at all.

But the question of how to implement a test for checking if an arbitrary given argument denotes a token-register is a different question and thus is out of the scope of this question.

Still, a few thoughts on this that get one used to the idea that TeX may not allow you to implement every testing-routine one hundred percent accurately and reliably:

In case of the argument denoting a token-register the argument might consist of

• a single \toksdef-token,
• \toks⟨s.th. that expands to a TeX-⟨number⟩-quantity with a value in range of available token-registers⟩.

So a check would imply handling the case of the first token of the argument being the \toks-primitive in which case you would need to check if (recursive) expansion of following tokens yields only a valid TeX-⟨number⟩-quantity. As following tokens might form an expansion-based algorithm, the check would imply the task of checking if an arbitrary expansion-based algorithm terminates at all, and if so, if termination goes along with only delivering a valid TeX-⟨number⟩-quantity. Checking if an arbitrary expansion-based algorithm terminates at all reminds me of the halting-problem.