Different command behavior if argument is not a single token

Question

In defining a new LaTeX command, I'm looking for a way to distinguish whether its argument is a single token or not. In other words, I would like \foo x to be equivalent to \something{x}, and \foo{xy} to \somethingelse{xy} (as opposed to \something{xy}). Would that be possible somehow, possibly without using evil hacks?

I tried looking at the documentation for xparse and etoolbox, but I could not find anything useful. Any hints?

Answer 1

EDITED to make it expandable! (see SUPPLEMENT)

\documentclass{article}
\newcommand\foo[1]{\fooaux#1\endfoo}
\def\fooaux#1#2\endfoo{\if\relax\detokenize{#2}\relax
  \expandafter\fooproper\else\expandafter\fooalt\fi{#1#2}}
\newcommand\fooproper[1]{FOO! #1}
\newcommand\fooalt[1]{bar #1}
\begin{document}
\foo a
\foo{A}
\edef\z{\foo{ab}}% EXPANDABLE
\z
\end{document}

---

SUPPLEMENT

Ulrich rightly points out that the above simple solution has several potential chinks, depending on what the OP expects to get thrown at the \foo macro. In particular, if the argument is a delimited space, or contains a leading space as part of a delimited argument, the space will not be detected. Further, if the argument contains the macro \endfoo, likewise that will spoof the result.

At the expense of expandability, I address Ulrich's concerns about spaces by calling on the facilities of tokcycle, which has no problem discerning spaces in its input stream. Thus, I show various examples and how much can be discerned about their content on the fly.

The only restriction is that the argument stream not contain the token endtokcycraw, which is reserved by the tokcycle package.

\documentclass{article}
\usepackage{tokcycle}
\newcommand\foo{\def\leadingspace{F}\futurelet\foofirst\fooA}
\newcommand\fooA{\ifx\foofirst\bgroup\expandafter\foodelim\else
  \expandafter\fooundelim\fi}
\newcounter{tokcnt}
\newcommand\foodelim[1]{%
  \setcounter{tokcnt}{0}%
  \tokcycle
  {\stepcounter{tokcnt}}%
  {\addtocounter{tokcnt}{2}\processtoks{##1}}%
  {\stepcounter{tokcnt}}%
  {\ifnum\value{tokcnt}=0\relax\def\leadingspace{T}\fi\stepcounter{tokcnt}}%
  {#1}%
  \tctestifnum{\value{tokcnt}=0}{\foonil}{%
  \tctestifnum{\value{tokcnt}=1}{\fooone}{\foomulti}}{#1}}
\newcommand\fooundelim[1]{Undelimited argument to foo: #1}
\newcommand\foonil[1]{Delimited null argument to foo}
\newcommand\fooone[1]{Delimited single token to foo: #1%
  \if T\leadingspace (Space)\fi}
\newcommand\foomulti[1]{Delimited multi-token to foo: #1%
 \if T\leadingspace\ (Leading Space)\fi}
\begin{document}
\foo a \par
\foo{} \par
\foo{ } \par
\foo{A} \par
\foo{ B} \par
\foo{C D} \par
\foo{ab}
\end{document}

Answer 2

expl3 has a conditional just for that: \tl_if_single_token:nTF. It will return true only if the argument is exactly a single token. It returns false if the argument is empty, or if the argument is a braced group containing any number of tokens. expl3 also has \tl_if_single:nTF that returns true on x as well as on {x}, whereas \tl_if_single_token:nTF returns false on the latter.

\documentclass{article}
\ExplSyntaxOn
\cs_new_eq:NN \IfSingleTokenTF \tl_if_single_token:nTF
\cs_new_eq:NN \IfSingleItemTF \tl_if_single:nTF
\ExplSyntaxOff
\newcommand\foo[1]{%
  \texttt{>#1: }%
  \IfSingleTokenTF{#1}%
    {\something}%
    {\somethingelse}{#1}}
\newcommand\something[1]{(single:#1)}
\newcommand\somethingelse[1]{(multiple:#1)}
\begin{document}
\foo{}
\foo{ }
\foo{x}
\foo{ x}
\foo{xy}
\foo{{xy}}
\end{document}

---

As Frank noted in the comment, the code above works fine for XeTeX and LuaTeX because those engines tread UTF-8 characters like ä or as a single token. pdfTeX, on the other hand, treats ä as two and as four tokens, which when typesetting combine to produce a single glyph, thus the test above results as multiple tokens.

Depending on your use case (or if you want cross-engine compatibility) you might want to treat UTF-8 characters as a single token, in which case a bit more code is needed to check for those cases in pdfTeX:

The code first does the \tl_if_single_token:nTF test to eliminate the obvious case that the argument is indeed a single token. If that returns false, then we look at meaning of the first token in the argument, and if it is \UTFviii@(two|three|four)@octets then we know the first token is an UTF-8 char, in which case we remove the remaining tokens that compose the char and then test if what remains is empty: if it is, the only thing in the argument was the UTF-8 char, so return true, otherwise return false.

\documentclass{article}
\ExplSyntaxOn
\bool_lazy_or:nnTF
    { \sys_if_engine_luatex_p: }
    { \sys_if_engine_xetex_p: }
  {
    \cs_new_eq:NN \IfSingleTokenTF \tl_if_single_token:nTF
    \use_none:n
  }
  { \makeatletter \use:n }
  {
    \makeatother
    \scan_new:N \s__fudo
    \prg_new_conditional:Npnn \fudo_if_single_token:n #1 { T, F, TF }
      {
        \tl_if_single_token:nTF {#1}
          { \prg_return_true: }
          {
            \tl_if_head_is_N_type:nTF {#1}
              {
                \use:e
                  {
                    \exp_not:N __fudo_if_single_token_aux:w
                      \exp_not:o { \token_to_meaning:N #1 }
                      \tl_to_str:n { UTFviii@ one @octets } ~ \s__fudo
                      \exp_not:n {#1} \s__fudo
                  }
              }
              { \prg_return_false: }
          }
      }
    \use:e
      {
        \cs_new:Npn \exp_not:N __fudo_if_single_token_aux:w
          #1 \tl_to_str:n { UTFviii@ } #2 \tl_to_str:n { @octets } ~
          #3 \s__fudo #4 \s__fudo
      }
      {
        \str_case:nnTF {#2}
          {
            { one   } { \exp_after:wN __fudo_single_chk:w \use_none:n }
            { two   } { \exp_after:wN __fudo_single_chk:w \use_none:nn }
            { three } { \exp_after:wN __fudo_single_chk:w \use_none:nnn }
            { four  } { \exp_after:wN __fudo_single_chk:w \use_none:nnnn }
          }
          { #4 \s__fudo }
          { \prg_return_false: }
      }
    \cs_new:Npn __fudo_single_chk:w #1 \s__fudo
      {
        \tl_if_empty:nTF {#1}
          { \prg_return_true: }
          { \prg_return_false: }
      }
    \cs_new_eq:NN \IfSingleTokenTF \fudo_if_single_token:nTF
  }
\ExplSyntaxOff
\newcommand\foo[1]{%
  \texttt{>#1: }%
  \IfSingleTokenTF{#1}%
    {\something}%
    {\somethingelse}{#1}}
\newcommand\something[1]{(single:#1)}
\newcommand\somethingelse[1]{(multiple:#1)}
\begin{document}
\foo{}
\foo{ }
\foo{x}
\foo{ä}
% \foo{鸟}
% \foo{}
\foo{ x}
\foo{xy}
\foo{{xy}}
\end{document}

The code does not understand é (U+0065 Latin Small Letter E + U+0301 Combining Acute Accent) as a single character. You have to input the proper Unicode U+00E9 Latin Small Letter E with Acute.

Answer 3

Preliminary remarks:

1. The special case of the argument being empty/consisting of zero tokens also is one of the cases where the argument is not a single token. (In the example below the corresponding expandable test is \UD@CheckWhetherNull.)

2. TeX strips off the outermost level of matching surrounding curly braces from a macro-argument if present: If \foo is a macro processing a single undelimited argument, then \foo x and \foo{x} yield the same.

3. Explicit space-tokens (explicit character-tokens of catcode 10 and charcode 32) need special treatment when processed via macros: When gathering tokens that form an undelimited argument TeX discards preceding explicit space-tokens. Therefore an explicit space-token can only be processed as undelimited macro-argument if nested in braces. (In the example below the corresponding expandable test is \UD@CheckWhetherLeadingExplicitSpace.)

4. Curly braces ({ or }) also get tokenized as tokens, as explicit character-tokens of catcode 1(begin group) or 2(end group): With \foo{{x}} the argument of \foo consists of the three tokens {1, x11 and }2. Thus this is a case where the argument is not a single token. (In the example below the corresponding expandable test is \UD@CheckWhetherBrace.)

In the example below with the expandable macro \CheckWhetherSingleToken the carrying out of the tests mentioned above is triggered via \romannumeral-expansion. Therefore in expansion-contexts the result is obtained after two expansion-steps, e.g., after "hitting" \CheckWhetherSingleToken via \expandafter twice/after "hitting" the first token of \CheckWhetherSingleToken's toplevel-expansion via \expandafter.

(The gist of \romannumeral-expansion is:

1. TeX expands expandable tokens while gathering the first token of a TeX-⟨number⟩-quantity that is to be converted by \romannumeral.
2. In case the value of the TeX-⟨number⟩-quantity gathered is not positive TeX will silently "eat"/discard the tokens forming the TeX-⟨number⟩-quantity without delivering anything in return.

Therefore you can (ab?)use \romannumeral for tricking TeX into doing a lot of expansion- and shuffling-around-of-tokens/macro-arguments-work as long as it is ensured that in the end the resulting token-sequence begins with a TeX-⟨number⟩-quantity whose value is not positive.)

\errorcontextlines=10000
\documentclass[a4paper]{article}
\makeatletter

%%=============================================================================
%% Paraphernalia:
%%    \UD@firstoftwo, \UD@secondoftwo, \UD@Exchange, \UD@PassFirstToSecond, 
%%    \UD@removespace, \UD@stopromannumeral,  \UD@CheckWhetherNull, 
%%    \UD@CheckWhetherBrace, \UD@CheckWhetherLeadingSpace, 
%%=============================================================================
\newcommand\UD@firstoftwo[2]{#1}%
\newcommand\UD@secondoftwo[2]{#2}%
\newcommand\UD@Exchange[2]{#2#1}%
\newcommand\UD@PassFirstToSecond[2]{#2{#1}}%
@ifdefinable\UD@removespace{\UD@Exchange{ }{\def\UD@removespace}{}}%
@ifdefinable\UD@stopromannumeral{\chardef\UD@stopromannumeral=`^^00}%
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is empty>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked is not empty>}%
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\newcommand\UD@CheckWhetherNull[1]{%
  \romannumeral\expandafter\UD@secondoftwo\string{\expandafter
  \UD@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
  \UD@secondoftwo\string}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@secondoftwo}{%
  \expandafter\UD@stopromannumeral\UD@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether argument's first token is a catcode-1-character
%%.............................................................................
%% \UD@CheckWhetherBrace{<Argument which is to be checked>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked has a leading
%%                        explicit catcode-1-character-token>}%
%%                      {<Tokens to be delivered in case that argument
%%                        which is to be checked does not have a
%%                        leading explicit catcode-1-character-token>}%
\newcommand\UD@CheckWhetherBrace[1]{%
  \romannumeral\expandafter\UD@secondoftwo\expandafter{\expandafter{%
  \string#1.}\expandafter\UD@firstoftwo\expandafter{\expandafter
  \UD@secondoftwo\string}\expandafter\UD@stopromannumeral\UD@firstoftwo}{%
  \expandafter\UD@stopromannumeral\UD@secondoftwo}%
}%
%%-----------------------------------------------------------------------------
%% Check whether brace-balanced argument starts with a space-token
%%.............................................................................
%% \UD@CheckWhetherLeadingExplicitSpace{<Argument which is to be checked>}%
%%                                     {<Tokens to be delivered in case <argument
%%                                       which is to be checked> does have a
%%                                       leading explicit space-token>}%
%%                                     {<Tokens to be delivered in case <argument
%%                                       which is to be checked> does not have a
%%                                       a leading explicit space-token>}%
\newcommand\UD@CheckWhetherLeadingExplicitSpace[1]{%
  \romannumeral\UD@CheckWhetherNull{#1}%
  {\expandafter\UD@stopromannumeral\UD@secondoftwo}%
  {%
    % Let's nest things into \UD@firstoftwo{...}{} to make sure they are nested in braces
    % and thus do not disturb when the test is carried out within \halign/\valign:
    \expandafter\UD@firstoftwo\expandafter{%
      \expandafter\expandafter\expandafter\UD@stopromannumeral
      \romannumeral\expandafter\UD@secondoftwo
      \string{\UD@CheckWhetherLeadingExplicitSpaceB.#1 }{}%
    }{}%
  }%
}%
@ifdefinable\UD@CheckWhetherLeadingExplicitSpaceB{%
  \long\def\UD@CheckWhetherLeadingExplicitSpaceB#1 {%
    \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
    {\UD@Exchange{\UD@firstoftwo}}{\UD@Exchange{\UD@secondoftwo}}%
    {\expandafter\expandafter\expandafter\UD@stopromannumeral
     \expandafter\expandafter\expandafter}%
     \expandafter\UD@secondoftwo\expandafter{\string}%
  }%
}%
%%=============================================================================
\newcommand\CheckWhetherSingleToken[1]{%
  \romannumeral\UD@CheckWhetherNull{#1}{\UD@secondoftwo}{%
    \UD@CheckWhetherBrace{#1}{\UD@secondoftwo}{%
      \UD@CheckWhetherLeadingExplicitSpace{#1}{%
         \expandafter\UD@CheckWhetherNull\expandafter{\UD@removespace#1}%
      }{%
         \expandafter\UD@CheckWhetherNull\expandafter{\UD@firstoftwo{}#1}%
      }%
    }%
  }%
  {\expandafter\UD@stopromannumeral\UD@firstoftwo}%
  {\expandafter\UD@stopromannumeral\UD@secondoftwo}%
}%
%%=============================================================================
\newcommand\something[1]{This is something with single-token-argument: (#1)}
\newcommand\somethingelse[1]{This is something else with multi-token-argument or zero-token-argument=empty argument: (#1)}
\newcommand\foo[1]{%
  \CheckWhetherSingleToken{#1}{\something}{\somethingelse}{#1}%
}%
\makeatother
\begin{document}
\noindent
\verb|\foo x| yields: \foo x\
\verb|\foo{x}| yields: \foo{x}\
\verb|\foo{{x}}| yields: \foo{{x}}\
\verb|\foo{xy}| yields: \foo{xy}\
\verb|\foo{{xy}}| yields: \foo{{xy}}\
\verb|\foo{ }| yields: \foo{ }\
\verb|\foo{ a}| yields: \foo{ a}\
\verb|\foo{}| yields: \foo{}
\end{document}