Detokenizing without extra spaces?

Question

I have a need to pass through potentially arbitrary characters untouched and found this macro:

\makeatletter
\def\test#1{\expandafter\zap@space\detokenize{#1} \@empty}
\makeatother

The problem is that \detokenize inserts spaces after commands in the expansion so I get rid of these with LaTeX's \zap@space. Unfortunately, I need to keep any spaces which \detokenize did not produce. I suspect that there is some cunning way to do this by redefining the catcode of the space character but it is somewhat beyond me.

For example (yes, it's regular expressions I need to pass through ...),

\test{\A\d{2,}.+\z}

Should expand to \A\d{2,}.+\z

but

\test{A test}

should expand to A test and not lose the space that was there originally.

I should say, that \verb doesn't work as these regexp strings are the value of a keyval pair:

\macro[key=\test{#1}]

Update It seems too hard to do this in general without making other things rather messy so I've settled for requiring that any regexps used are canonicalised to use no literal spaces in them. It's always possible to replace them with '\x20' for example in anyway so this means that \zap@space will only be zapping spaces created by \detokenize, which is fine. Many thanks for all of the answers though as they are very instructive.

Answer 1

It is impossible to distinguish \A . from \A. once TeX has converted those into tokens: the only solution if you need to preserve those spaces is to read the argument verbatim.

However, if you are fine with that, then the simplest method is to update the l3kernel and l3experimental bundles (and l3packages) to a very recent version (Februrary 2012), then use tools from the l3regex package to add \string in front of each token in the argument, and expand. The code below does that (replace \tl_show:N by whatever you want to do to the string).

\documentclass{article}
\usepackage{l3regex}
\ExplSyntaxOn
\cs_new_protected:Npn \test #1
  {
    \tl_set:Nn \l_tmpa_tl {#1}
    \regex_replace_all:nnN { . } { \c{string} \0 } \l_tmpa_tl
    \tl_set:Nx \l_tmpb_tl { \l_tmpa_tl }
    % now \l_tmpb_tl contains what you want:
    \tl_show:N \l_tmpb_tl
  }
\ExplSyntaxOff
\begin{document}
  \test{\A\d{2,}.+ Hello, world!\z}
\end{document}

How does it work? \regex_replace_all:nnN performs a replacement on a stored token list, so we need to store the argument.

\tl_set:Nn    % Set locally
  \l_tmpa_tl  % the "local temporary token list" `\l_tmpa_tl`
  {#1}        % to contain "#1" (the argument).
\regex_replace_all:nnN % Replace every occurrence of
  { . }                % any token, even braces etc.
  {                    % by
    \c{string}         %   \string
    \0                 %   what was matched (the token)
  } \l_tmpa_tl         % in \l_tmpa_tl
\tl_set:Nx        % Set locally, with expansion,
  \l_tmpb_tl      % the "local temporary token list b"
  { \l_tmpa_tl }  % to (the expansion of) `\l_tmpa_tl`
\tl_show:N    % Show the contents of
  \l_tmpb_tl  % the token list variable `\l_tmpb_tl` 

Of course, under the hood, l3regex does a lot of work so it will depend on how many such regular expressions you have to go through.

EDIT: A plain TeX solution for the very specific task your are asking for. I am assuming that the strings never contain the character ^^A (char code 1). The idea is to use \lowercase to change all true space tokens to some recognizable character. Then \detokenize, and loop through the result one character at a time (this automatically skips spaces) replacing ^^A by a space.

\catcode64=11
\long\def\test#1%
  {%
    \begingroup
      % Ensure that every character is preserved by \lowercase.
      \count@\z@
      \loop\ifnum\count@<256
        \lccode\count@\z@
        \advance\count@\@ne
      \repeat
      % Except spaces, changed to ^^A
      \lccode32=\@ne
      \lowercase
        {%
          \endgroup
          \edef\result{\expandafter\test@\detokenize{#1}\relax}%
        }%
  }
% Then map {^^A => space, space =>} onto the string.
\def\test@#1%
  {%
    \ifx#1\relax\test@end\fi
    \ifnum`#1=\@ne\space\else#1\fi
    \test@
  }
\def\test@end\fi#1\test@{\fi}
\catcode64=12
\test{ab c\d e{f} \fg }\show\result

Answer 2

Despite the question being tagged as tex-core I would like to point at xparse. It has the argument specification v that does the detokinization without spaces as far as I understand it. Form the manual:

Arguments of type “v” are read in verbatim mode, which will result in the grabbed argument consisting of tokens of category code 12 (“other”), except spaces, which are given category code 10 (“space”).

\DeclareDocumentCommand\foo{v}{\ttfamily #1}

And using it with

\foo!\A\d{2,}.+\z!

produces the same output as

\verb!\A\d{2,}.+\z!

There are no extra spaces introduced by xparse. In this sense the contents of argument #1 is “untouched”.

Answer 3

A truly general solution seems difficult but here is my attempt.

Call \spaceparse with the argument to be detokenized and parsed for spaces after commands. The result of the parsing is available in the macro \result. You would need to call \result to see the outcome.

Because \detokenize doubles the hash character, we first reverse that action. If you don't require this default action, then use the star (*) form of \spaceparse.

You can copy this into a package and call the package.

\documentclass{article}
\usepackage{catoptions}
% No conflict with etoolbox.sty:
% \usepackage{etoolbox}
\makeatletter
\robust@def*\spaceparse{\cpt@testst\sp@ceparse}
\robust@def*\sp@ceparse#1{%
  \begingroup
  \edef\@tempa{\detokenize{#1}}%
  \ifboolTF{cpt@st}{}{\s@expandarg\cpt@pophash\@tempa\@tempa}%
  \edef\@tempa##1{##1\expandcsonce\@tempa\@space\cpt@nil}%
  \edef\@tempb##1{\def##1####1\@space####2\cpt@nil}%
  \@tempb\@tempb{%
    \ifblankTF{##2}{%
      \toks@\expandafter{\the\toks@##1}%
    }{%
      \countbackslash{##1}%
      \ifnum\nr=\@ne
        \xifinsetTF{\@car##2\relax\@nil}\cpt@oth@rchars{%
          \toks@\expandafter{\the\toks@##1}%
        }{%
          \cptexpanded{\toks@{\the\toks@\unexpanded{##1}\@space}}%
        }%
      \else
        \cptexpanded{\toks@{\the\toks@\unexpanded{##1}\@space}}%
      \fi
      \@tempb##2\cpt@nil
    }%
  }%
  \@tempa{\toks@{}\@tempb}%
  \edef\result{\the\toks@}%
  \postgroupdef\result\endgroup
}
\robust@def*\countbackslash#1{%
  \begingroup
  \@tempcnta\z@
  \def\@tempa##1{%
    \def\@tempa####1##1####2\@nil{%
      \ifblankTF{####2}{}{%
        \advance\@tempcnta\@ne
        \@tempa####2\@nil
      }%
    }%
    \@tempa#1##1\@nil
  }%
  \s@expandarg\@tempa\@backslashchar
  \cptexpanded{\endgroup\def\noexpand\nr{\the\@tempcnta}}%
}
\makeatother

Tests:

\def\x{##1\A\d{2,}.+\z A B\\x y}
% Content of \x is already read:
\expandafter\spaceparse\expandafter{\x}
\show\result

\spaceparse{xx\x{f} \fg x}
\show\result

\spaceparse{ab c\d e{f} \fg x}
\show\result

\spaceparse{#1\A\d{2,}.+\z A B}
\show\result

\begin{document}

\end{document} 

Answer 4

Expandable answer: \test below is defined in such a way that it can be used in any context: \edef\foo{\test{...}} works, and similarly \typeout{\test{...}}. This relies on undocumented experimental internal features of LaTeX3, and I only guarantee that it works with today's version of expl3 (it should work with versions from January 8th 2012 until...?).

\RequirePackage{expl3}
\ExplSyntaxOn
\cs_new_nopar:Npn \test
  { \tl_act:NNNnn \test_normal:nN \test_group:nn \test_space:n { } }
\cs_new:Npn \test_space:n #1 { \tl_act_output:n {~} }
\cs_new:Npn \test_normal:nN #1 #2
  { \exp_args:No \tl_act_output:n { \token_to_str:N #2 } }
\cs_new:Npx \test_group:nn #1 #2
  {
    \exp_not:N \exp_args:Nf \exp_not:N \tl_act_output:n
      { \exp_not:N \test { \iow_char:N \{ #2 \iow_char:N \} } }
  }
\ExplSyntaxOff
\typeout{\test{\a b c}}
\typeout{\test{\d*?z{2,} ??\b\p{Lu}}}
\stop

All the magic recursion through brace groups, and finding spaces, is done by \tl_act:NNNnn. Don't ask.