What's special about \: in the implementation of \pgfutil@ifnextchar

Question

There is the following snippet of code in pgfutil-common.tex (with a very similar snippet in pgfmathutil.code.tex`):

{%
  \def\:{\global\let\pgfutil@sptoken= } \:
  \def\:{\pgfutil@xifnch} \expandafter\gdef\: {\futurelet\pgfutil@let@token\pgfutil@ifnch}
}

I've been trying to understand this by means of writing my own version:

\documentclass{article}
\makeatletter

\newcommand\aetest{\ae@test}
\def\ae@test(#1){%%
  \textsf{#1}%%
  \ae@ifnextchar H%%
  {\ae@true}{\ae@false}}

\def\ae@true#1;{ \textbf{(true):#1}}
\def\ae@false#1;{ \textbf{(false):#1}}

\long\def\ae@ifnextchar#1#2#3{%%
  \let\ae@reserved@d=#1%%
  \def\ae@reserved@a{#2}%%
  \def\ae@reserved@b{#3}%%
  \futurelet\ae@let@token\ae@ifnch}
\def\ae@ifnch{%%
  \ifx\ae@let@token\ae@sptoken
    \let\ae@reserved@c\ae@xifnch
  \else
    \ifx\ae@let@token\ae@reserved@d
      \let\ae@reserved@c\ae@reserved@a
    \else
      \let\ae@reserved@c\ae@reserved@b
    \fi
  \fi
  \ae@reserved@c}
{%%
  \def\:{\global\let\ae@sptoken= } \: 
  \def\:{\ae@xifnch} \expandafter\gdef\: {\futurelet\ae@let@token\ae@ifnch}%%
}

\makeatother
\begin{document}

\aetest(This test is) Hello;

\aetest(This test is) Ciao;

\end{document}

The above code compiles correctly and as expected.

The mentioned snippet has been rewritten as:

{%%
  \def\:{\global\let\ae@sptoken= } \: 
  \def\:{\ae@xifnch} \expandafter\gdef\: {\futurelet\ae@let@token\ae@ifnch}%%
}

I believe I understand what this snippet is supposed to do:

(a) \let the token \ae@sptoken to a space token

(b) Scan ahead over any space token in the token list to the next non-space token.

I have two questions:

(1) Why the need to use \: instead of some other control sequence?

(2) Why isn't the space following \expandafter\gdef\: eaten after expansion?

On page 39 of The TeXBook, Knuth writes:

.... spaces are not ignored after control sequences inside a token list; the ignore-space rule applies only in an input file, during the time that strings of characters are being tokenized.

Because of this, I thought I could rewrite the first line of this snippet as:

\def\ae@colon{\global\let\ae@sptoekn= }\ae@colon

I was under the impression that, since the space after the equal sign has already been tokenized, it wouldn't be lost. But what seems to be happening is that when \ae@colon is expanded, this tokenized space is irrelevant. Instead, \ae@sptoken is \let to \def. Though I'm not sure how to verify this or not, LaTeX just complains that \ae@xifnch is undefined. Hence my first question.

Regarding \expandafter\gdef\: and the following space: I was thinking that this space should be lost since \: is expanded before the space is tokenized. Therefore I thought one of the following might work:

\expandafter\gdef\:{....} %%<-- no space following "\:"

or

\gdef\ae@xifnch{....}

or

\gdef\ae@xifnch {...} %%<-- space after \ae@xifnch

But neither work. Instead, LaTeX hangs. I assume that's because it keeps parsing the same space token and is unable to remove it from the list.

So somehow when \expandafter\gdef\: is expanded, the following space is preserved. Hence my second question.

Regarding the choice of a colon, I know there's nothing special with \:. I've experimented with \! and \,. Both work perfectly fine. So, the trick here has something to do with how control symbols (control sequence consisting of one non-letter) work. But, I'm somehow missing how \: is achieving the desired effect here.

Answer 1

I think I now understand what's happening here.

The syntax for \let is as follows

\let<control sequence><equal><one optional space><token>

If I write, where * indicates a space

\let\ae@sptoken=*

then * is just interpreted as the <one optional space> and the next token is grabbed for the \letting.

In the case of \def\:{\let\ae@sptoken= }, the space following the = is the <one optional space>. Now since \: is a command symbol, the space following it will be converted into a space token. Given that when LaTeX reaches the end of the line it converts the end of line character into a space (provided LaTeX is in state M). That means that

\:

expands to

\let\ae@sptoken=**

(again I'm using * to indicate the space). And so \ae@sptoken is \let to a space.

A similar effect can be achieved by:

\def\ae@colon{\let\ae@sptoken= }
\expandafter\ae@colon\space

The trick here is in finding a way to smuggle in two adjacent spaces immediately following the =.

Regarding

\expandafter\gdef\: {....}

I misunderstood when spaces are converted into tokens. The conversion happens prior to the expansion of \expandafter. Thus when the above line is expanded it becomes

\gdef\ae@xifnch*{....}

where * is for the space which, now that it's a token, will not be gobbled by the command string.

A similar effect can be achieve by:

\expandafter\gdef\expandafter\ae@sptoken\space{....}