\scantokens not playing nice with \def or \gdef etc

Question

I working in a context where I'm changing catcodes and need to rescan the tokens before proceeding to process the text. But, I'm running into problems. Even with this bare minimum code, I still have issues.

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
    \scantokens{\gdef\ae@input{::#1::}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}

Resulting in the error:

! Undefined control sequence.
\aeinput ...ae@input {::#1::}}\endgroup \ae@input 

I get the same error if I write:

\scantokens{\xdef\ae@input{::#1::}}%%

Why is this not compiling?

Also, if I do something like

\expandafter\gdef\expandafter\ae@input\expandafter{\scantokens{::#1::}}%%

I get the following error:

! File ended while scanning definition of \ae@input.

Even adding \everyoef{\noexpand} doesn't help.

HOWEVER, the following does work (which makes me even more confused).

\xdef\ae@input{\scantokens{::#1::}}%%

So, what's happening here?

Where does this question come from?

I thought I had a simple answer for inputing text within an environment for which I've already changed the catcode of . Basically, in that question I have an environment

\begin{speech}
   inside here periods have a different catcode

 \end{speech}

The OP for that question wanted to do something like

\begin{speech}
  \input{text.tex}
\end{speech}

Clearly, \input is not going to work here because . has the wrong catcode. I thought I could easily remedy this (and allow the OP to continue writing \input{text.tex}) by doing something like

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
  \catcode`\.=\periodcatcode
  \scantokens{\gdef\ae@input{\input{#1}}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

An alternative approach would have been to do something like

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
  \catcode`\.=\periodcatcode
  \scantokens{\gdef\ae@filehandle{#1}}%%
  \endgroup
  \input{\ae@filehandle}%%
  }%%
\makeatother

That way, the input file would handle periods appropriately according to the speech environment.

Something like

\newcommand\aeinput[1]{%%
  \begingroup
  \catcode`\.=\periodcatcode
  \scantokens{\input{#1}}%%
  \endgroup
}

Only superficially looks like it could work, but the input file would have not have the correct catcode for the periods anymore.

Answer 1

Remember that \scantokens applies the category codes in force when it's used. Thus in

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
    \scantokens{\gdef\ae@input{::#1::}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}

the \scantokens line is inserted in the doucment, where @ is not a letter. So \gdef\ae@input defines \ae as a macro to be followed by @input with replacement text ::<#1>::. You need to change the category codes inside your group

\documentclass{article}

\makeatletter
\newcommand\aeinput[1]{%%
  \begingroup
    \makeatletter
    \scantokens{\gdef\ae@input{::#1::}}%%
  \endgroup
  \ae@input
  }%%
\makeatother

\begin{document}
\aeinput{HELLO}
\end{document}

As you observe, whilst \scantokens is expandable, getting that right is a bit tricky as it has some oddities about the 'end of file' for the virtual file it uses. Those can be got around (see the \tl_rescan:nn code and similar in expl3), but it's non-trivial.