Characters made active and used the same way to define macros behave differently in verbatim environments

Question

The following MCE shows that Unicode and ASCII characters made active and used exactly the same way to define macros, when compiled with lualatex, behave differently in verbatim environments: & works like a charm but ° fails with the following error:

! Argument of \FV@GetKeyValues has an extra }.
<inserted text> 
\par 
l.29 °Bar° in verbatim.

Do you know why these different behaviors?

\documentclass{article}
\usepackage{fancyvrb}
\begin{document}
\catcode`&=\active
\long\def &#1&{\textbf{#1}}
\catcode`\°=\active
\long\def °#1°{\textbf{#1}}
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
°Bar° in text.
\begin{VerbatimOut}{example-2.tex}
°Bar° in verbatim.
\end{VerbatimOut}
\end{document}

Edit

In fact, the different behaviors don't come from Unicode vs ASCII characters since the same problem arises for ! instead of °:

\documentclass{article}
\usepackage{fancyvrb}
\begin{document}
\catcode`&=\active
\long\def &#1&{\textbf{#1}}
\catcode`!=\active
\long\def !#1!{\textbf{#1}}
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
!Bar! in text.
\begin{VerbatimOut}{example-2.tex}
!Bar! in verbatim.
\end{VerbatimOut}
\end{document}

the compilation of which giving:

! Argument of \FV@GetKeyValues has an extra }.
<inserted text> 
\par 
l.29 !Bar! in verbatim.

Edit 2

Here is the context of this question. For a LaTeX tutorial, I'm displaying snippets of LaTeX code and their results. For some of them, I want to highlight some LaTeX commands both in the code and in the result. And for this, I'm using something like the following (in my real files, the LaTeX commands are highlighted with the same colored background as the result and, for this, I used an extended version of this but the whole code would crowd the example):

\documentclass{article}
\usepackage{lmodern}
\usepackage{xcolor}
\usepackage{listings}
\usepackage{fancyvrb}
\lstset{%
  basicstyle=\ttfamily,
  frame=single,
  framesep=2pt,
  aboveskip=1ex,
  moredelim=**[is][\color{red!50}]{°}{°},
}
\newenvironment{showexample}{%
  \VerbatimOut{example.tex}%
}{
  \endVerbatimOut%
  \lstinputlisting{example.tex}
  \input{example.tex}
}
\makeatletter
\g@addto@macro\dospecials{\do\°}
\g@addto@macro@sanitize{@makeother\°}
\makeatother
\catcode`\°=\active
\long\def °#1°{\colorbox{red!50}{#1}}
\begin{document}
\begin{showexample}
°\emph{Bar}°: ``Bar'' is emphazised.
\end{showexample}
\end{document}

Answer 1

Every verbatim environment (also fancyvrb's descendants) explicitly sets & to category code 12 before starting to read the material.

The fancyvrb package has a facility to state category codes of other characters.

\documentclass{article}
\usepackage{fancyvrb}
\begin{document}
\catcode`&=\active
\long\def&#1&{\textbf{#1}}
\catcode`\°=\active
\long\def °#1°{\textbf{#1}}
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
°Bar° in text.
\begin{VerbatimOut}[codes={\catcode`°=12}]{example-2.tex}
°Bar° in verbatim.
\end{VerbatimOut}
\end{document}

This will be the content of example-2.tex

°Bar° in verbatim.

You can also set it “globally”:

\documentclass{article}
\usepackage{fancyvrb}
\fvset{codes={\catcode`°=12}}
\begin{document}
\catcode`&=\active
\long\def&#1&{\textbf{#1}}
\catcode`\°=\active
\long\def °#1°{\textbf{#1}}
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
°Bar° in text.
\begin{VerbatimOut}{example-2.tex}
°Bar° in verbatim.
\end{VerbatimOut}
\end{document}

On the other hand, verbatim does no special treatment to !, because it's normally an other character (category code 12), so if you set it to active it will remain such inside verbatim. So your second example requires

codes={\catcode`!=12}

Answer 2

Currently I use texlive of debian-linux. After updating the debian-package luaotfload-tool, compiling the following via lualatex

\documentclass{article}
\usepackage{iftex}
\ifPDFTeX
  \usepackage[T1]{fontenc}
  \usepackage{textcomp}
  \usepackage[utf8]{inputenc}
\else
  \usepackage{fontspec}
\fi
\usepackage{verbatim, fancyvrb}
\makeatletter
\newcommand\exchange[2]{#2#1}
\newcommand\local@addto@macro[2]{%
  \expandafter\exchange\expandafter{\expandafter\toks@\expandafter{\the\toks@}}{%
    \toks@\expandafter{#1#2}\edef#1{\the\toks@}%
  }%
}%
\makeatother
\begin{document}
\begingroup
\makeatletter
% & is already in the specials-list
\local@addto@macro\dospecials{\do\°}
\local@addto@macro\dospecials{\do!}
\makeatother
\catcode`&=\active
\long\def &#1&{\textbf{#1}}
\catcode`\°=\active
\long\def °#1°{\textbf{#1}}
\catcode`!=\active
\long\def !#1!{\textbf{#1}}
\message{%
  Here \string° and \string! are active, thus they need
  to be added to LaTeX's^^Jverb-mechanism's specials-list:%
}%
\show\dospecials
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
°Bar° in text.
\begin{VerbatimOut}{example-2.tex}
°Bar° in verbatim.
\end{VerbatimOut}
!Baz! in text.
\begin{VerbatimOut}{example-3.tex}
!Baz! in verbatim.
\end{VerbatimOut}
\endgroup
\message{%
  Here \string° and \string! are not active any more, thus they don't
  need to be added to^^JLaTeX's verb-mechanism's specials-list any more:%
}%
\show\dospecials
\section{This is \texttt{example-1.tex}:}
\verbatiminput{example-1.tex}
\section{This is \texttt{example-2.tex}:}
\verbatiminput{example-2.tex}
\section{This is \texttt{example-3.tex}:}
\verbatiminput{example-3.tex}
\end{document}

gets me:

Of course specifying additional special characters via fancyverb's codes=-key, as shown in egreg's answer, is a much better approach. (I seldom use fancyverb, thus i wasn't aware of that key yet while writing my answer. ;-) )

I think usually it should not be a problem to use \g@addto@macro (which redefines the macro with appended tokens globally) instead of \local@addto@macro (where the redefinition is restricted to the current scope as long as \globaldefs is not positive).

I did not do so because I tend to not have things from the kernel changed permanently in order to reduce the risk of badly interfering with changes introduced by whatsoever other packages.

One of these packages could be the inputenc-package: When using utf8-encoding with 8bit-TeX-engines, inputenc makes some characters active which should not be switched to category-code 12(other) when displaying verbatim-material (because they are defined to "look" at following bytes/characters for finding out which multibyte-sequence's/unicode-character's glyph/letter to deliver to the .pdf-output-file) but probably should be switched to category-code 12(other) when writing verbatim-material to external text file (as a means of ensuring that they get written "as is" rather than having attempts at expanding them at writing-time).

Another approach could probably be providing definitions for active &/°/! where checking whether things are processed in verbatim-mode and forking accordingly (delivering the catcode-12(other)-pendant of the active character in question in case of verbatim-processing) takes place. Does somebody know a save/sure/reliable criterion for detecting whether verbatim-processing takes place? (Off the cuff checking the definition of \do (which should be equal to \@makeother) came to my mind, but I doubt that this would be save...)

---

On 8bit-TeX-engines processing utf8-encoded input-files by means of the inputenc-package, the utf8-degree-sign-character usually yields an active character-token of character-code 194(decimal) followed by an active character-token of character-code 176(decimal).

Therefore on 8bit-TeX-engines processing utf8-encoded input-files by means of the inputenc-packages, attempts of expanding these character-tokens need to be prevented at the time when fancyverb's VerbatimOut-environment writes things to file by adding directves to the \dospecials-list for switching these characters to catcode 12(other) before tokenizing the content of the environment.

(Let's hope that on the 8bit-engine these characters get written as they are and that no character translation (.tcx-file) is in use at writing time, transforming (some of) them into ^^-notation or whatever.)

At least on my system the following compiles fine both with lualatex/xelatex (utf8/multibyte-character-engines) and with pdflatex (8bit/single-byte-engine):

\documentclass{article}
\usepackage{iftex}
\newif\ifEightBitEngine
\ifXeTeX\EightBitEnginefalse\else
  \ifLuaTeX\EightBitEnginefalse\else\EightBitEnginetrue\fi
\fi
\makeatletter
\newcommand\exchange[2]{#2#1}
\newcommand\local@addto@macro[2]{%
  \expandafter\exchange\expandafter{\expandafter\toks@\expandafter{\the\toks@}}{%
    \toks@\expandafter{#1#2}\edef#1{\the\toks@}%
  }%
}%
\makeatother
\ifEightBitEngine
  \usepackage[T1]{fontenc}%
  \usepackage{textcomp}%
  \usepackage[utf8]{inputenc}%
\else
  \usepackage{fontspec}%
\fi
\usepackage{verbatim, fancyvrb}
\begin{document}
\begingroup
\makeatletter
% & is already in the specials-list
\local@addto@macro\dospecials{\do!}
\ifEightBitEngine
   \newcommand\makeutfbyteother[1]{%
      \begingroup
      \lccode~=#1\relax \lowercase{\endgroup\catcode~}=12\relax
   }%
   \local@addto@macro\dospecials{\makeutfbyteother{194}}%
   \local@addto@macro\dospecials{\makeutfbyteother{176}}%
\else
  \local@addto@macro\dospecials{\do\°}%
\fi
\makeatother
% Now the changes to \dospecials are in effect.
% They need to be as both our own redefinitions are in effect and (active)
% characters actually forming the single bytes of multibyte-encoded 
% utf8-characters will be written to external file.
%\show\dospecials
\catcode`&=\active
\long\def &#1&{\textbf{#1}}
\ifEightBitEngine\else
  \catcode`\°=\active
\fi
\long\def °#1°{\textbf{#1}}
\catcode`!=\active
\long\def !#1!{\textbf{#1}}
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
°Bar° in text.
\begin{VerbatimOut}{example-2.tex}
°Bar° in verbatim.
\end{VerbatimOut}
!Baz! in text.
\begin{VerbatimOut}{example-3.tex}
!Baz! in verbatim.
\end{VerbatimOut}
\endgroup
% Now the changes to \dospecials are not in effect any more.
% They don't need to be as both our own redefinitions are not in effect any more
% and characters will not be written to file.
%\show\dospecials
\section{This is \texttt{example-1.tex}:}
\verbatiminput{example-1.tex}
\section{This is \texttt{example-2.tex}:}
\verbatiminput{example-2.tex}
\section{This is \texttt{example-3.tex}:}
\verbatiminput{example-3.tex}
\end{document}

Using fancyverb's codes=-key, as shown in egreg's answer, you can probably do something like the following, which (at least on my system) also compiles fine both with lualatex/xelatex (utf8/multibyte-character-engines) and with pdflatex (8bit/single-byte-engine):

\documentclass{article}
\usepackage{iftex}
\newif\ifEightBitEngine
\ifXeTeX\EightBitEnginefalse\else
  \ifLuaTeX\EightBitEnginefalse\else\EightBitEnginetrue\fi
\fi
\ifEightBitEngine
  \usepackage[T1]{fontenc}%
  \usepackage{textcomp}%
  \usepackage[utf8]{inputenc}%
  \newcommand\saveactivechar[1]{%
    \begingroup
    \lccode\~=#1\relax \lowercase{\endgroup\expandafter\let\csname utfbyte\romannumeral\expandafter\string~\endcsname=~}%
  }%
  \newcommand\restoreactivechar[1]{%
    \begingroup
    \lccode\~=#1\relax \lowercase{\endgroup\expandafter\let\expandafter~\expandafter=\csname utfbyte\romannumeral\expandafter\string~\endcsname}%
  }%
\else
  \usepackage{fontspec}%
\fi
\usepackage{verbatim, fancyvrb}
\begin{document}
\catcode`&=\active
\long\def&#1&{\textbf{#1}}
\ifEightBitEngine
   \saveactivechar{194}%
   \saveactivechar{176}%
\else
  \catcode`\°=\active
\fi
\long\def °#1°{\textbf{#1}}
\catcode`!=\active
\long\def !#1!{\textbf{#1}}
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
°Bar° in text.
\begin{VerbatimOut}[codes={\ifEightBitEngine\catcode194=12 \catcode176=12 \else\catcode`°=12 \fi}]{example-2.tex}
°Bar° in verbatim.
\end{VerbatimOut}
!Baz! in text.
\begin{VerbatimOut}[codes={\catcode`!=12}]{example-3.tex}
!Baz! in verbatim.
\end{VerbatimOut}
\section{This is \texttt{example-1.tex}:}
\verbatiminput{example-1.tex}
\section{This is \texttt{example-2.tex}:}
\begingroup
\ifEightBitEngine
  \restoreactivechar{194}%
  \restoreactivechar{176}%
\else
  \catcode`°=12 %
\fi
\verbatiminput{example-2.tex}
\endgroup
\section{This is \texttt{example-3.tex}:}
\begingroup
\catcode`!=12 %
\verbatiminput{example-3.tex}
\endgroup
\end{document}

You can also automatize the saving and restoring of the category-code currently assigned to a character-code and the meaning of the active character with that character-code:

\documentclass{article}
\usepackage{iftex}
\newif\ifEightBitEngine
\ifXeTeX\EightBitEnginefalse\else
  \ifLuaTeX\EightBitEnginefalse\else\EightBitEnginetrue\fi
\fi
\ifEightBitEngine
  \usepackage[T1]{fontenc}%
  \usepackage{textcomp}%
  \usepackage[utf8]{inputenc}%
\else
  \usepackage{fontspec}%
\fi
\usepackage{verbatim, fancyvrb}
\newcommand\savechar[1]{%
  \begingroup
  \lccode\~=#1\relax \lowercase{% \endgroup \expandafter\let\csname charmeaning\romannumeral\expandafter\string~\endcsname=~%
    \expandafter\edef\csname charcatcode\romannumeral\expandafter\string~\endcsname }{\the\catcode#1}% }% \newcommand\restorechar[1]{% \begingroup \lccode~=#1\relax
  \lowercase{%
    \endgroup
    {\expandafter}\expandafter\let\expandafter~\expandafter=\csname charmeaning\romannumeral\expandafter\string~\endcsname \catcode\expandafter\string~=\csname charcatcode\romannumeral\expandafter`\string~\endcsname\relax
  }%
}%
\begin{document}
\savechar{\&}% \catcode&=\active
\long\def&#1&{\textbf{#1}}
\ifEightBitEngine
   \savechar{194}%
   \savechar{176}%
\else
  \savechar{\°}% \catcode\°=\active
\fi
\long\def °#1°{\textbf{#1}}
\savechar{\!}% \catcode!=\active
\long\def !#1!{\textbf{#1}}
&Foo& in text.
\begin{VerbatimOut}{example-1.tex}
&Foo& in verbatim.
\end{VerbatimOut}
°Bar° in text.
\begin{VerbatimOut}[codes={\ifEightBitEngine\catcode194=12 \catcode176=12 \else\catcode`°=12\fi}]{example-2.tex}
°Bar° in verbatim.
\end{VerbatimOut}
!Baz! in text.
\begin{VerbatimOut}[codes={\catcode`!=12}]{example-3.tex}
!Baz! in verbatim.
\end{VerbatimOut}
\section{This is \texttt{example-1.tex}:}
\begingroup
\restorechar{`&}%
\verbatiminput{example-1.tex}
\endgroup
\section{This is \texttt{example-2.tex}:}
\begingroup
\ifEightBitEngine
  \restorechar{194}%
  \restorechar{176}%
\else
  \restorechar{`\°}%
\fi
\verbatiminput{example-2.tex}
\endgroup
\section{This is \texttt{example-3.tex}:}
\begingroup
\restorechar{`!}%
\verbatiminput{example-3.tex}
\endgroup
\end{document}