Code redefining ampersand stops working when put in command

Question

I want to redefine & in a command. I tried to adapt this code that redefines ^ in an environment but couldn't make it work.

\documentclass[]{article}
\begin{document}

\newcommand{\myampersand}{?}

% https://tex.stackexchange.com/a/392770/82912
\begingroup%
\catcode`&=\active%
\gdef\redefineampersand{\def&{\myampersand}}%
\endgroup%

\begingroup%
\catcode`&=\active%
\redefineampersand%
a & b
\endgroup%

\newenvironment{newampersandenv}{%
    \catcode`&=\active%
    \redefineampersand%
}{}
\begin{newampersandenv}
    a & b
\end{newampersandenv}

% I would like to do this but it doesn't work:
\newcommand{\newampersandcmd}[1]{
    \begin{newampersandenv}
        #1
    \end{newampersandenv}
}
%\newampersandcmd{a & b} % Misplaced alignment tab character &. \newampersandcmd{a & b}

% I tried to write it another way but none of those work:

\newcommand{\testa}{
    \begin{newampersandenv}
        a & b
    \end{newampersandenv}
}
%\testa % Misplaced alignment tab character &. \testa

\newcommand{\testb}{
    \catcode`&=\active%
    \redefineampersand%
    a & b
}
%\testb % Misplaced alignment tab character &. \testb

\newcommand{\testc}{
    \catcode`&=\active%
    \redefineampersand%
    \begingroup
    a & b
    \endgroup
}
%\testc % Misplaced alignment tab character &. \testc

\newcommand{\testd}{
    \begingroup
    \catcode`&=\active%
    \redefineampersand%
    a & b
    \endgroup
}
%\testd % Misplaced alignment tab character &. \testd

\end{document}

For the record: For some use cases, it would probably be easier to replace &s instead of redefining it: https://tex.stackexchange.com/a/54987/82912 — xavierm02, Mar 22 '18 at 16:05

Phelype Oleinik · Accepted Answer · 2018-03-22T21:15:39.877

This will work:

\documentclass[]{article}
\begin{document}

\newcommand{\myampersand}{?}

% https://tex.stackexchange.com/a/392770/82912
\begingroup%
\catcode`&=\active%
\gdef\redefineampersand{\def&{\myampersand}}%
\endgroup%

\begingroup%
\catcode`&=\active%
\redefineampersand%
a & b
\endgroup%

\newenvironment{newampersandenv}{%
    \catcode`&=\active%
    \redefineampersand%
}{}
\begin{newampersandenv}
    a & b
\end{newampersandenv}

% I would like to do this but it doesn't work:
\def\newampersandcmd{%
  \begin{newampersandenv}%
  \@newampersandcmd%
}

\def\@newampersandcmd#1{%
      #1%
  \end{newampersandenv}%
}

\newampersandcmd{a & b}

% I tried to write it another way but none of those work:

\begingroup
\catcode`&=\active
\gdef\testa{%
    \begin{newampersandenv}%
        a & b%
    \end{newampersandenv}%
}
\endgroup

\testa

\begingroup
\catcode`&=\active
\gdef\testb{%
    \redefineampersand%
    a & b%
}
\endgroup

\testb

\end{document}

What is happening is a difference of what rules are in force when TeX reads the commands (i.e. executes the \newcommands), and the rules when TeX executes the defined commands.

Let's use, for example, this macro:

\newcommand{\testa}{%
  \catcode`&=\active
  a & b%
}

When TeX reads the \newcommand it turns the input into a list of tokens, and each token has a category code assigned. But the catcodes that are assigned are the ones that are in force when the command is read, so a & b will be translated to catcodes 11 10 4 10 11 which are, respectively, letter, space, alignment tab, space, and letter. But the \catcode&=\activewas never executed when TeX was reading the command, so the&` was still an alignment tab character.

When TeX will execute this command, it will change the catcode of & to \active, then write the list of tokens a & b with the catcodes 11 10 4 10 11. The catcode of the tokens didn't change form the definition time to the execution time.

However, if you move the catcode change outside the command definition:

\catcode`&=\active
\newcommand{\testa}{%
  a & b%
}

TeX will read a & b with catcodes 11 10 13 10 11, and from now on every time the macro \testa is used, these will be the catcodes that TeX will use, independently from the catcodes that are in force when the command is used.

This is the reason that the \makeatletter and \makeatother thing work. When you define an "internal" LaTeX command with an @ in the name you do, for example:

\makeatletter
\def\wrappermacro{%
  \my@m@cr@withm@ny@@@@%
}
\def\my@m@cr@withm@ny@@@@{hello}
\makeatother

the catcode of the @ is changed to 11 at the time the command is defined and changed back to 12 shortly after. And when you use the \wrappermacro, the \my@m@cr@withm@ny@@@@ will work without problems because the catcodes that were in force at the definition time are the ones that count.

Thank you! I really don't understand why this fix is needed when the & is directly in the body of the command thought. — xavierm02, Mar 22 '18 at 16:04
@xavierm02 Ulrike Fischer said I'm more or less correct so... I think that with your version of the command, TeX somehow reads the & before doing the \catcode change, thus the error. My version, on the other hand, first does the \catcode change (by calling \begin{newampersandenv}), and only then calls the \@newampersandcmd, which reads the arguments and writes the & with the new \catcode. — Phelype Oleinik, Mar 22 '18 at 16:21
@xavierm02 Ooh, and when they are in the body of the command, I think it's the same principle. TeX reads the body of the command without expanding anything, so the & still means an alignment tab. Maybe if the command is defined with an \edef that wouldn't happen. I'll check this later and add to the answer :) — Phelype Oleinik, Mar 22 '18 at 16:31
The real reason is that with the OP's definition, in \newampersandcmd{1&2} the & is absorbed and tokenized before \redefineampersand is executed, so the category code of & is still 4 and such remains forever. — egreg, Mar 22 '18 at 16:50
@PhelypeOleinik many thanks for this answer! I was trying to embed a definition like \newampersandcmd into an enclosing command, but got a strange behaviour and I was wondering whether you would know what the problem could be. If I define a simple command \NewDocumentCommand{\DoSomethingFunny}{m}{\newampersandcmd{#1}}, the error message Misplaced... appears. But \DoSomethingFunny{\the\catcode&}will return13- so I thought&` must have the correct catcode at time of execution? — Felix Emanuel, Mar 17 '21 at 17:07
@FelixEmanuel The problem is that \newampersandcmd first changes the catcode of &, and then grabs the argument, whereas your \DoSomethingFunny command first grabs, then changes the catcode, and that order does not work because when it grabs the argument, TeX freezes the catcode of the grabbed & character (catcode 4). \DoSomethingFunny{\the\catcode`&} then says 13 because it queries the current catcode associated with the character &, and not the catcode of that & token. This is the same issue that trying to put something verbatim in a definition like \def\x{\verb|&|} \x. — Phelype Oleinik, Mar 17 '21 at 17:23

xavierm02 · Answer 2 · 2018-03-26T15:40:42.540

My final goal was to have a way to build a macro \makearraysfake such that \makearraysfake[?]{\begin{array}{ll}a&b\end{array}} behaves exactly like a?b. In other words, I wanted to remove all arrays, and merge all the cells (on a line) by concatenating the contents with some given delimiter in between. The following code does this.

(My use case for this was that I use LyX and use arrays to allow putting as many arguments as I want in some enumeration, but I didn't like the spaces added by array.)

\documentclass{article}
\usepackage{xparse}
\usepackage{amsmath}
\usepackage{etoolbox}



% Define global versions of \newcommand and \renewcommand
% https://tex.stackexchange.com/a/51750/82912
\makeatletter
\def\gnewcommand{\g@star@or@long\gnew@command}
\def\grenewcommand{\g@star@or@long\grenew@command}
\def\g@star@or@long#1{% 
    \@ifstar{\let\l@ngrel@x\global#1}{\def\l@ngrel@x{\long\global}#1}}
\def\gnew@command#1{\@testopt{\@gnewcommand#1}0}
\def\@gnewcommand#1[#2]{%
    \kernel@ifnextchar [{\@gxargdef#1[#2]}%
    {\@argdef#1[#2]}}
\let\@gxargdef\@xargdef
\patchcmd{\@gxargdef}{\def}{\gdef}{}{}
\let\grenew@command\renew@command
\patchcmd{\grenew@command}{\new@command}{\gnew@command}{}{}
\makeatother



% This macro is used to apply the trick of https://tex.stackexchange.com/a/422603/82912
\newcommand{\identityendgroup}[1]{%
    #1%
    \endgroup%
}%

% The \makearraysfake command takes two arguments and returns the second one where arrays are removed (but their content remains) and ampersands are replaced by the first argument.
\begingroup%
\catcode`&=\active%

\gnewcommand{\makearraysfake}[1][]{
    \begingroup%
    \catcode`&=\active%
    \def&{#1}%
    \renewenvironment{array}[2][]{}{}%
    \identityendgroup%
}%

\gnewcommand{\makeampersandactive}{
    \begingroup%
    \catcode`&=\active%
    \def&{}%
    \identityendgroup%
}

\endgroup%

\begin{document}

It prevents the errors when \& is used in the second argument, and replaces it by the first argument:

$$
%\boxed{a&b} % Misplaced alignment tab character &. $$\boxed{a&b}
%\not= 
\makearraysfake[b]{\boxed{a&c}}
=
\boxed{abc}
$$

It removes the spaces added by array:

$$
\boxed{
    \begin{array}{ll}
    a&c
    \end{array}
}
\not=
\makearraysfake[]{
    \boxed{
        \begin{array}{ll}
        a&b
        \end{array}
    }
}
=
\boxed{ab}
$$

It can be used in other macros, provided that they are placed in a \textbackslash makeampersandactive macro:

\newcommand{\identity}[1]{#1}

$$
\makeampersandactive{
    \identity{
        \makearraysfake[]{
            \boxed{
                \begin{array}{ll}
                a&b
                \end{array}
            }
        }
    }
}
$$

\end{document}

Code redefining ampersand stops working when put in command

2 Answers2