Commands break when processing token by token

Question

I'm trying to make a command that goes in \everymath which formats certain specified characters (automatically puts v in \mathbf, etc.). That part I've actually already figured out, but what's giving me problems is the processing itself. It seems to be separating commands within math mode, either displaying incorrectly or just completely breaking.

This is a minimal version of it that just steps through and changes nothing (in theory):

\ExplSyntaxOn
\NewDocumentCommand{\mformat}{+m}{%
  \exp_after:wN #1 \mode_if_math:TF { \mformat } { }
}
\ExplSyntaxOff

It's used like so:

\everymath{\mformat}

Here's a few examples of how it breaks:

$\vec{v}$
Expected:

Actual: (equivalent to \vec{}v)

\(\mathbf{v}\) % Just gives "Missing } inserted." error

Any ideas on how to fix this?

Edit: Something new I noticed; if I suround the problem commands with braces (like with mathbf, ${\mathbf{v}}$ works fine), they suddenly work perfectly. No idea why this happens.

I've marked egreg's answer as the solution, since it answered the question I asked, but for posterity here is the solution to the question I should have asked (based on PhelypeOleinik's comments)

\ExplSyntaxOn
% Used to keep track of already active characters
\tl_new:N \g__mformat_mathactive_tl
% Used to figure out which characters need to be made normal again
\tl_new:N \l__mformat_remove_mathactive_tl
% Used to keep track of added characters from this iteration
\tl_new:N \l__mformat_used_tl
% Using https://tex.stackexchange.com/a/611898/261875
% and https://tex.stackexchange.com/a/299805/261875
\NewDocumentCommand{\mformat}{m}{
  % By default remove all previous active characters
  \tl_set_eq:NN \l__mformat_remove_mathactive_tl \g__mformat_mathactive_tl
  \tl_set:Nn \l__mformat_used_tl {}
\tl_if_empty:nTF { #1 } {} {
    % Parse the formatting
    \cs_set:Npn __mformat_parse:w ##1[##2]##3\relax {
      % Process each character in the set
      \tl_map_inline:nn { ##2 } {
        \tl_if_in:NnTF \g__mformat_mathactive_tl { ####1 } {
          % If this character is already active, keep it active
          \tl_remove_once:Nn \l__mformat_remove_mathactive_tl { ####1 }
      % Check if the character has been used this iteration
      \tl_if_in:NnTF \l__mformat_used_tl {####1} {
        % Helper needed to have something expandable once
        \cs_set_eq:Nc \__mformat_letter_helper: 
        { __mformat_letter_new_####1: }
        % Add a formatting option to the letter
        \cs_set:cx { __mformat_letter_new_####1: }  { 
          \exp_not:N ##1 { \exp_not:o \__mformat_letter_helper: } 
        }
      } {
        % Record that this has been used
        \tl_put_right:Nn \l__mformat_used_tl { ####1 }
        % Define what the letter will now resolve to
        \cs_set:cx { __mformat_letter_new_####1: }  { 
          \exp_not:N ##1 {\mathchar\use:c { __mformat_mathcode_####1: }} 
        }
      }

      \char_gset_active_eq:nc { `####1 } { __mformat_letter_new_####1: }
    } {
      % Record that this is now an active character
      \tl_gput_right:Nn \g__mformat_mathactive_tl { ####1 }
      % Record that this has been used
      \tl_put_right:Nn \l__mformat_used_tl { ####1 }

      % Record the normal character so it can be used later
      \cs_new:cx { __mformat_mathcode_####1: } 
      { \the\mathcode`####1 }

      % Define what the letter will now resolve to
      \cs_new:cx { __mformat_letter_new_####1: }  { 
        \exp_not:N ##1 {\mathchar\use:c { __mformat_mathcode_####1: }} 
      }
      \char_gset_active_eq:nc { `####1 } { __mformat_letter_new_####1: }

      % Set the character to be active in math mode
      \char_set_mathcode:nn { `####1 } { &quot;8000 }
    }
  }

  % If there's no more character sets, finish, otherwise recurse
  \tl_if_empty:nTF { ##3 } { } { \__mformat_parse:w ##3\relax }
}

% Begin recursive parsing
\__mformat_parse:w #1\relax

}
% \tl_show:N \l__mformat_remove_mathactive_tl
  % Remove the active status from the characters that need it
  \tl_map_inline:Nn \l__mformat_remove_mathactive_tl {
    \tl_gremove_once:Nn \g__mformat_mathactive_tl {##1}
% Reset the math code
\char_set_mathcode:nn { `##1 } { \use:c { __mformat_mathcode_##1: } }

% Deregister functions
\cs_undefine:c { __mformat_letter_new_##1: }
\cs_undefine:c { __mformat_mathcode_##1: }

}
}
\NewDocumentCommand{\std}{m}{ \mathchar\use:c { __mformat_mathcode_#1: } }
\ExplSyntaxOff

Which is used like

\mformat{\mathbb[R]\mathbf[vw]}

Parsing token-by-token like that is dangerous, because you lose braces everywhere. a^{b^c} becomes a^b^c and boom. If you want to go the token-by-token route, I suggest \peek_analysis_map_inline:n, though maybe it's easier to make v math-active instead... — Phelype Oleinik, May 12 '22 at 02:01
@PhelypeOleinik I'll definitely look into that, though what do you mean by making v math-active? Is that a variant of the macro or something else. — Kevin Fisher, May 12 '22 at 02:07
You can make a character behave as a macro in math mode, then define it such that (for example) v prints \mathbf{v}. The basic idea is in the third snippet here, and a friendly interface is \DeclareMathActive from here. — Phelype Oleinik, May 12 '22 at 02:20

score 4 · Answer 1 · answered May 12 '22 at 07:44

The default font for every character in math mode is already explicitly declared, so you just need to adjust the declarations for v. I use bold roman here (matching \mathbf) although your "expected" image showed italic.

\documentclass{article}
\DeclareSymbolFont{boldrm}     {OT1}{cmr}{bx}{n}
\DeclareSymbolFontAlphabet{\mathbf}   {boldrm}
\DeclareMathSymbol{v}{\mathalpha}{boldrm}{`v}
\begin{document}
$\vec{v} + \vec{w}$
\end{document}

egreg · Accepted Answer · 2022-05-12T15:10:00.250

This looks very much like an XY-problem. You say

and changes nothing (in theory)

Sorry, that's not true. The first level expansion of

\mode_if_math:TF { \mformat } { }

is

\if_mode_math: \__prg_TF_true:w \fi: \use_ii:nn {\mformat}{}

If you have $\vec{v}$ you obtain

\vec\if_mode_math: \__prg_TF_true:w \fi: \use_ii:nn {\mformat}{}

and so the argument to \vec is \if_mode_math:. Example:

\documentclass{article}
\ExplSyntaxOn
\NewDocumentCommand{\mformat}{+m}{%
  \exp_after:wN #1 \mode_if_math:TF { \mformat } { }
}
\ExplSyntaxOff
\everymath{\mformat}
\renewcommand{\vec}[1]{\showtokens{#1}}
\begin{document}
$\vec{v}$
\end{document}

On the console I get

No file mform.aux.
> \if_mode_math: .
\vec #1->\showtokens {#1}
l.13 $\vec

It would be better with \exp_last_unbraced:Nf

\documentclass{article}
\ExplSyntaxOn
\NewDocumentCommand{\mformat}{+m}{%
  \exp_last_unbraced:Nf #1 { \mode_if_math:TF { \mformat } { } }
}
\ExplSyntaxOff
\everymath{\mformat}
\renewcommand{\vec}[1]{\showtokens{#1}}
\begin{document}
$\vec{v}$
\end{document}

where the console prints

> v.
\vec #1->\showtokens {#1}

But that's not really good either: with something like

\mathrm{x}

the result is a long series of errors. And even if you fix them by doing a long list of cases, something like

\mathrm{abc}

would turn into

\exp_last_unbraced:Nf abc { \mode_if_math:TF { \mformat } { } }

which is definitely not what you want to see.

Should be \exp_last_unbraced:Nf #1 { \mode_if_math:TF { \mformat } { } }. Doesn't change the outcome in this case, but would break if it used the internal \::f_unbraced — Phelype Oleinik, May 12 '22 at 13:14
Thanks for this, it really makes it clear what's going wrong. I'll try some of the other solutions that have been proposed — Kevin Fisher, May 12 '22 at 16:24

Commands break when processing token by token

2 Answers2