I recommend to have Mathematica not write \text or \operatorname but write some "placeholder-control-sequence" which in LaTeX and in packages used by many usually is not defined. This way you are free to define that placeholder-control-sequence in the preamble/wherever according to your liking without the need of overriding/overloading/redefining control-sequences that are already defined/in use for whatever purposes where they should not be redefined.
Be that as it may:
You can patch the \text-command to check whether its argument consists exactly of the token-sequence S11, u11, b11, s11, t11.
As \text is defined "robust" you actually need to patch the command whose name is \text⟨space⟩.
In the example below the check is implemented by means of a macro which processes delimited arguments rather than by defining temporary macros and doing \ifx-comparison. This way you don't have LaTeX (re)define temporary macros whenever \text is carried out. This in turn might save some of LaTeX's memory when \text is used more often.
With the delimited argument the phrase passed in as argument by the user is nested between exclamation-marks (!)/between something that does not occur within the phrase Subst. Then checking is done on the phrase !Subst!. This way you can crank out the case of the phrase passed in as argument by the user containing ! (and thus differing from the phrase Subst) before doing the checking by means of a !Subst!-delimited argument. Cranking out this case is needed in order to ensure that the phrase passed in as argument by the user does not consist of a set of tokens differing from the delimiter !Subst! but nevertheless containing and therefore matching that delimiter.
\documentclass[12pt]{article}
\usepackage{amsmath}
\makeatletter
%%-----------------------------------------------------------------------------
%% Check whether argument is empty:
%%.............................................................................
%% \UD@CheckWhetherNull{<Argument which is to be checked>}%
%% {<Tokens to be delivered in case that argument
%% which is to be checked is empty>}%
%% {<Tokens to be delivered in case that argument
%% which is to be checked is not empty>}%
%%
%% This macro is explained in detail in:
%% <https://tex.stackexchange.com/a/522506/118714>
%% (The name given to it in that explanation is \CheckWhetherEmpty.)
%%
%% The gist of this macro comes from Robert R. Schneck's \ifempty-macro:
%% <https://groups.google.com/forum/#!original/comp.text.tex/kuOEIQIrElc/lUg37FmhA74J>
\newcommand\UD@CheckWhetherNull[1]{%
\romannumeral0\expandafter\@secondoftwo\string{\expandafter
\@secondoftwo\expandafter{\expandafter{\string#1}\expandafter
\@secondoftwo\string}\expandafter\@firstoftwo\expandafter{\expandafter
\@secondoftwo\string}\@firstoftwo\expandafter{} \@secondoftwo}%
{\@firstoftwo\expandafter{} \@firstoftwo}%
}%
%%-----------------------------------------------------------------------------
\@ifdefinable\UD@GobbleToExclam{\long\def\UD@GobbleToExclam#1!{}}%
%%-----------------------------------------------------------------------------
\@ifdefinable\UD@SubstFork{%
\long\def\UD@SubstFork#1!Subst!#2#3!!!!{#2}%
}%
\newcommand\UD@CheckWhetherSubst[3]{%
\romannumeral0%
\expandafter\UD@CheckWhetherNull\expandafter{\UD@GobbleToExclam#1!}{%
\UD@SubstFork
!#1!{ #2}% Case: #1 = Subst
!Subst!{ #3}% Case: #1 is empty or something else without !
!!!!%
}{ #3}% Case: #1 is something else with !
}%
%%-----------------------------------------------------------------------------
%% Patch the command \text:
%% \text is robust, so actually the command \text<space> needs to be patched.
%%-----------------------------------------------------------------------------
\@ifdefinable\UD@Saved@Command@text{%
\expandafter\let\expandafter\UD@Saved@Command@text\expandafter=\csname text \endcsname
}%
\long\@namedef{text }#1{%
\UD@CheckWhetherSubst{#1}{\operatorname}{\UD@Saved@Command@text}{#1}%
}%
\makeatother
\begin{document}
\begin{align*}
y &= -2 \text{Subst}\left(\int \frac{1}{(3+x)^2} \, dx,x\right)\\
&= -2 \operatorname{Subst}\left(\int \frac{1}{(3+x)^2} \, dx,x\right)\\
&\operatorname{Subst}&(\text{produced by {\tt\string\operatorname\{Subst\}}})\\
&\text{Subst}&(\text{{\tt\string\operatorname}-variant produced by {\tt\string\text\{Subst\}}})\\
&\text{Subst\empty}&(\text{{\tt\string\text}-variant produced by {\tt\string\text\{Subst\string\empty\}}})\\
&\text{!Subst!}\\
&\text{SubstSubst}\\
&\text{Some bla bla}
\end{align*}
\end{document}

TeXFormfor that. It is IMHO much easier to tell Mathematica to make this an\operatornamewhen you export it from Mathematica. In general,TeXFormcan really greatly help avoiding headache. (My head aches because of the italic differentiald. Please use\mathrm{d}x.) – May 10 '20 at 07:52TeXFormalready. But I do not know how to tellTeXFormto only change this specific case to operatorname. I do not want to change all\text{...}to\operatorname{...}, only this one. But I can still do it in mathematica, using string change if needed. Just thought Latex macros can also do this, but was not sure. – Nasser May 10 '20 at 07:56\text{Subst}on the fly or you could redefine\textin a way that makes\textauto-detect if its argument isSubstand switch to\operatornamein that case. But it is not clear whether those two options would be acceptable for you (not everyone wants to use LuaTeX; the\textauto-detection sounds like a bad hack). I think the best way is to tell Mathematica to produce the right TeX code. – moewe May 10 '20 at 08:00\renewcommandis only to change the command name to another name, and not to to change\command{agument}to another\command{argument}. In this case, no problem, I can easily change it in Mathematica before writing it to file with string replace command. Thanks. – Nasser May 10 '20 at 08:03\operatornamecommands. I agree with @moewe. – May 10 '20 at 08:18etoolbox's\ifstrequalinside a redefinition of\text, but as others said, this sounds ugly (this wouldn't work if\textis redefined byamsmathenvironment mid-document, but I doubt this is the case). This way would not require LuaTeX. – frougon May 10 '20 at 08:32\textis not at all a good idea. If you really want all occurrences of\text{Subst}to become\operatorname{Subst}, any text editor replaces this much faster than you copy any TeX solution in the document. Chances are you do not want to replace all of them and/or the TeX solution breaks (for instance because the arXiv has another version ofexpl3, say). Or one of you collaborators has the same great idea of also hacking\text, and the hacks clash. Let just Mathematica do it, much better. – May 10 '20 at 08:45\renewcommandredefines what the macro does with its argument. It is possible to check 'string equality' with the input, so you could in theory redefine\textto branch behaviour on whether or not the input isSubst. But that has a few caveats that have to do with how TeX handles text/strings. – moewe May 10 '20 at 08:47