The following MWE example demonstrates the problem:
\documentclass{article}
\makeatletter
\def\arctg{\mathop{\operator@font ArcTg}\nolimits}
\makeatother
\usepackage{ifxetex,ifluatex}
\ifxetex
\usepackage{fontspec}
\setmainfont{Cambria}
\usepackage{unicode-math}
\setmathfont{Cambria Math}
\else\ifluatex
\usepackage{fontspec}
\defaultfontfeatures{Renderer=Basic,Ligatures={TeX}}
\usepackage{unicode-math}
\setmathfont{Cambria Math}
\else
\usepackage[T1,T2A]{fontenc}
\usepackage[utf8]{inputenc}
\fi\fi
\begin{document}
\ifxetex
This is Xe\LaTeX
\else\ifluatex
This is Lua\LaTeX
\else
This is pdf\LaTeX
\fi\fi
\subsubsection{Function $arctg(x)$}
Function
\begin{equation}
\arctg(x)
\end{equation}
\subsubsection{Function $\cos(x)$}
\begin{equation}
\cos(x)
\end{equation}
\tableofcontents
\end{document}
I tried both Miktex and TL. At first run XeTeX produces .toc file with extremely long record:
\contentsline {subsubsection}{\numberline {0.0.2}Function $\mathop {\Umathcode 97=7\symoperators 97\scan _stop: \Umathcode 98=7\symoperators 98\scan _stop:........
(here less than 1 percent is shown). At the second run xetex fails to finish the compilation. I did not chech if LuaLatex also fails. The compilation succeeds if unicode-math is not loaded.
\DeclareMathOperator{\arctg}{ArcTg}not suficient? – Zarko Aug 10 '17 at 03:51\protect\sinin section headers. Using theamsmathpackage also fixes this error by redefining\sinin another way (which is compatible withunicode-math). Or do you want to understand what happens inside the definition and why it expands? – Sergei Golovan Aug 10 '17 at 06:15\subsubsection{Function $\arctg(x)$}rather than\subsubsection{Function $arctg(x)$}, right? What happens if you insert the missing backslash character? – Mico Aug 10 '17 at 07:14\sinhappens, which is expected because the definition is basically the same. – Sergei Golovan Aug 10 '17 at 08:37amsmaththe commands such as\sinare all fragile, so they should be prefixed by\protectwhen in moving arguments. – egreg Aug 10 '17 at 16:08