Why does adding \cprotect\section{} cause bookmarks in PDF to get corrupted? Lots of hyperref removing `\@ifnextchar' messages

Question

Once I added \cprotect around my section titles (I need to do this because I could have math in some of them). Now when I open a PDF file, all bookmarks are lost, and in their place I see something cpt on each one and section and subsections names do not show up. I got an answer on this before to use \usepackage[bookmarks=false]{hyperref}, but at the time I did not know what this meant. Now I noticed it causes PDF books marks to get lost, so I can't use such a solution.

MWE

\documentclass[12pt]{book}
\usepackage{cprotect}
\usepackage{hyperref}

\begin{document}

\chapter{A}
\cprotect\section{B}
\cprotect\subsection{C}
stuff

\cprotect\subsection{D}
stuff

\end{document}

Compiled using lualatex

.....
Chapter 1.
(./foo3-1.cpt)

Package hyperref Warning: Token not allowed in a PDF string (Unicode):
(hyperref)                removing `\@ifnextchar' on input line 8.

(./foo3-2.cpt)

Package hyperref Warning: Token not allowed in a PDF string (Unicode):
(hyperref)                removing `\@ifnextchar' on input line 9.

(./foo3-3.cpt)

Package hyperref Warning: Token not allowed in a PDF string (Unicode):
(hyperref)                removing `\@ifnextchar' on input line 12.

And the PDF file bookmarks look like this (Adobe PDF reader)

Removing cprotect and now it works:

\documentclass[12pt]{book}
\usepackage{hyperref}
\begin{document}

\chapter{A}
\section{B}
\subsection{C}
stuff

\subsection{D}
stuff

\end{document}

gives

I want to use cprotect, but not lose PDF bookmarks. Is there a way to do this?

TL 2020 on Linux

Update to answer comment

Please edit your posting to give an actual example of a \section command that's causing grief.

Here is an example which fails on LuaTeX, unless I use \cprotect, and when I use \cprotect bookmarks are lost. This happens because I am using \usepackage{Baskervaldx} which I like the font

\documentclass[12pt]{book}

\usepackage{unicode-math}
\defaultfontfeatures{Scale=MatchLowercase}
\setmathfont{Asana Math}
\usepackage{Baskervaldx}

\usepackage{amsmath}
\usepackage{hyperref}

\begin{document}
\tableofcontents

\chapter{A}
\section{$\cos\left(  A+B\right)  $ and $\sin\left(  A+B\right)  $}%

\subsection{C}
stuff
\subsection{D}
stuff

\end{document}

Compile using LuaLaTeX gives

Package hyperref Warning: Token not allowed in a PDF string (Unicode):
(hyperref)                removing `math shift' on input line 15.

! Improper alphabetic constant.
<to be read again>
\math@bgroup
l.15 \section{$\cos\left(  A+B\right)  $ and $\sin\left(  A+B\right)  $}
                                                                      %
?

But if I use cprotect it works it compiles with no error, but no bookmarks now

\documentclass[12pt]{book}

\usepackage{unicode-math}
\defaultfontfeatures{Scale=MatchLowercase}
\setmathfont{Asana Math}
\usepackage{Baskervaldx}

\usepackage{amsmath}
\usepackage{hyperref}

\usepackage{cprotect}
\begin{document}
\tableofcontents

\chapter{A}
\cprotect\section{$\cos\left(  A+B\right)  $ and $\sin\left(  A+B\right)  $}%

\subsection{C}
stuff
\subsection{D}
stuff

\end{document}

gives

I have many many such examples. Here is another

 \section{ this is $\zeta$ }%

gives

Package hyperref Warning: Token not allowed in a PDF string (Unicode):
(hyperref)                removing `math shift' on input line 15.

! Improper alphabetic constant.
<to be read again>
\mitzeta
l.15 \section{ this is $\zeta$ }
                              %
?

Please note that these all fail, because I am using the font

\usepackage{unicode-math}
\defaultfontfeatures{Scale=MatchLowercase}
\setmathfont{Asana Math}
\usepackage{Baskervaldx}

I could of course not use the above font, and then it will compile OK and bookmarks will remain there (but without the Math which is OK with me). So maybe I have to do this and use the above font, which I like, but having the bookmarks is more important. So this is an option if there is no other solution.

Is it possible to tell hyperref, if it finds something it can put in the bookmark, only for that section it is OK to replace it with .cpt, but not for everything?

The problem is that I pre-process the whole LaTeX file, and add \cprotect around each section and subsection just in case they have math in them. So now all bookmarks are lost.

I can not do this case by case, since I have 10's of thousands of such entries.

Note on error found and solution

This is too small to write in comment, so I am adding it here.

An error is generated due to wrong order of packages. It has nothing to do with luacode.

This fails

% !TEX TS-program = lualatex
\documentclass{book}

\usepackage{amsmath,mleftright}
\usepackage{unicode-math}
\usepackage{Baskervaldx}
\setmathfont{Asana Math}[Scale=MatchLowercase]
\usepackage{xcolor}
\usepackage[colorlinks,allcolors=blue,linktocpage]{hyperref}

\begin{document}

\section{Solve numerically the ODE $u''''+u=f$ using point collocation method}

test

\end{document}

Compiled using LuaLaTeX gives

t) (./foo3.out)
! Undefined control sequence.
\g__um_prime_font_cmd_tl ->\l__um_font

l.14 \section{Solve numerically the ODE $u''''+u=f$ using point collocation method}

?

The fix is to put \usepackage{Baskervaldx} after \setmathfont{Asana Math}, so the order becomes

\usepackage{amsmath,mleftright}
\usepackage{unicode-math}
\setmathfont{Asana Math}[Scale=MatchLowercase]
\usepackage{Baskervaldx}
\usepackage{xcolor}
\usepackage[colorlinks,allcolors=blue,linktocpage]{hyperref}

And now it compiles OK. It has nothing to do with math in section. Here is an example:

% !TEX TS-program = lualatex
\documentclass{book}
\usepackage{amsmath,mleftright}
\usepackage{unicode-math}
\usepackage{Baskervaldx}
\setmathfont{Asana Math}[Scale=MatchLowercase]

\usepackage{xcolor}
\usepackage[colorlinks,allcolors=blue,linktocpage]{hyperref}

\begin{document}

\section{test}

Solve $y''(x)-3 y(x) = -x^2$ over $x=0\ldots1$ with boundary conditions
$x(0)=0$ and $x(1)=0$ using piecewise linear trial functions.
\end{document}

It compiles with an error:

! Undefined control sequence.
\g__um_prime_font_cmd_tl ->\l__um_font

l.17 Solve $y''(
              x)-3 y(x) = -x^2$ over $x=0\ldots1$ with boundary conditions
?

Again, changing the order of packages, the error is gone. This is why I was getting some error testing Mico's nice code.

Answer 1

I think that using \cprotect in the current context constitutes a pretty severe abuse of the macro. Moreover, as you've discovered, it doesn't work properly since the bookmarks for the pdf viewer program are no longer being generated correctly.

Since you're using LuaLaTeX, I would like to suggest that you pursue a different approach, viz., set up a Lua function which operates at a very early stage, i.e., before TeX starts its usual processing routines. By assigning the Lua function to LuaTeX's process_input_buffer preprocessor callback, it can sweep over all instances of \section, \subsection, and \subsubsection and automatically identify any and all instances of inline math material and place these instances in \texorpdfstring directives, in essence "sanitizing" the math expression for use hyperref's bookmarking routines. For instance,

\subsection{$x^2+y^2=z^2$}

will be replaced "on the fly" with

\subsection{\texorpdfstring{$x^2+y^2=z^2$}{x2+y2=z2}}

and

\section{$\cos\left(A+B\right)$ \textcolor{red}{and} $\sin\left(A+B\right)$}

will be replaced on the fly with

\section{\texorpdfstring{$\cos\left(A+B\right)$}{cos(A+B)} 
         \textcolor{red}{and}   
         \texorpdfstring{$\sin\left(A+B\right)$}{sin(A+B)}}

The code below provides two LaTeX utility macros and two Lua functions. The LaTeX macros are called \texorpdfOn and \texorpdfOff; they serve to activate and deactivate a Lua function called fix_headers. Upon activation, i.e., upon assignment to LuaTeX's process_input_buffer callback, fix_headers checks all input lines; each time it comes across an instance of \section, \subsection, or \subsubsection or its "starred" variants, the Lua function next checks if the argument of that command contains inline math material by searching for pairs of the character $. If a match occurs, a subsidiary Lua function called strip_math is called to generate one or more instances of

\texorpdfstring{$<unmodified math>$}{<sanitized math>}

inside the arguments of \section, \subsection, etc.

The input requirements are as follows:

• Every sectioning command and its argument must be on the same input line. This is definitely the most stringent requirement.

• In any given line of input, there is at most one instance of \section, \subsection, or \subsubsection or of one of the starred variants of these commands. (This is probably more a general input sanity check. However, I thought I should mention it anyway.)

• There are no instances of verbatim material which contain sectioning instructions which, in turn, contain inline-math material. E.g., no instances of \verb+\subsection{$1+1=2$}+. (This could be relaxed by excluding all inline-verbatim material and the contents of environments such as verbatim, Verbatim, and comment from further processing; please pose a new question if this is a concern in practice. Alternatively, run \texorpdfOff just before reaching the verbatim material. Later, upon exiting the verbatim material, you may run \texorpdfOn again.)

• There are no commands named \Xsection, \xyzsection etc. in the document (This requirement is imposed mostly for programming convenience. If need be, this requirement could be relaxed without too much extra work.)

• The arguments of \chapter and \chapter* do not contain inline math material. (This requirement could also be relaxed without too much extra work.)

• The $ character is used to delimit inline math material in the sectioning headers. (Instances of \$, which are used to typeset the $ symbol itself, are permitted.)

• There is no display-math material in the arguments of \section, \subsection, etc. In particular, there are no instances of $$ in the arguments \section, \subsection, etc.

• Nested \frac expressions are not allowed. Non-nested \frac expressions are ok, though. Non-nested expressions of the form \frac{<numer>}{<denom>} are displayed in the bookmark as (<numer>)/(<denom>).

I will keep my fingers crossed that these input requirements aren't too burdensome.

---

% !TEX TS-program = lualatex
%% (compile twice to update the ToC and bookmarks)
\documentclass{book} % or some other suitable document class
\usepackage{luacode} % for 'luacode*' environment
\begin{luacode*}

function strip_math ( u ) 
  -- Drop the '$' delimiters:
  v = u:sub  ( 2 , -2 ) 
  -- Three types of math directives that need to be modified:
      -- directives that need to be removed, e.g, \left and \biggr
      -- directives that need to be modified, e.g., \mid and \prime
      -- all others: just remove the leading backslash (\cos,\int,\log, ...)
  -- Remove all fence-sizing instructions from the input stream:
  v = v:gsub ("\\m?left" , "" ) 
  v = v:gsub ("\\m?right", "" )
  v = v:gsub ("\\[bB]igg?[lrm]?" , "" )
  -- Replace "\frac{...}{...}" with inline-fraction notation:
  v = v:gsub ("\\frac%s-(%b{})%s-(%b{})" , "(%1)/(%2)" ) 
  -- Delete '_' and '^' characters from input stream:
  v = v:gsub ("[%_%^]" , "" )   
  -- Change '\mid' to '|'
  v = v:gsub ("\\mid" , "|" )
  -- Change \prime to '
  v = v:gsub ("\\prime" , "'" )
  -- Finally, change '\int' to 'int', '\sum` to 'sum', '\det' to 'det', etc.
  v = v:gsub ("\\(%a+)", "%1" ) 
  -- Return a "\texorpdfstring" directive:
  return "\\texorpdfstring{"..u.."}{"..v.."}"
end
function fix_headers ( s )
  s = s:gsub ( "(\%l-section[%*]?)%s-(%b{})" ,
        function ( x , y )
        -- Set aside all instances of "$" (if any):
        y = y:gsub ( "\%$", "@@@@@@@@" )
        -- Apply 'strip_math' function if inline-math found: 
        y = y:gsub ( "%b$$" , strip_math )
        -- Restore instances of "$":
        y = y:gsub ( "@@@@@@@@" , "\$" )
        return x..y
        end )
  return s
end

\end{luacode*}
%% Define a couple of utility LaTeX macros:
\newcommand\texorpdfOn{\directlua{luatexbase.add_to_callback(
  "process_input_buffer", fix_headers , "fix_headers" )}}
\newcommand\texorpdfOff{\directlua{luatexbase.remove_from_callback(
  "process_input_buffer", "fix_headers" )}}  

\usepackage{amsmath,mleftright}
\usepackage{unicode-math}
\setmainfont{Baskerville 10 Pro} % pick a suitable text font
\setmathfont{Asana Math}[Scale=MatchLowercase] % pick a suitable math font

\usepackage{xcolor}
\usepackage[colorlinks,allcolors=blue,linktocpage]{hyperref}

\begin{document}
\texorpdfOn % Activate the Lua function 'fix_headers'

\setcounter{secnumdepth}{3} % just for this example
\setcounter{tocdepth}{3}

\tableofcontents

\chapter{AAA}
\section{$\cos\left(  A+B\right)  $ \textcolor{red}{and} $\sin\left(  A+B\right)  $}
\subsection{$\det\bigl(A\bigr)$}
\subsubsection{$\ln \mleft[x\mright]$}
\subsubsection{$x^2+y^2=z^2$}
\subsection{$\int f(x)\,dx$}
\section{\textcolor{violet}{Hello World}}
\section{$\frac{a+b}{c+d}$ or $\frac{u}{v}$}
\subsection{$1+1+1=3$, and \$1+\$1+\$1=\textdollar3}
\subsection{Solve numerically the ODE $u''''+u=f$ using\dots}
\end{document}

Answer 2

The issue doesn't depend on the particular fonts, but on unicode-math.

Using \cprotect is not the solution: you have nothing verbatim in the titles.

You can incrementally collect the “problematic” commands:

\documentclass[12pt]{book}

\usepackage{unicode-math}
\defaultfontfeatures{Scale=MatchLowercase}
%\setmathfont{Asana Math}
%\usepackage{Baskervaldx}

\usepackage{amsmath}
\usepackage{hyperref}

\pdfstringdefDisableCommands{%
  \def\sin{sin}\def\cos{cos}% <-- add here
  \let\left\relax
  \let\right\relax
}

\begin{document}
\tableofcontents

\chapter{A}
\section{$\cos\left(  A+B\right)  $ and $\sin\left(  A+B\right)  $}%

\subsection{C}
stuff
\subsection{D}
stuff

\end{document}

Answer 3

The problem with \cos and \sin can be resolved by making \operator@font robust. I opened an issue for it at the unicode-math github https://github.com/wspr/unicode-math/issues/550

This resolves one problem, it does not mean that every math will work without error.

\documentclass[12pt]{book}

\usepackage{unicode-math}
\setmathfont{Asana Math}
\usepackage{hyperref}
\makeatletter
\ExplSyntaxOn
\cs_set_protected:Npn \operator@font
  {
    \__um_switch_to:n {literal}
    \__um_fontswitch:n { \g__um_operator_mathfont_tl }
  }
\ExplSyntaxOff
\makeatother
\begin{document}
\tableofcontents

\chapter{A}
\section{$\cos\left(  A+B\right)  $ and $\sin\left(  A+B\right)  $}%


\end{document}

Problems with \zeta and similar can be avoid by loading hyperref with the psdextra option:

\documentclass{article}
\usepackage{unicode-math}
\usepackage[psdextra]{hyperref}

\begin{document}

\section{$\zeta$}

\end{document}