\@startsection, hyperref and newunicodechar; protection needed?

Question

When working with custom defined environments using \@startsection I found, after switching to newunicodechar for defining my unicode characters, that the sectioning headings give this peculiar output:

Here is the code:

\documentclass{article}

\usepackage[colorlinks]{hyperref}

\newcounter{lemma}
\makeatletter
\def\lemmamark{\@gobble}
\newenvironment{lemma}[1]{\@startsection{lemma}{3}{-1em}{\baselineskip}{.1\baselineskip}{\bfseries}{Lemma: #1}}{}
\makeatother

\usepackage{newunicodechar}
\def\textomega{ω} \newunicodechar{ω}{\ifmmode \omega \else \textomega \fi}

\usepackage{fontspec}
\setmainfont[Ligatures=TeX]{STIXGeneral}

\begin{document}

\begin{lemma}{Properties of ω text}
    Some Test
\end{lemma}

\section{Properties of ω}
    More Testing

\end{document}

The following changes do not have any influence:

• Using standard sectioning commands instead of \@startsection (as shown in the example output)
• Removing the \ifmmode conditional (I just left it here, so that you can see the reason for using \newunicodechar at all)
• Using \protect inside of \def\textomega
• Using \renewcommand or \DeclareRobustCommand to define \textomega
• Using the predefined \textomega

I know that I could probably use unicode-math here, but I'm still on TeX Live 2011 and didn't find it usable enough in the general situation.

I'll just post a self-answer on how to circumvent this problem, but what I'm much more interested in, is: Why does this happen? Is this maybe a bug in hyperref or newunicodechar?

---

It gets stranger still! With other, similar characters there is no such problem. For example ψ instead of ω will work with

\def\textpsi{ψ} \newunicodechar{ψ}{\ifmmode \psi \else \textpsi \fi}

Answer 1

Package hyperref generates bookmarks and tries to catch the bookmark titles from LaTeX's section system. The most difficult step is the conversion from a title given as TeX string to a string suitable for PDF bookmark titles. First many macros are redefined to get more sensible output for bookmarks. Also hyperref makes use of NFSS2 and uses encodings PD1 and PU for PDFDocEncoding or Unicode (only these two encodings are allowed in PDF bookmark strings). Then the string is expanded (\edef) and in the cleanup afterwards the tokens are checked. An important step is \HyPsd@CheckCatcodes that controls the tokens. For example, $ for switching to and from math are filtered out. Unhappily the case of active characters is forgotten that are defined using e-TeX's \protected. This is the case for character ω because of \newunicodechar. It gets expanded in the line \ifcat#1A% letter of \HyPsd@CheckCatcodes. The expansion of ω expands \textomega, that expands in PU encoding to \83\311 the octal representation for U+03C9. In this case \8 consists of two tokens both with catcode 12 (other), the \ifcat succeeds and the remaining tokens leak out: 3\311 and the A from the intended token comparison.

I will fix this in hyperref 2012/07/30 v6.82w. An extra test for protected active characters is added and the characters replaced by characters with catcode 12 (other) with the same character code. However the macro text of the active character, that might print something else, must be ignored. Macros defined using \protected are not expanded in \edef. The only way to support such macros and characters is to add appropriate redefinitions of these macros for \pdfstringdef, e.g.:

\newunicodechar{ω}{ohm}
\pdfstringdefDisableCommands{%
  \def ω{ohm}%
}

BTW, keep in mind that \text<charname> macros are usually defined using the NFSS2 system (\DeclareTextSymbol, \DeclareTextCommand, ...). These commands define, how the macro is typeset depending on the current encoding. They overwrite previous definitions of the macro silently.

Answer 2

Any of the following changes will remove the spurious line with “3\311A”:

• Removing the hyperref package
• Removing the statement with \newunicodechar{ω}
• Putting a \protect in front of \textomega inside of \newunicodechar{ω}

• Using a \protect like this:

  \newcommand\unichar[3]{\newunicodechar{#1}{\protect{\ifmmode#2\else#3\fi}}}
  \def\textomega{ω} \unichar{ω}{\omega}{\textomega}
  

The last solution has the benefit of shortness when using a lot of these statements.

---

I found out some more things. The problem is, that fontspec is overwriting my definition of \textomega and so when ω, defined by \newunicodechar, gets expanded, fontspecs \textomega is used and not mine. That explains why on the other hand using different commands from \textomega like \textpsi work: as you will be told when you try to use it, \textpsi is not defined in the EU1 encoding.

So I could simply do my definitions after loading fontspec, but there is a better solution, that completely circumvents the problem:

\newunicodechar{ω}{\ifmmode\omega\else ω\fi}

As I concluded from one of egreg's answers, it is perfectly ok to use the unicode character itself in the argument of \newunicodechar. So now no \textomega is needed anymore.