Small-caps, old-style numbers, and some ligatures produce odd symbols in PDF copy text?

Question

This is a problem I've had consistently with XeLaTeX. Here's a small example:

\documentclass{article}
\usepackage{fontspec}
\setmainfont[Ligatures=TeX,Numbers={OldStyle,Proportional}]{Minion Pro}

\begin{document}
The \textsc{nato} office 1234
\end{document}

The file compiles and displays just fine. But if I try to copy its text from Acrobat to another application, I get garbage characters in place of all small-caps, old-style numbers, and the ligature Th:

e  office 

This also means that if I search the PDF, there are no hits for "The", "NATO", or "1234". And God help me if the file gets indexed by Google.

Now if I switch to LuaLaTeX, I don't have this problem:

The \textsc{nato} office 1234

The nato office 1234

Then if I use uppercase small-caps, I can get exactly what I want:

The \mbox{\addfontfeatures{Letters=UppercaseSmallCaps}NATO} office 1234

The NATO office 1234

But try as I might with XeLaTeX, it still gives the same garbled copy text. The problem seems to be that in the resulting PDF file, LuaTeX generates "correct" CMap entries while XeTeX generates bugged ones which reference the Private Use Area:

LuaTeX                                        XeTeX
--------------------------------------------- ---------------------------------------------

/CIDInit /ProcSet findresource begin          /CIDInit /ProcSet findresource begin
12 dict begin                                 12 dict begin
begincmap                                     begincmap
...                                           ...
1 begincodespacerange                         1 begincodespacerange
<0000> <FFFF>                                 <0000> <FFFF>
endcodespacerange                             endcodespacerange
0 beginbfrange                                5 beginbfchar
endbfrange                                    <0044> <0063>
13 beginbfchar                                <0046> <0065>
<0044> <0063>                                 <0050> <006F>
<0046> <0065>                                 <0107> <E062>
<0050> <006F>                                 <010F> <FB03>
<0107> <00540068>                             endbfchar
<010F> <FB03>                                 1 beginbfrange
<015D> <0031>                                 <015D> <0160> <F731>
<015E> <0032>                                 endbfrange
<015F> <0033>                                 4 beginbfchar
<0160> <0034>                                 <05C9> <E000>
<05C9> <0041>                                 <05D6> <E044>
<05D6> <004E>                                 <05D7> <E049>
<05D7> <004F>                                 <05DC> <E061>
<05DC> <0054>                                 endbfchar
endbfchar                                     endcmap
endcmap                                       CMapName currentdict /CMap defineresource pop
CMapName currentdict /CMap defineresource pop end
end                                           end
end

Indeed, if I just "transplant" all these CMap entries from LuaTeX's PDF into XeTeX's PDF, the problem goes away! So:

1. Is there a way to make XeTeX emulate the way LuaTeX makes CMap entries?
2. If not, is there a way to edit the font to remove any references to the Private Use Area which may "tempt" XeTeX to make these incorrect CMap entries?
3. If none of that is possible, is there a way to edit the CMap entries in PDFs made by XeTeX, automatically and without needing a LuaTeX run?

---

EDIT: Since Jörg asked, here's how I did the "CMap transplant":

1. Make a file test.tex with the following contents:

   \documentclass{article}
   \usepackage{fontspec}
   \setmainfont[Ligatures=TeX,Numbers={OldStyle,Proportional}]{Minion Pro}
   \begin{document}
   The \mbox{\addfontfeatures{Letters=UppercaseSmallCaps}NATO} office 1234
   \end{document}
   

2. Compile it with xelatex and lualatex and uncompress each PDF with pdftk:

   xelatex test
   mv test.pdf test_x.pdf
   lualatex test
   mv test.pdf test_l.pdf
   pdftk test_x.pdf output test_xu.pdf uncompress
   pdftk test_l.pdf output test_lu.pdf uncompress
   

3. Open each uncompressed PDF file in a text editor and look for the sections above. Delete the section from XeTeX's PDF and replace it with the equivalent section from LuaTeX's PDF (from begincodespacerange to endcmap).

4. The PDF is corrupted now, but pdftk will fix it:

   pdftk test_xu.pdf output test_xf.pdf
   

Now test_xf.pdf will have the correct copy text. This is a neat proof-of-concept but it's useless for several reasons:

• You have to compile in XeTeX and LuaTeX. If my files all compiled in LuaTeX then I'd just use LuaTeX and be done with it. That's the ideal solution here anyways.

• You can't make a "key file" that compiles in LuaTeX and then just put its CMap into every PDF you make using XeTeX, since XeTeX appears to assign random input codes (e.g., <0044>, <0046>, <05C9>) every time the file is changed. Transplanting a CMap which uses a different set of input codes results in no characters being selectable.

• If you use more than one font, you need to fix the CMap for each one. This applies even to the same typeface in two different optical sizes.

So the only way I can see this work is if someone made a program which had a built-in list of fonts and their problematic PUA references, which would go through every CMap section in a PDF line-by-line, identify the PUA references by their targets, and then change the targets according to the list. But that seems far too much work and at a certain point you just have to migrate to LuaTeX.

Answer 1

A partial answer to your second question regarding PUA encodings:

You can remove PUA encodings with ttx, a tool that is included in the Adobe Font Development Kit for OpenType (AFDKO).

However, new fonts such as Arno Pro do not have PUA encodings for smallcaps at all, but compiling an document such as yours still results in small caps not being searchable. Hence, the notion that Jonathan Kews states here is unfortunately wrong. The fault are not PUA encodings, but faulty cmaps generated by XeTeX.

Having said that, in your question you state that you can transplant a correct cmap generated by LuaTeX to a faulty map generated by XeTeX. Maybe a workaround would be to create a correct cmap for every possible glyph and then use that for XeTeX. That would be completely font (and font revision) dependent, of course, but it should work for your private purposes.

Unfortunately I cannot try that as I don't know how to "transplant" a cmap. Could you please elaborate how you did that?

Edit: And I just realised that everything works perfectly with Junicode and EB Garamond (when you specify the SC font separately), i.e. something like:

\documentclass{article}

\usepackage{fontspec}
\setmainfont{Junicode}

\begin{document}

{\addfontfeature{Letters=UppercaseSmallCaps}DIE STRAßE IST ZU SCHMAL FÜR AUTOS.

{\addfontfeatures{Numbers=OldStyle}12345}

\end{document}

compiles perfectly with searchable PDF. Maybe I give up on trying to understand what is going on...