OS X: umlauts in UTF8-NFD yield "Package inputenc Error: Unicode char \u8:̈ not set up for use with LaTeX"

Question

After switching to OS X one of the first things I had to learn the hard way is that many non-ASCII characters, such as the German ü can be encoded in (at least) two different forms in UTF8:

• U+00FC (LATIN SMALL LETTER U WITH DIAERESIS): Normalized Form C (NFC)
• U+0075 U+0308 (LATIN SMALL LETTER U WITH COMBINING DIAERESIS): Normalized Form D (NFD)

(The glory details are all described here)

Basically, all operating systems and applications today use NFC only, with the exception of Mac OS X, in which some applications (e.g., OpenOffice or the HFS+ file system) use NFD. The result is that if you copy & paste some text from such an application (e.g., the output of the ls command) into your LaTeX document, everything looks fine.

\documentclass{article}
\usepackage[utf8]{inputenc} % comment out for lualatex/xelatex
\usepackage[T1]{fontenc}    % comment out for lualatex/xelatex

\begin{document}
äöüÄÖÜß
\end{document}

However, when compiling with pdflatex:

! Package inputenc Error: Unicode char \u8:̈ not set up for use with LaTeX.

A often given answer with respect to unicode problems is "use lualatex/xelatex". However, that does not seem to help here either. If compiling with lualatex/xelatex, the output does not contain the umlauts:

Question: The inputenc package with [utf8] is apparently not able not handle NFD. Is it possible to extend it so that the above does compile?

---

WARNING

Note that the MWE, if copied & pasted from here into a new document, actually does compile. Apparently either my browser or the SE site transparently transforms NFD to NFC. (For Safari and Crome that seems to be the case indeed; I have also tried Firefox without success). I have yet to figure out how to provide some piece of text in NFD here.

---

Excursus: A Bit of Extra Background on HFS+

I first stumbled over this issue when trying to put the output of a ls command into my LaTeX document: The source of many, many problems in OS X is that the HFS+ file system uses (for some totally weird reasons) NFD. Even worse: HFS+ transparently transforms all NFC characters it gets as input into NFD internally. Practically, this means that the filenames you get out are different than those you have put in: If you create a file ü (the keyboard delivers NFC) and then list the directory (the file system delivers NFD) , the name looks same, but in fact is different. A short illustration test (executed in an empty dir):

$ echo ü; echo ü | xxd; touch ü; ls; ls | xxd
ü
0000000: c3bc 0a                                  ...
ü
0000000: 75cc 880a                                u...

This is the reason so many tools (unison, svn, git, ...) or bash's tab completion choke on OS X on filenames containing umlauts – and that you cannot use the output of ls directly in your LaTeX document.

Answer 1

It's really difficult to support combining characters in inputenc. The package works by making all non-ascii characters "active" with a macro definition. For T1 and similar 8 bit encoding this definition just directly typesets the character in that slot, but for UTF-8, it triggers a little parser looking ahead for the next few bytes of the UTF8 multi-byte encoding to work out which character is needed. Classic TeX typesets a character token as soon as it finds one and places it on the horizontal list being constructed. The important thing to note though is that horizontal lists can not be de-constructed there is no \lastchar primitive analogue of \lastskip or \unskip. This means that by the time you find the combining accent character the base has been typeset into the list and become inaccessible.

There are some TeX tricks that you can play in some cases to get TeX to scan ahead and find these things but they are inherently fragile and tend to break any other loaded package. It is a lot more robust to pass the file through a Unicode normalisation tool and get it back into NFC form.

Or of course use a unicode aware TeX such as luatex or xetex.

Answer 2

(see possible solutions at the end.)

A survey of NFC and NFD UTF-8 forms in XeLaTeX input

xelatex almost handles NFD form almost out-of-the-box. You will need to load the xltxtra package, which you probably always want to load when using XeLaTeX, anyway.

Here's an example bash-script to create a test document (mkutest.sh):

#! /bin/bash
(
  TEXT="åäöüÅÄÖÜß"
  cat <<'EOF'
\documentclass{article}
\usepackage{xltxtra}
\begin{document}
EOF
  echo
  uconv -f utf-8 -t utf-8 -x nfc <<<"UTF-8-NFC: $TEXT"
  echo
  uconv -f utf-8 -t utf-8 -x nfd <<<"UTF-8-NFD: $TEXT"
  echo
  cat <<'EOF'
\end{document}
EOF
) > utest.tex

This script uses uconv (from ICU, See note 1 below) to create the two representations (NFC and NFD) of the same text and adds the XeLaTeX pre-/post-amble. This script should be "safe" to copy from the web page, since it uses the converter and the text input to it can be in any UTF-8 form. (See note 2 below for a version that does not depend on uconv.)

The created file looks like this (utest.tex):

\documentclass{article}
\usepackage{xltxtra}
\begin{document}

UTF-8-NFC: åäöüÅÄÖÜß

UTF-8-NFD: åäöüÅÄÖÜß

\end{document}

(Note: This may not yield the desired file if just copied from the web. See the warning in the question.)

The result of running this through XeLaTeX is a PDF with the text:

where the two lines does not look exactly the same (even apart from the label). The accents in the first line look OK, but the accents of the capital letters in the second line are vastly misaligned.

So, although XeLaTeX can handle NFD form, it may not do it properly...

If \usepackage{xltxtra} is omitted the PDF looks like:

which corroborates the example use of XeLaTeX in the question. Furthermore: Note that nothing at all shows up in the first row and the ß is missing on the second row. This is because the loaded fonts don't have the glyphs to render this. The xltxtra loads the package fontspec, which by default loads the font "Latin Modern". Without this only legacy fonts are loaded, which does not at all play nice with unicode text.

I have tested with different fonts (system fonts loaded with the fontspec command \setmainfont{<name of font>}). The result have been somewhat diverse. For all fonts that have the needed glyphs the first line looks correct. The second line, however, can come out in some different forms. For example with the accents after the base letters, as if they were non-combining; or with missing-glyph-boxes after the base letters...

As Khaled noted, XeTeX can normalize its input to NFC. Adding \XeTeXinputnormalization=1 to the preamble, before any non NFC-text is read, and still using \usepackage{xltxtra} and/or other means to set up proper fonts, the output is:

This time the two lines does look exactly the same (apart from the label).

---

What to do?

If using XeTeX, \XeTeXinputnormalization=1 is definitely a solution. Just remember that you have to properly set up the fonts.

The other way to go, which works with all(?) programs that support UTF-8 NFC text input, is to convert the input files beforehand.

To massage the files into NFC form one can, for example, use uconv (from ICU^{See note 1 below}) as I did in the MWE-generator above.

$ uconv -o outfile.tex -f utf-8 -t utf-8 -x nfc infile.tex

(This works with UTF-16 encoding -- and others -- too. Just change the from (-f) and to (-t) options appropriately.)

Disclamer: Use this command at your own risk. Be sure to keep the original file until you can verify the result.

This should probably be safe to run on any (7-bit) ASCII or UTF-8 encoded tex file. If the file is already in NFC the conversion should not change anything, since it is idempotent. Files containing only 7-bit ascii are already in NFC, since 7-bit ASCII is a subset of UTF-8 and contains no combining characters that could make the text non-NFC.

---

Notes

1. The uconv utility from ICU is in the package libiuc-dev on my Ubuntu 12.04 64-bit.
   (I think it is among the examples for the ICU4C library, but I could not find any info about the it from a quick search on the homepage. I'm a bit confused...)

2. As requested by David in his comment I have made a version of the MWE-generator that does not depend on uconv.

   #!/bin/bash
   (
     echo '\documentclass{article}'
     echo '\usepackage{xltxtra}'
     echo '\begin{document}'
     echo
     echo -e 'UTF-8-NFC: \xc3\xa5\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x85\xc3\x84\xc3\x96\xc3\x9c\xc3\x9f'
     echo
     echo -e 'UTF-8-NFD: \x61\xcc\x8a\x61\xcc\x88\x6f\xcc\x88\x75\xcc\x88\x41\xcc\x8a\x41\xcc\x88\x4f\xcc\x88\x55\xcc\x88\xc3\x9f'
     echo
     echo '\end{document}'
   ) > utest.tex
   

   This version only depends on that echo -e interprets \xHH (and that echo without -e does not).

   I kept the other version (above, in the main text) since it allows for easy changes in the sample text.

   For the interested, the hex escapes are generated by uconv -x '[:Cc:]>; ::nfc;' <<<"$TEXT" | hexdump -v -e '/1 "%02x "' | sed -e 's/[[:xdigit:]][[:xdigit:]]/\\x\0/g; s/ //g' for NFC, &sim. for NFD.

Answer 3

Apparently, there is no easy solution to this problem within LaTeX itself. So if one does not want to use XeLaTeX with \XeTeXinputnormalization=1, as suggested by Khaled, the best approach is probably to normalize the input (source) to NFC, as suggested by David.

This is what I have been doing in the past. It is, however, less easy as it sounds: There are surprisingly few tools available that actually can do this kind of normalization. In particular iconv and the system library behind it, which is used by most editors (including vim) for encoding transformations, is not NFD capable. So in the following I would like to present tools and approaches to do the conversion. I make this answer community wiki, so please contribute to it if you know more tools.

The Poor Man's Approach (OS X): TextEdit

• Open the document (or just paste the NFD text) in TextEdit
• Set TextEdit to plain text format (Format->Make Plain Text)
• Save the document in some other encoding (not UTF-8) that contains the characters represented in NFD. (For instance for German umlauts you choose Western (ISO Latin 9)).
• Save the document again in UTF-8 encoding (or just copy the text from TextEdit into your LaTeX document). The resulting encoding is now NFC.

The advantage of this approach is that it works without further third-party tools. It requires, however, some manual steps and the transition over some intermediate encoding implies the danger of loosing uncommon characters.

Real Conversion Tools

uconf

The uconv utility is the most complete converter I have found so far. It actually is an example application of the ICU project, an extensive set of open-source libraries and APIs to deal with unicode issues that is targeted at software developers. It seems to be available in some Linux distros, but unfortunately is not yet part of OS X nor MacPorts. So you have to download and compile it from the sources, which, however, works flawlessly.

---

Update March 2015: Meanwhile, uconv can be obtained easily for most platforms: On MacPorts by installing the package icu, Homebrew users install icu4c, and on Ubuntu the package is named icu-devtools.

---

Once, uconf is available, you can use it like iconv, with the exception that you have an extra -x command line option to specify the normalization form. The following, executed in a bash shell, converts paper.tex to paper-nfc.tex:

uconv -f utf-8 -t utf-8 -x NFC paper.tex >paper-nfc.tex

A useful one-liner for OS X is the following. It recodes the content of the clipboard to NFC:

pbpaste | uconv -f utf-8 -t utf-8 -x NFC | pbcopy

Today I use this when copying content, such as the output of ls or text blocks from an OpenOffice document into my LaTeX document. Just execute the above line after copying and before pasting and everything is fine. (For some extra comfort, one might create and Automator workflow that runs this line and assign it a keyboard shortcut to the workflow.)

Online Normalization Tools

Yet another possibility would be to use some online web service for transformations. However, so far I have found only this one:

• Unicode Normalization Test Page: Supports just a single line; also provides output in hex.

Answer 4

I want to supplement the David's answer. He wrote that there is no possibility to deal with NFD utf8 codes in 8bit TeX (i.e. in pdfTeX) because the base character is set to horizontal mode and it is not accessible when we read the next code.

This is not exactly true because pdfTeX includes encTeX extension. This extension is able to declare utf8 codes as one byte or one control sequence. It reads byte sequences and converts them to control sequences at input processor level. CSplain activates encTeX during format generation and reads the table with the basic set of NFC utf8 codes. It pre-loads the 8bit font with all characters mentioned in the example here and the NFC utf8 codes are mapped to the character slots in the font. So, we can simply write åäöüÅÄÖÜß and we get åäöüÅÄÖÜß. The DFC codes are not prepared by csplain but they can be simply added by the following code.

% use csplain format
% syntax: \mubyte control-sequence byte-sequence\endmubyte
\mubyte\nfdaring     a^^cc^^8a\endmubyte \let\nfdaring=å
\mubyte\nfdadieresis a^^cc^^88\endmubyte \let\nfdadieresis=ä
\mubyte\nfdodieresis o^^cc^^88\endmubyte \let\nfdodieresis=ö
\mubyte\nfdudieresis u^^cc^^88\endmubyte \let\nfdudieresis=ü
\mubyte\nfdAring     A^^cc^^8a\endmubyte \let\nfdAring=Å
\mubyte\nfdAdieresis A^^cc^^88\endmubyte \let\nfdAdieresis=Ä
\mubyte\nfdOdieresis O^^cc^^88\endmubyte \let\nfdOdieresis=Ö
\mubyte\nfdUdieresis U^^cc^^88\endmubyte \let\nfdUdieresis=Ü

UTF-8-NFC: åäöüÅÄÖÜß

UTF-8-NFD: åäöüÅÄÖÜß % NFD codes generated by the uconv, see above the accepted answer

\end

Note, that the utf8 codes used in \let command arguments are normal NFC codes. They are re-interpreded by encTeX as one byte (one slot in the preloaded 8-bit font).

Note2: the byte sequence for DFC codes begins by the base character. If this base character follows the next declared bytes then this byte sequence is transformed by encTeX to the declared control sequence. If the base character isn't followed by the declared code then it is treated as normal single character (inside control sequences or as a simple character to be typeset).

Answer 5

In case anyone writes the documents with python. Make sure you save it as UTF-8:

with open(f'{filename}.tex', 'w', encoding='utf-8') as tex_file:
    tex_file.write(latex_document)