"Missing character" or "does not map to ascii" message with biblatex and biber

Question

I started drafting a question here but then, through some further work, found a solution. I though it might be worth putting in what I found here for others.

The problem relates to using biber as a back end to biblatex. It is also at least in part a consequence of using BibTeX citations provided unmodified by certain publishers. When running biber, I would get a series of errors that looked like this:

"\x{fffd}" does not map to ascii at Biber/Output/BBL.pm line 422.

I was using a reference file that already contained ~100 references, so sorting these out was not easy (the line 422, did not refer to a line in my reference file). An apparent solution to this problem was to use the -U switch when running biber (biber -U Thesis_1 for a project called Thesis_1). This enabled the expanded unicode character set. However, it then turned out that special characters created with such commands as \"{u}, in order to produce a 'u' with an umlaut would not appear in the pdf. In the log file for pdflatex, instances of messages like:

Missing character: There is no à in font ptmr7t!

would appear (with the -U switch in biber, but not without the switch). It seems that turning on acceptance of unicode characters would also turn off the way that they can be represented in latex with ascii.

I gather that there can be other causes of similar error messages from reading some other forum messages, but I will put in the solution for this instance as an answer. If someone things the question could be modified to be more helpful then please let me know.

Since I was asked, here is a minimal working example:

\documentclass{article}

\usepackage[sorting=none,
           %style=chem-rsc,
           backend=biber,
           ]{biblatex}
\usepackage{filecontents}

\begin{filecontents}{theRefs.bib}
@ARTICLE{Bhalla2000,
  author = {Bhalla, A. S. and Guo, R. and Roy, R.},
  title = {The perovskite structure – a review of its role in ceramic science
    and technology},
  journal = {Materials Research Innovations},
  year = {2000},
  volume = {4},
  pages = {3-26},
}
\end{filecontents}

\addbibresource{theRefs.bib}

\begin{document}
    Here is a reference\cite{Bhalla2000}
    \printbibliography{}
\end{document}

Answer 1

It may be more useful for you to use the --output_safechars option to biber. This will convert all UTF-8 to LaTeX macros when writing the .bbl. The biber PDF manual explains the encoding/decoding issues you may have using PDFLaTeX and inputenc. If you have more trouble, it would be useful to see your usepackage call for biblatex along with the bblencoding and bibencoding values in your .bcf file.

Actually, your problem is probably solved by adding

\usepackage[utf8]{inputenc}

to your MWE. Your PDFLaTeX isn't set up to deal with any UTF-8 characters otherwise. With this, you should just need to run biber with no extra options. However, inputenc does not support all characters, for instance {\u{\i}} is not supported (as of TeXlive 2012).

Answer 2

So, what I found is that though my reference file was not unicode, there were many characters in the file that were part of the ANSI character set and not part of the ASCII set. Another words, they contained characters numbered between 128 and 255. These included such symbols as:

• "–" (character 150 or 0x96) which is a variation of a hyphen
• "°" (character 176 or 0xb0) which is a degree symbol
• " " (character 160 or 0xa0) which is called no break space

In LaTeX, using the ASCII character set, you might represent the hyphen with -, -- or --- depending on the type of mark you want, $^\circ$ for the degree and ~ for a non line breaking space.

My solution was to first locate all characters in the file with a value between 128 and 255. Using a text editor that can search for regular expressions, the following search term should suffice: [€-ÿ]. You should be able to generate the special characters manually with the keyboard by holding down Alt and typing 0128 on the number-pad (Alt and 0255 for the last character). Then go through manually and change all the ANSI characters to a LaTeX representation in ASCII.

Hope this is sensible and makes sense.

I will leave the question as unaccepted for a while, in case someone else wishes to suggest a better solution.