Sanitize U+301 infection and others in references

Question

Too often I messed with bad formated BibTeX files from references databases that are infected with active characters as % or & (usually in the abstract, that is not printed but produce an error anyway) but I can live with this inconvenient and replace those by with \% or \&.

Also with some input encoding errors due to some UTF8 character that cannot be managed by pdflatex. In references this error is more disturbing that in the main text, but in best cases, is only change some greek letter as γ. In others is the insidious zero width U+200B, but with I can search and replace even this character.

The worse for me is the obnoxious U+301, specially when is not in over a consonant but hiding in vowels as normal acute characters, that are perfectly managed using inputenc, i.e., the normal e character (U+0065) plus U+301 look exactly like the kind é (U+00E9) and so on with á, Á, í, etc., so if is hard detect the poisonous diacritic mark.

Disappointingly, search U+0065 plus U+301 find also the single character U+00E9 and vice versa in some editors, so I ended with the pain of searching & replacing every accented letter (single or double) with the single character versión.

So the question is specifically is: There are clevers ways to sanitize .bib files with U+301 tildes?

(Of course, a more general answer, covering some automagic cleaning of all unrecognized and active characters will be welcome.)

The minimal (not) working example:

\documentclass{article}
\begin{filecontents}{test.bib}
@article{xx,
author={González, M. and Ruíz, P.}
title={Mañana hará calor & bochorno con un 90% de humedad}
}   
\end{filecontents}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage[spanish]{babel}
\begin{document}
Hello  \cite{xx}
\bibliography{test}
\bibliographystyle{plain}
\end{document}

Answer 1

To remove this question of the increasing unanswered list and hopefully help novices to deal with these problemas, I answer myself explaining how to use and test uconv but I left as a community wiki answer. I'm not searching reputation here. ;)

In a spanish keyboard for instance, users are accustomed to obtain a single character as "á" typing A + ´ , but really they can be stored as two combining characters or some program could decide that is better convert the single character in two characters (as they were typed originally). In unicode jargon, this is to follow the called "Normalization Form D" (NFD) where "D" stand for "canonical Decomposition" and the opposite is the Normalization Form C (NFC) where the C stand for "Composition". It mean canonical Decomposition followed by canonical Composition (There are more normalizations forms, but this is not relevant for this question).

Supposedly this is nothing about the end user should be worried, because the supposed equivalence should be resolved in the guts of the programs, but reality is not so beautiful: pdflatex only digest texts with NFC and seem that most TeX\BibTeX editors still do not take care of this.

The problems are not only vowels with accents. There is a huge list of possible combining characters as Å,ṩ, etc., so search and replace every case is not a realistic solution in this case.

There are many pages explaining what is NFC, many explaining how to implement with this or that programing language, some pointing to tools that can use NFC outputs (for instance, msort, ucto, rsync or convmv) but of no use to just transcoding a text file, and a few point to the right tool for this purpose:

The easiest way to ensure a NFC is use the uconv in Linux (or Windows with Cygwin)

uconv -x any-nfc inputfile > ouputfile

Example:

Thex.texfile contain only the string "más más" where the first "á" is U+00E1 (NFC) and the second is U+0061 + U+301, but and they look equal:

$ cat x.tex
más más

Although they are not equal, as it is clear in a hexadecimal display:

$ hd x.tex
00000000  6d c3 a1 73 20 6d 61 cc  81 73                |m..s ma..s|
0000000a

Normalization to form C only:

$ uconv -x any-nfc x.tex > y.tex

Testing the result:

$ hd y.tex
00000000  6d c3 a1 73 20 6d c3 a1  73                   |m..s m..s|
00000009

Please do not confuse this program with iconv that is also for transcoding files but afaik does not deal with the decomposed characters in UTF-8 except in version for Mac OS X.

Another solution is in the above link of a simple perl script but own made executables left to end user the work of their "maintenance" (make it widely available, backups, etc.).

On the other hand, problems with single characters maybe are more affordable with a home made script for search and replace most common offending characters, but again, the list of not recognized characters in LaTeX is enough big to make this a very poor very cleaning solution. Maybe some back and forth trascoding to leave more "exotic character" on the road? I mean, for example UTF8 to ISO8859-15, then ISO8859-15 to UTF-8 with only precomposed characters. This work for example to lost hidden and dangerous "zero with spaces" but I am not aware of how important could be lost some characters in real documents. If you know a better way, it would be nice to read them here.