Wrong order of references in bibliography with diacritical characters

Question

I see incorrect order of references in the bibliography. I use UTF-8 (xelatex) and all names are inputet directly in UTF-8 in the literatura.bib file. Language of the work is English, but referenced authors are from various languages.

\documentclass{article}
\usepackage{natbib}
\bibliographystyle{plainnat}
\begin{document}
\cite{Hughes1985,Horandl2006a,Stepanek2011,Soltis2016,Zhang2017}
\bibliography{literatura}
\end{document}

Renders as:

Hörandl should be before Hughes and Štěpánek between Soltis and Zhang. This order is incorrect. How to fix it?

Here is content of the literatura.bib:

@article{Hughes1985,
author={Hughes, Jane and Richards, A John},
doi={10.1038/hdy.1985.32},
isbn={0018-067X},
issn={0018-067X},
journal={Heredity},
month={apr},
number={2},
pages={245--249},
title={{Isozyme inheritance in diploid \textit{Taraxacum} hybrids}},
url={https://www.nature.com/articles/hdy198532},
volume={54},
year={1985}
}
@article{Horandl2006a,
author={Hörandl, Elvira},
doi={10.1111/j.1469-8137.2006.01769.x},
issn={0028-646X},
journal={New Phytologist},
month={may},
number={2},
pages={525--538},
title={{The complex causality of geographical parthenogenesis}},
url={https://nph.onlinelibrary.wiley.com/doi/full/10.1111/j.1469-8137.2006.01769.x},
volume={171},
year={2006}
}
@article{Stepanek2011,
author={Štěpánek, Jan and Kirschner, Jan and Jarolímová, Vlasta and Kirschnerová, Ludmila},
issn={00327786},
journal={Preslia},
number={4},
pages={537--564},
title={{\textit{Taraxacum nigricans}, \textit{T.~alpestre} and allies in the \textit{Taraxacum} sect. \textit{Alpestria}: Taxonomy, geography and conservation status}},
url={http://www.preslia.cz/2011.html\#stepanek},
volume={83},
year={2011}
}
@article{Soltis2016,
author={Soltis, Douglas E and Visger, Clayton J and Marchant, D Blaine and Soltis, Pamela S},
doi={10.3732/ajb.1500501},
isbn={0002-9122},
issn={0002-9122},
journal={American Journal of Botany},
month={jul},
number={7},
pages={1146--1166},
pmid={27234228},
title={{Polyploidy: Pitfalls and paths to a paradigm}},
url={https://onlinelibrary.wiley.com/doi/abs/10.3732/ajb.1500501},
volume={103},
year={2016}
}
@article{Zhang2017,
archivePrefix={arXiv},
arxivId={1207.3907},
author={Zhang, Yingxiao and Iaffaldano, Brian J and Zhuang, Xiaofeng and Cardina, John and Cornish, Katrina},
doi={10.1186/s12870-016-0967-1},
eprint={1207.3907},
isbn={1287001609671},
issn={1471-2229},
journal={BMC Plant Biology},
month={dec},
number={34},
pages={1--14},
pmid={28152978},
publisher={BMC Plant Biology},
title={{Chloroplast genome resources and molecular markers differentiate rubber dandelion species from weedy relatives}},
url={https://bmcplantbiol.biomedcentral.com/articles/10.1186/s12870-016-0967-1},
volume={17},
year={2017}
}

Answer 1

BibTeX doesn't really do non-ASCII chars. In most situations it can cope fine by just passing the characters along, but as soon as BibTeX needs to 'understand' the characters, things go wrong. BibTeX, for example, can't sort non-ASCII chars (I'm saying that in this generality, even though my BibTeX provided by MikTeX on Windows sorted your MWE just fine, TeX live's BibTeX on Ubuntu, however, gave the same unexpected output).

If you want to stick with BibTeX and want sensible sorting of non-ASCII chars, you need to mark those characters up with macros as described in How to write “ä” and other umlauts and accented letters in bibliography?. You would write

author={H{\"o}randl, Elvira},

and

author={{\v S}t{\v e}p{\'a}nek, Jan and Kirschner, Jan and 
        Jarol{\'i}mov{\'a}, Vlasta and Kirschnerov{\'a}, Ludmila},

BibTeX will then sort characters with diacritics by just ignoring the diacritics. So {\"o} (ö) sorts as o. This is probably fine for English-language texts and I guess what you expect. But it may not be right for languages with more complicated sorting rules such as Swedish ('ö' is a separated letter from 'o' at the end of the alphabet) or German (traditionally name lists sort 'ö' as 'oe').

I don't think this approach is particularly satisfying especially since you use the fully Unicode-aware XeLaTeX engine. Going back to ASCII-with-macros just doesn't feel right.

BibTeX8 is a version of BibTeX that can deal with 8-bit encoded files (so no UTF-8, but things like latin1 would be OK). BibTeX8 can sort non-ASCII according to local rules with help of its .csf files.

Again this doesn't feel quite right, since you can't actually use full UTF-8. Additionally, you have to specify the .csf you wan to sort with, so that may be a bit awkward.

BibTeXu may be another system that could help you out here, but documentation is scarce, it doesn't seem to be developed any more and does not seem to have a vocal user base (at least I can't find a lot on google, but that could also mean that it works so wonderfully that no one had to complain).

Biber finally offers full Unicode/UTF-8 support and would be able to deal with your .bib file as it is without modification. But Biber can not be used together with natbib. Biber is the backend for the biblatex package, a reimplementation of bibliography and citation handling in TeX. See bibtex vs. biber and biblatex vs. natbib.

But if you want to use Biber you have to leave natbib behind and switch to biblatex (What to do to switch to biblatex?). So switching to biblatex is only an option (1) if you are not forced to use a particular .bst style/template and (2) if you are not submitting your work to a journal or publisher (most publisher have a workflow that is incompatible with biblatex, see Biblatex: submitting to a journal).