Texindy sorting Icelandic

Question

I use command texindy -L icelandic -M lang/icelandic/utf8 dict_main.idx to create a list of the names of the photographes, their authors and licenses. But the sorting is not correct (for example Icelandic alphabet ends in this letter order þ æ ö)

MWE:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[]{makeidx}
\usepackage[icelandic, czech]{babel}
\makeindex
\begin{document}
Hello
\index{Þari - Franz Eugen Kohler, Public Domain}
\index{Þistill - ŠARŽÍK František, COPYRIGHT/PD}
\index{Önd - Karney, Lee, PD}
\index{Æðarkóngur - Whitehouse, Laura L., PD}
\index{Avókadó - Forest \& Kim [[p:2684;Starr]], CC-BY}
\index{Auðnutittlingur - Arnstein Rønning, CC BY-SA 3.0}
\index{Asni - Zicha Ondřej, COPYRIGHT/CC-BY-NC}
\index{Á - hvalur.org, CC Unported Licence}
\index{Álft - Bukovský Jiří, COPYRIGHT/CC-BY-NC}
\index{Álka - Jack Spellingbacon from Scotland, CC BY-SA 3.0}

\printindex
\end{document}

Running this commands:

  pdflatex test.tex
  texlua utftexindy.lua -L icelandic test.idx
  pdflatex test.tex

Details of the code in this answer.

Answer 1

I tried to create an index with lualatex. I turned off several packages, esp. babel as there are some hard-coded declarations of characters as shorthands and it wasn't working well with utf-8 coded characters. You probably need those packages for typesetting, but in some cases we may omit them just for index generation.

If we check make-rules-alphabets (it is written in Perl), we spot this line in the /alphabets/icelandic/utf8.pl.in file:

['A', ['a','A'],['á','Á']@u{,['ǫ́','Ǫ́']}],

As I understand it, the @u{} part makes the letters equal for some sorting phase(s). It is included in some other lines:

['E', ['e','E']@u{,['ę','Ę']},['ë','Ë'],['é','É']],
['Æ', ['æ','Æ']@u{,['ǽ','Ǽ'],['ę́','Ę́'],['ǿ','Ǿ']},['œ','Œ'],['ä','Ä']],
['Ö', ['ö','Ö'],['ø','Ø']@u{,['ǫ','Ǫ']}],

We may expect the same behavior. So A and Á as well as Æ and Ǽ are equal at some point. If it is not a desired sorting, xindy community is probably fixing it. I am not certain of that, but it is frequent part of code in alphabets/general/utf8.pl.in where diacritics for sorting purposes is ignored very often.

I believe there is a small bug/mistyping though. It is common to use capital letters for word groups in index, but there is:

I believe this is wrong: ['ð', ['ð','Ð']],
And this should be the correct form: ['Ð', ['ð','Ð']],

We may spot it in the following example too that there is a small letter eth, not the capital letter. I enclose the TeX code and a preview of page 2. We run these lines:

lualatex mal-icelandic.tex
xindy -M texindy -L icelandic -C utf8 mal-icelandic.idx
lualatex mal-icelandic.tex

%! lualatex mal-icelandic.tex
%! xindy -M texindy -L icelandic -C utf8 mal-icelandic.idx
%! lualatex mal-icelandic.tex
% or with two changes: +xltxtra and -luatextra, we run xelatex
\documentclass{article}
%\usepackage[T1]{fontenc}
%\usepackage[utf8]{inputenc}
%\usepackage[icelandic,czech]{babel}
\usepackage{luatextra} % for lualatex run
%\usepackage{xltxtra} % for xelatex run
\usepackage[colorlinks]{hyperref}%hyperindex=false
\usepackage{makeidx}
\makeindex
\begin{document}
The first paragraph of text\ldots
\index{Þari - Franz Eugen Kohler, Public Domain}
\index{Þistill - ŠARŽÍK František, COPYRIGHT/PD}
\index{Önd - Karney, Lee, PD}
\index{Æðarkóngur - Whitehouse, Laura L., PD}
\index{Avókadó - Forest \& Kim [[p:2684;Starr]], CC-BY}
\index{Auðnutittlingur - Arnstein Rønning, CC BY-SA 3.0}
\index{Asni - Zicha Ondřej, COPYRIGHT/CC-BY-NC}
\index{Á - hvalur.org, CC Unported Licence}
\index{Álft - Bukovský Jiří, COPYRIGHT/CC-BY-NC}
\index{Álka - Jack Spellingbacon from Scotland, CC BY-SA 3.0}
\index{Å - a fake index entry}
% a bug? ['ð',  ['ð','Ð']],
\index{Ð - another fake index entry}
\index{E - a testing index entry}
\index{Ǽ a fake}
\begingroup
\pagestyle{empty}
\def\thispagestyle#1{}
\printindex
\endgroup
\end{document}

---

Edit 1: An improved, general version (Icelandic sorting + European Western style with many letters with diacritics)

Please download these two files to your working directory (I cannot post the first file directly here as it contains some special characters which TeX.SX doesn't display):

wget http://striz7.fame.utb.cz/tex-sx/is/icelandicmal.xdy
wget http://striz7.fame.utb.cz/tex-sx/is/icelandicmal-test.xdy

I have created a new set of sorting rules for Icelandic language mixed with general sorting rules for the Western Europe. So you can find letter groups as C, Q, W, Z and Å even if they are not in Icelandic alphabet. There are many letters with diacritics added so sorting words in Czech, Slovak, Polish, German and probably many more languages is taken into account (see general sorting in Xindy).

To get a list of letters (letter groups, order of letters) I use:

lualatex typesetme.tex

I run these lines to get an index:

lualatex mal-icelandicmal.tex
xindy -M texindy -M icelandicmal-test -M mal-style mal-icelandicmal.idx
lualatex mal-icelandicmal.tex

This is a list of letters (code, preview) and an example of index (code, preview). Please test it if it fits your needs.

%! lualatex typesetme.tex
\documentclass[a4paper]{article}
\pagestyle{empty}
\usepackage{luatextra}
\newenvironment{alphabet}{\begin{tabular}{*{16}{l}}%
   }{\end{tabular}}
\addtolength{\voffset}{-1in}
\addtolength{\textheight}{1in}

\begin{document}
\section{Icelandicmal}
\subsection{Alphabet}
\begin{alphabet}
a\,A\\
á\,Á & à\,À & ă\,Ă & â\, & ã\,à & ä\,Ä & ą\,Ą\\
b\,B\\
c\,C & č\,Č & ć\,Ć & ĉ\,Ĉ & ç\,Ç\\
d\,D & ď\,Ď\\
ð\,Ð & đ\,Đ\\
e\,E\\
é\,É & è\,È & ě\,Ě & ê\,Ê & ë\,Ë & ę\,Ę\\
f\,F\\
g\,G & ĝ\,Ĝ & ğ\,Ğ\\
h\,H & ĥ\,Ĥ & ı\,I\\
i\,I\\
í\,Í & ì\,Ì & î\,Î & ï\,Ï\\
j\,J & ĵ\,Ĵ\\
k\,K\\
l\,L & ĺ\,Ĺ & ľ\,Ľ & ł\,Ł\\
m\,M\\
n\,N & ń\,Ń & ň\,Ň & ñ\,Ñ\\
o\,O\\
ó\,Ó & ő\,Ő & ò\,Ò\\
p\,P\\
q\,Q\\
r\,R & ŕ\,Ŕ & ř\,Ř\\
s\,S & ś\,Ś & š\,Š & ŝ\,Ŝ & ş\,Ş\\
t\,T & ť\,Ť\\
u\,U\\
ú\,Ú & ù\,Ù & ŭ\,Ŭ & ů\,Ů & û\,Û & ü\,Ü & ű\,Ű\\
v\,V\\
w\,W\\
x\,X\\
y\,Y\\
ý\,Ý & ÿ\,Ÿ\\
z\,Z & ź\,Ź & ż\,Ż & ž\,Ž\\
þ\,Þ\\
æ\,Æ & ǽ\,Ǽ & œ\,Œ\\
ö\,Ö & ø\,Ø & ǿ\,Ǿ & ô\,Ô & õ\,Õ\\
å\,Å
\end{alphabet}
\subsection{Ligatures}
\begin{flushleft}
`ß' is sorted like `s\,s', but \emph{after} it in otherwise equal words.
\end{flushleft}
\subsection{Upper-/lowercase words}
Capitalized or uppercase words are sorted \emph{before} otherwise equal lowercase words.
\subsection{Special characters}
The order of special characters and letters is:
\begin{flushleft}
?\hspace{4mm}!\hspace{4mm}.\hspace{4mm}letters\hspace{4mm}-\hspace{4mm}'
\end{flushleft}
\end{document}

%! lualatex mal-icelandicmal.tex
%! xindy -M texindy -M icelandicmal-test -M mal-style mal-icelandicmal.idx 
%! lualatex mal-icelandicmal.tex
% or with two changes: +xltxtra and -luatextra, we run xelatex
\documentclass{article}
%\usepackage[T1]{fontenc}
%\usepackage[utf8]{inputenc}
%\usepackage[icelandic,czech]{babel}
\usepackage{luatextra} % for lualatex run
%\usepackage{xltxtra} % for xelatex run
\usepackage[colorlinks]{hyperref}%hyperindex=false
\usepackage{makeidx}
\makeindex
\usepackage{filecontents}
\def\mygroup#1{\textbf{#1}}
\begin{filecontents*}{mal-style.xdy}
;; mal-style.xdy
(markup-letter-group :open-head "~n\mygroup{" :close-head "}")
\end{filecontents*}

\begin{document}
The first paragraph of text\ldots
\index{Þari -- Franz Eugen Kohler, Public Domain}
\index{Þistill -- Šaržík František, COPYRIGHT/PD}
\index{Önd -- Karney, Lee, PD}
\index{Æðarkóngur -- Whitehouse, Laura L., PD}
\index{Avókadó -- Forest \& Kim [[p:2684;Starr]], CC-BY}
\index{Auðnutittlingur -- Arnstein Rønning, CC BY-SA 3.0}
\index{Asni -- Zicha Ondřej, COPYRIGHT/CC-BY-NC}
\index{Á -- hvalur.org, CC Unported Licence}
\index{Álft -- Bukovský Jiří, COPYRIGHT/CC-BY-NC}
\index{Álka -- Jack Spellingbacon from Scotland, CC BY-SA 3.0}
\index{Å -- a fake index entry}
\index{Ð -- another fake index entry}
\index{E -- a testing index entry}
\index{Ǽ a fake}
\begingroup
\pagestyle{empty}
\def\thispagestyle#1{}
\printindex
\endgroup
\end{document}

---

Edit 2: A minimalistic version (Icelandic sorting and its 32 letters only)

Please download two new files:

wget http://striz7.fame.utb.cz/tex-sx/is-min/icelandicmalmin.xdy  
wget http://striz7.fame.utb.cz/tex-sx/is-min/icelandicmalmin-test.xdy  

We run the following four lines:

pdflatex mal-icelandicmalmin.tex  
texlua iec2utf.lua <mal-icelandicmalmin.idx >mal-temp.idx  
xindy -M texindy -M icelandicmalmin-test -M mal-style -o mal-icelandicmalmin.ind mal-temp.idx  
pdflatex mal-icelandicmalmin.tex  

The iec2utf.lua library programmed by michal.h21 is working well. In case you would like to review the included letters, please run:

pdflatex typesetme.tex

I enclose two new files tested with pdflatex, a list of letters in Xindy's style and a preview of sample Icelandic index.

The typesetme.tex file:

%! pdflatex typesetme.tex
\documentclass[a4paper]{article}
\pagestyle{empty}
%\usepackage{luatextra}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\newenvironment{alphabet}{\begin{tabular}{*{16}{l}}%
   }{\end{tabular}}
\addtolength{\voffset}{-1in}
\addtolength{\textheight}{1in}

\begin{document}
\section{Icelandicmalmin}
\subsection{Alphabet}
\begin{alphabet}
a\,A\\
á\,Á\\
b\,B\\
d\,D\\
ð\,Ð\\
e\,E\\
é\,É\\
f\,F\\
g\,G\\
h\,H\\
i\,I\\
í\,Í\\
j\,J\\
k\,K\\
l\,L\\
m\,M\\
n\,N\\
o\,O\\
ó\,Ó\\
p\,P\\
r\,R\\
s\,S\\
t\,T\\
u\,U\\
ú\,Ú\\
v\,V\\
x\,X\\
y\,Y\\
ý\,Ý\\
þ\,Þ\\
æ\,Æ\\
ö\,Ö
\end{alphabet}
%\subsection{Ligatures}
%\begin{flushleft}
%`ß' is sorted like `s\,s', but \emph{after} it in otherwise equal words.
%\end{flushleft}
\subsection{Upper-/lowercase words}
Capitalized or uppercase words are sorted \emph{before} otherwise equal lowercase words.
\subsection{Special characters}
The order of special characters and letters is:
\begin{flushleft}
?\hspace{4mm}!\hspace{4mm}.\hspace{4mm}letters\hspace{4mm}-\hspace{4mm}'
\end{flushleft}
\end{document}

The mal-icelandicmalmin.tex file:

%! pdflatex mal-icelandicmalmin.tex
%! texlua iec2utf.lua <mal-icelandicmalmin.idx >mal-temp.idx
%! xindy -M texindy -M icelandicmalmin-test -M mal-style -o mal-icelandicmalmin.ind mal-temp.idx
%! pdflatex mal-icelandicmalmin.tex
%
% iec2utf.lua <--- https://github.com/michal-h21/iec2utf
\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[icelandic,czech,english]{babel}
%\usepackage{luatextra} % for lualatex run
%\usepackage{xltxtra} % for xelatex run
\usepackage[colorlinks]{hyperref}%hyperindex=false
\usepackage{makeidx}
\makeindex
\usepackage{filecontents}
\def\mygroup#1{\textbf{#1}}
\begin{filecontents*}{mal-style.xdy}
;; mal-style.xdy
(markup-letter-group :open-head "~n\mygroup{" :close-head "}")
\end{filecontents*}

\begin{document}
The first paragraph of text\ldots
\index{Þari -- Franz Eugen Kohler, Public Domain}
\index{Þistill -- Šaržík František, COPYRIGHT/PD}
\index{Önd -- Karney, Lee, PD}
\index{Æðarkóngur -- Whitehouse, Laura L., PD}
\index{Avókadó -- Forest \& Kim [[p:2684;Starr]], CC-BY}
\index{Auðnutittlingur -- Arnstein Rønning, CC BY-SA 3.0}
\index{Asni -- Zicha Ondřej, COPYRIGHT/CC-BY-NC}
\index{Á -- hvalur.org, CC Unported Licence}
\index{Álft -- Bukovský Jiří, COPYRIGHT/CC-BY-NC}
\index{Álka -- Jack Spellingbacon from Scotland, CC BY-SA 3.0}
\index{Ð -- another fake index entry}
\index{E -- a testing index entry}
\index{É -- a testing index entry}
\begingroup
\pagestyle{empty}
\def\thispagestyle#1{}
\printindex
\endgroup
\end{document}