Longer post
I have prepared two versions for Pāḷi language to be used by xindy: general one (English sorting rules plus several new letters and a portion of sorting rules from Pāḷi) and a minimal one (for Pāḷi language only).
I am running examples with lualatex, those work with xelatex too. The engines use Latin Modern for typesetting. Several phonetic letters weren't rendered correctly, so I used Code2000 font. If you want to run pdflatex, please, consider to use iec2utf, that's a Lua script programmed by michal-h21.
I have used several resources (listed below) to get Pāḷi words as test cases. In case I misspelled some of them, I am sorry for that.
To 1. General version (Sorting rules of some Western languages plus Pāḷi)
I wanted to make sure that not only letters will be added but also that Pāḷi words will be sorted properly. I have extracted new letters from http://www.omniglot.com/writing/pali.htm and http://en.wikipedia.org/wiki/Pali. Moreover, I've also kept in mind this file http://pratyeka.org/narada/pali_alphabets.pdf and I've added several phonetic symbols to the alphabet as an experiment if Xindy can handle that (it can).
I have downloaded and installed Code2000 font to make sure that all letters, especially those uncommon ones, will be displayed correctly.
I enclose a list of 36 letter groups:
% run: lualatex or xelatex typeset-paligeneral.tex
% Alphabet has 36 elements.
\documentclass[a4paper]{article}
\pagestyle{empty}
%\usepackage{luatextra}
%\usepackage{xltxtra}
\newenvironment{alphabet}{\begin{tabular}{*{16}{l}}}{\end{tabular}}
\addtolength{\voffset}{-0.5in}
\addtolength{\textheight}{2in}
\usepackage{fontspec}
% http://web.archive.org/web/20101122142710/http://code2000.net/code2000_page.htm
\setmainfont{Code2000}
\begin{document}
\section{Paligeneral}
\subsection{Alphabet}
\begin{alphabet}
a\,A & á\,Á & à\,À & ă\,Ă & â\, & ã\,à & ä\,Ä & ą\,Ą & å\,Å & æ\,Æ & ǽ\,Ǽ\\
ā\,Ā\\
b\,B\\
c\,C & ć\,Ć & ĉ\,Ĉ & ç\,Ç & č\,Č\\
d\,D & ð\,ð & đ\,đ & ď\,Ď\\
ḍ\,Ḍ & ɖ\,Ɖ\\
e\,E & é\,É & è\,È & ě\,Ě & ê\,Ê & ë\,Ë & ę\,Ę & þ\,Þ\\
f\,F\\
g\,G & ĝ\,Ĝ & ğ\,Ğ\\
h\,H & ĥ\,Ĥ & ı\,I\\
i\,I & í\,Í & ì\,Ì & î\,Î & ï\,Ï\\
ī\,Ī\\
j\,J & ĵ\,Ĵ\\
k\,K\\
l\,L & ĺ\,Ĺ & ł\,Ł & ľ\,Ľ\\
ḷ\,Ḷ & ɭ\,ɭ\\
m\,M\\
ṃ\,Ṃ & ŋ\,Ŋ & ɱ\,ɱ\\
n\,N & ń\,Ń & ň\,Ň\\
ṅ\,Ṅ & ɲ\,Ɲ\\
ñ\,Ñ\\
ṇ\,Ṇ & ɳ\,ɳ\\
o\,O & ó\,Ó & ő\,Ő & ò\,Ò & ö\,Ö & ø\,Ø & ǿ\,Ǿ & ô\,Ô & õ\,Õ & œ\,Œ\\
p\,P\\
q\,Q\\
r\,R & ŕ\,Ŕ & ř\,Ř\\
s\,S & ś\,Ś & ŝ\,Ŝ & ş\,Ş & š\,Š\\
t\,T & ť\,Ť\\
ṭ\,Ṭ & ʈ\,Ʈ\\
u\,U & ú\,Ú & ù\,Ù & ŭ\,Ŭ & ů\,Ů & û\,Û & ü\,Ü & ű\,Ű\\
ū\,Ū\\
v\,V\\
w\,W\\
x\,X\\
y\,Y & ý\,Ý & ÿ\,Ÿ\\
z\,Z & ź\,Ź & ż\,Ż & ž\,Ž
\end{alphabet}
\subsection{Ligatures}
\begin{flushleft}
`ß' is sorted like `s\,s', but \emph{after} it in otherwise equal words.
\end{flushleft}
\subsection{Upper-/lowercase words}
Capitalized or uppercase words are sorted \emph{before} otherwise equal lowercase words.
\subsection{Special characters}
The order of special characters and letters is:
\begin{flushleft}
?\hspace{4mm}!\hspace{4mm}.\hspace{4mm}letters\hspace{4mm}-\hspace{4mm}'
\end{flushleft}
\end{document}

Usage
Please download these two files to your working directory:
wget http://striz7.fame.utb.cz/tex-sx/pi/utf8.pl-paligeneral.in.xdy
wget http://striz7.fame.utb.cz/tex-sx/pi/utf8.pl-paligeneral.in-test.xdy
We run these three lines on the first example (mal-paligeneral.tex):
lualatex mal-paligeneral.tex
xindy -M texindy -M utf8.pl-paligeneral.in-test -M bonus mal-paligeneral.idx
lualatex mal-paligeneral.tex
If you wish to check out the vector versions, please download:
wget http://striz7.fame.utb.cz/tex-sx/pi/typeset-paligeneral.pdf
wget http://striz7.fame.utb.cz/tex-sx/pi/mal-paligeneral.pdf
This is the content of the TeX file, I will enclose a preview of it at the end of this post so we can compare those two versions.
% run: lualatex or xelatex mal-paligeneral.tex
% uncomment line 9 or 10, respectively
%
%lualatex mal-paligeneral.tex
%xindy -M texindy -M utf8.pl-paligeneral.in-test -M bonus mal-paligeneral.idx
%lualatex mal-paligeneral.tex
%
\documentclass[a4paper]{article}
\usepackage{luatextra} % for lualatex engine
%\usepackage{xltxtra} % for xelatex engine
\pagestyle{empty}
\usepackage{makeidx}
\makeindex
\usepackage[colorlinks]{hyperref}
\usepackage{filecontents}
\begin{filecontents*}{bonus.xdy}
(markup-letter-group :open-head "~n \textbf{" :close-head "}")
\end{filecontents*}
\begin{document}
Regular text.
\index{pāḷi}
\index{ānāpānasati}
\index{saṃsāra}
\index{ñāṇavimutti}
% Pali Buddhist Dictionary
\index{insight}
\index{paññā}
\index{vipassanā}
\index{ñāṇa}
\index{nirvāṇa}
\index{nibbāna}
\index{permanency}
%\index{vipallāsa}
%\index{personality}
%\index{sakkāya}
%\index{diṭṭhi}
% sortedCanonList.txt
\index{aādiparokkhāyañca}
\index{sammāyojitajālavāta}
\index{ṃsati}
\index{ḷukhappa}
\index{homaidaṃ}
\index{sumanosmīti}
\index{sāmākanīvāre}
\index{vīriyārambhassa}
\index{jeṭṭhasissa}
\index{ñattikammavācāpi}
\index{ñattikammaṃ}
\index{ārammaṇamariyādā}
\index{aāgamo}
\begingroup
\def\thispagestyle#1{}
\printindex
\endgroup
\end{document}
To 2. Minimal version (Pāḷi language only)
This is a list of 41 letter groups used in the alphabet.
% run: lualatex or xelatex typeset-paliminimal.tex
% Alphabet has 41 elements.
\documentclass[a4paper]{article}
\pagestyle{empty}
%\usepackage{luatextra}
%\usepackage{xltxtra}
\newenvironment{alphabet}{\begin{tabular}{*{16}{l}}}{\end{tabular}}
\addtolength{\voffset}{-0.75in}
\addtolength{\textheight}{2in}
\usepackage{fontspec}
% http://web.archive.org/web/20101122142710/http://code2000.net/code2000_page.htm
\setmainfont{Code2000}
\begin{document}
\section{Paliminimal}
\subsection{Alphabet}
\begin{alphabet}
a\,A\\
ā\,Ā\\
i\,I\\
ī\,Ī\\
u\,U\\
ū\,Ū\\
e\,E\\
o\,O\\
k\,K\\
kh\,Kh\,KH\\
g\,G\\
gh\,Gh\,GH\\
ṅ\,Ṅ & ɲ\,Ɲ\\
c\,C\\
ch\,Ch\,CH\\
j\,J\\
jh\,Jh\,JH\\
ñ\,Ñ\\
ṭ\,Ṭ & ʈ\,Ʈ\\
ṭh\,Ṭh\,ṬH\\
ḍ\,Ḍ & ɖ\,Ɖ\\
ḍh\,Ḍh\,ḌH\\
ṇ\,Ṇ & ɳ\,ɳ\\
t\,T\\
th\,Th\,TH\\
d\,D\\
dh\,Dh\,DH\\
n\,N\\
p\,P\\
ph\,Ph\,PH\\
b\,B\\
bh\,Bh\,BH\\
m\,M\\
y\,Y\\
r\,R\\
l\,L\\
v\,V\\
s\,S\\
h\,H\\
ḷ\,Ḷ & ɭ\,ɭ\\
ṃ\,Ṃ & ŋ\,Ŋ & ɱ\,ɱ
\end{alphabet}
\subsection{Ligatures}
\begin{flushleft}
`ß' is sorted like `s\,s', but \emph{after} it in otherwise equal words.
\end{flushleft}
\subsection{Upper-/lowercase words}
Capitalized or uppercase words are sorted \emph{before} otherwise equal lowercase words.
\subsection{Special characters}
The order of special characters and letters is:
\begin{flushleft}
?\hspace{4mm}!\hspace{4mm}.\hspace{4mm}letters\hspace{4mm}-\hspace{4mm}'
\end{flushleft}
\end{document}

Usage
Please download these two files:
wget http://striz7.fame.utb.cz/tex-sx/pi/utf8.pl-paliminimal.in.xdy
wget http://striz7.fame.utb.cz/tex-sx/pi/utf8.pl-paliminimal.in-test.xdy
We run these three lines (the terms are identical to those used in the first example for comparison reasons):
lualatex mal-paliminimal.tex
xindy -M texindy -M utf8.pl-paliminimal.in-test -M bonus mal-paliminimal.idx
lualatex mal-paliminimal.tex
If you would like to check out the vector versions, the PDF files can be downloaded as follows:
wget http://striz7.fame.utb.cz/tex-sx/pi/typeset-paliminimal.pdf
wget http://striz7.fame.utb.cz/tex-sx/pi/mal-paliminimal.pdf
This is the content of the second TeX file and a preview of both versions is enclosed. The general version can be seen on left, the minimal version is the column of words on right.
% run: lualatex or xelatex mal-paliminimal.tex
% uncomment line 9 or 10, respectively
%
%lualatex mal-paliminimal.tex
%xindy -M texindy -M utf8.pl-paliminimal.in-test -M bonus mal-paliminimal.idx
%lualatex mal-paliminimal.tex
%
\documentclass[a4paper]{article}
\usepackage{luatextra} % for lualatex engine
%\usepackage{xltxtra} % for xelatex engine
\pagestyle{empty}
\usepackage{makeidx}
\makeindex
\usepackage[colorlinks]{hyperref}
\usepackage{filecontents}
\begin{filecontents*}{bonus.xdy}
(markup-letter-group :open-head "~n \textbf{" :close-head "}")
\end{filecontents*}
\begin{document}
Regular text.
\index{pāḷi}
\index{ānāpānasati}
\index{saṃsāra}
\index{ñāṇavimutti}
% Pali Buddhist Dictionary
\index{insight}
\index{paññā}
\index{vipassanā}
\index{ñāṇa}
\index{nirvāṇa}
\index{nibbāna}
\index{permanency}
%\index{vipallāsa}
%\index{personality}
%\index{sakkāya}
%\index{diṭṭhi}
% sortedCanonList.txt
\index{aādiparokkhāyañca}
\index{sammāyojitajālavāta}
\index{ṃsati}
\index{ḷukhappa}
\index{homaidaṃ}
\index{sumanosmīti}
\index{sāmākanīvāre}
\index{vīriyārambhassa}
\index{jeṭṭhasissa}
\index{ñattikammavācāpi}
\index{ñattikammaṃ}
\index{ārammaṇamariyādā}
\index{aāgamo}
\begingroup
\def\thispagestyle#1{}
\printindex
\endgroup
\end{document}

utf8.pl-paliminimal.in.xdyandutf8.pl-paliminimal.in-test.xdy? – tatojo Nov 06 '18 at 12:42