Based on Ulrike's answer, here is one way to invoke xindy to get it to sort .idx files created by Xe/LuaLaTeX. The trick is to use xindy directly (instead of texindy) and pass the -C utf8 flag.
Minimal Example
\documentclass{article}
\usepackage{luatextra}
\usepackage{makeidx}
\makeindex
\begin{document}
üäö
start
\index{a}\index{b}\index{ä}\index{ü}
end
\printindex
\end{document}
Compilation
lualatex filename.tex
xindy -M texindy -C utf8 -L german-duden filename.idx
lualatex filename.tex
In (pdf)latex you can use UTF-8 encoding and xindy in the following way:
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{makeidx}
\makeindex
\begin{document}
start
\index{a}\index{b}\index{ä}\index{ü}
end
\printindex
\end{document}
And then simply run texindy -L ⟨language⟩ ⟨filename⟩.idx.
In LuaTeX you can also use the luainputenc package to use legacy encodings.
\documentclass{article}
\usepackage{fontspec}
\usepackage[utf8]{luainputenc}
\usepackage{makeidx}
\makeindex
\begin{document}
start
\index{a}\index{b}\index{ä}\index{ü}
end
\printindex
\end{document}
Again, run texindy -L ⟨language⟩ ⟨filename⟩.idx.
Here the result for both examples:

inputenc/fontenc”hacks” to support the encoding. So this won't work. – Caramdir Aug 15 '11 at 16:55.idxfile created bypdflatexisn't actual unicode, but contains LaTeX-encoded umlauts (etc.). In comparison the modern engines create fully unicode-encoded.idxfiles.xindyis supposed to support this, I just don't know how. – Caramdir Aug 15 '11 at 16:58luainputenchack. Still I'd prefer a solution that uses Unicode throughout. While the solution works in this case, something like\index{α}would still cause trouble. – Caramdir Aug 15 '11 at 17:11fontspecbut it doesn't work. – Marco Daniel Aug 15 '11 at 17:12fontspecandluainputencseem to work together for me . – Caramdir Aug 15 '11 at 17:14texindy -C utf8, but this gives an error. Apparently it does work when usingxindydirectly. Could you please edit your answer to make it clearer that the first way is the correct one and the other options should only be used in legacy documents (or when working with(pdf)latex). – Caramdir Aug 15 '11 at 19:52lualatex xindy). – Caramdir Aug 17 '11 at 17:15xindy -M texindy -C utf8 am.idx) but it doesn’t work. I get this error:*** - PROGN: variable TEXINDY.XDY has no value. Can someone tell whats wrong. – Tobi Jun 16 '14 at 20:04