XeLaTeX with ucharclasses and Greek Support

Question

ucharclasses and Greek

I find that the package ucharclasses cannot correctly identify all the Unicode Greek.

e.g. if I use

\usepackage{fontspec} \usepackage[Greek]{ucharclasses} \setTransitionsForGreek{\fontspec{Palatino Linotype}}{\fontfamily{lmodern}\selectfont}

and then in the document entered

Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας

The output text would be

Πέτρος ἀ Ἰῦ Χριστοῦ ἐῖ παρεπιδήμοις διασπορᾶ Πόντου, Γαλατίας, Καππαδοκίας, Ἀ καὶ Βιθυνίας

i.e. some missing characters.

Even more weird, from the documentation I found out the Greek actually called Coptic, GreekAndCoptic and GreekExtended, but when I called them out individually, more characters will be rendered (but still missing some).

I tried not using ucharclasses but use \fontspec{Palatino Linotype} directly on those text and it works perfectly.

Any idea why is it? Thanks!

Edit 1

Thanks. I used font family, and I also did a test to show what I mean:

\usepackage{lmodern} \usepackage{fontspec} \usepackage[Greek]{ucharclasses} \newfontfamily\mylatin{Palatino Linotype} \newfontfamily\mygreek{Palatino Linotype} \setTransitionsForGreek{@EnterGreek@ \mygreek}{\mylatin @ExitGreek@}

The output will be this:

@EnterGreek@Πέτρος@ExitGreek@@EnterGreek@ἀ@ExitGreek@πόστολος@ExitGreek@ @EnterGreek@ Ἰ@ExitGreek@ησο@EnterGreek@ ῦ@ExitGreek@ @EnterGreek@ Χριστο@EnterGreek@ ῦ@ExitGreek@ @EnterGreek@ ἐ@ExitGreek@κλεκτο@EnterGreek@ ῖ@ExitGreek@ς@ExitGreek@ @EnterGreek@ παρεπιδήμοις@ExitGreek@ @EnterGreek@ διασπορ@EnterGreek@ ᾶ@ExitGreek@ς@ExitGreek@ @EnterGreek@ Πόντου, @EnterGreek@ Γαλατίας, @EnterGreek@ Καππαδοκίας, @En- terGreek@ Ἀ@ExitGreek@σίας@ExitGreek@ @EnterGreek@ κα@EnterGreek@ ὶ@ExitGreek@ @EnterGreek@ Βιθυνίας@ExitGreek@

Comaring the input:

Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας

Focusing on the 2nd word ἀπόστολος for example, it "exits Greek" after ἀ and treated πόστολος as Latin.

So I actually think it is a bug in ucharclasses.

Edit 2

I'm not sure if it is related to the limitation, rather than a bug, of ucharclasses. From the documentation:

However, be aware that this only “just works” for Unicode blocks. If you are working with typographically overlapping languages, such as combining English and Vietnamese in one document, things get a lot more complex if you want one font for English and another for Vietnamese. Both of these languagese use Latin blocks, so it is inherently impossible to tell which language is intended based on which Unicode block a character in a word belongs to.

However it didn't mention anything about Greek. Any idea?

Edit 3

MWE like this?

\documentclass[a4paper]{article} \usepackage{fontspec} \usepackage[Greek]{ucharclasses} \newfontfamily\mygreek{Palatino Linotype} \setTransitionsForGreek{@Begin@\mygreek}{\fontfamily{lmodern}\selectfont @End@} \begin{document} Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας \end{document} The tags @Begin@ and @End@ are to show the bug I'm talking about.

Edit 4

Thanks for the answer that solve the problem. I didn't included Latin and that is the root of all the problem.

So the correct code looks like this:

The Right Way

\documentclass[a4paper]{article} \usepackage{fontspec} \usepackage[Latin, Greek]{ucharclasses} \newfontfamily\mygreek{CMU Serif} \setDefaultTransitions{\fontfamily{lmodern}\selectfont}{} \setTransitionsForGreek{\mygreek}{} \begin{document} \fontfamily{lmodern}\selectfont \section{Testing Greek Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας} \begin{itemize} \item Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας \end{itemize} testing \end{document}

But I got another problem: the Latin in the section title, Testing Greek, is not bold.

If I use my old code, that gives the bug of not producing the right Greek:

The Old Way

\documentclass[a4paper]{article} \usepackage{fontspec} \usepackage[Greek]{ucharclasses} \newfontfamily\mygreek{CMU Serif} % \setDefaultTransitions{\fontfamily{lmodern}\selectfont}{} \setTransitionsForGreek{\mygreek}{\fontfamily{lmodern}\selectfont} \begin{document} \fontfamily{lmodern}\selectfont \section{Testing Greek Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας} \begin{itemize} \item Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας \end{itemize} testing \end{document}

Then although some Greek are missing, the Latin in the section title, Testing Greek, is now bold.

The Control

A Control test like this: \documentclass[a4paper]{article} % \usepackage{fontspec} % \usepackage[Greek]{ucharclasses} % \newfontfamily\mygreek{CMU Serif} % \setDefaultTransitions{\fontfamily{lmodern}\selectfont}{} % \setTransitionsForGreek{\mygreek}{\fontfamily{lmodern}\selectfont} \begin{document} \fontfamily{lmodern}\selectfont \section{Testing Greek }% Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας} \begin{itemize} \item testing %Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας \end{itemize} testing \end{document}

would gives the correct bold section title.

I also notice other problems in "the Right Way": since when it leaves the Greek Unicode block, it doesn't change back to the default font, then the next item bullet would be in Greek font, etc. i.e. the bullets would have different fonts. In "the Old Way" when I use \setTransitionsForGreek{\mygreek}{\fontfamily{lmodern}\selectfont}, since it always ends with \fontfamily{lmodern}\selectfont, the problem wasn't there.

I would say all these originates from the mystery that the Greek blocks ends somewhere between words, which I still regard it as a bug (See the test in the 1st edit). I have a guess on the possible reason of this mystery. Referring to my original question:

Greek actually called Coptic, GreekAndCoptic and GreekExtended...

May be the breaks are between the Coptic, GreekAndCoptic and GreekExtended.

Edit 5

I think I kind of find a solution although I don't understand why. If I use "the Right Way", without adding Latin in ucharclasses, the Latin in the section title would be bold as usual.

Answer 1

Try the following code:

\documentclass[12pt,a4paper]{article}
\usepackage[Latin,Greek]{ucharclasses}
\usepackage{fontspec}
\newfontfamily\mynormal{Palatino Linotype}
\setDefaultTransitions{\mynormal}{}
\newfontfamily\mygreek{Junicode} 
\setTransitionsForGreek{\mygreek}{}
\usepackage{lipsum}
\begin{document}
    \lipsum[1]
    Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας
\end{document}

Answer 2

Solution 1

Finally I find the answer from this post:

\documentclass{article} \usepackage{fontspec} \usepackage[Greek]{ucharclasses} \newcommand{\mylatin}{\renewcommand\rmdefault{lmr}\renewcommand\sfdefault{lmss}\renewcommand\ttdefault{lmtt}} \newfontfamily\mygreek{CMU Serif} % \setDefaultTransitions{\fontfamily{lmodern}\selectfont}{} \setTransitionsForGreek{\mygreek}{\mylatin} \begin{document} \section{Testing Greek Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας} \begin{itemize} \item Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας \end{itemize} testing \end{document}

I don't understand why \renewcommand\rmdefault{lmr}\renewcommand\sfdefault{lmss}\renewcommand\ttdefault{lmtt} work better than \fontfamily{lmodern}\selectfont, but it did.

And I find out \setDefaultTransitions is not needed if there's no other Unicode exists.

I also find out using Latin in \usepackage[latin, Greek]{ucharclasses} is also not neccessary, if in the 2nd arguement of \setTransitionsForGreek includes whatever I need to put in \setTransitionsForLatin. And I prefer this method since it makes sure whenever it leaves a certain unicode block it switch right back to the default, and things like item bullet will be using the same font.

Now it's almost perfect, but there's still problem handling the transition to Chinese and I don't understand why. May be I should start a new question later.

Lastly, after dozens of hours figuring out how to handle Unicode in LaTeX and still get many mysteries and quirks and still have unsolved problems(The breaking within Greek words, Chinese fonts transition, etc.)... So frustrating

Solution 2

I find that the origin of the problem is that I used newfontfamily, as suggested by some. I dig into the documentation of fontspec and it says that \fontspec{} command are for 1 time use. Now if I define newfontfamily, when I let the ucharclasses to choose it, it then change to that font forever. In order to go back, I orginally used \fontfamily{lmodern}\selectfont which somehow didn't work. The final resolution above is to use \renewcommand\rmdefault{lmr}\renewcommand\sfdefault{lmss}\renewcommand\ttdefault{lmtt}. However, the simplest way is to use fontspec at the very beginning and I don't have to combat with the "going back to default" process. This is what I mean:

\documentclass{article} \usepackage{fontspec} \usepackage[Latin, Greek]{ucharclasses} \setTransitionsForGreek{\fontspec{CMU Serif}}{} \begin{document} \section{Testing Greek Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας} \begin{itemize} \item Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας \end{itemize} testing \end{document}

Solution 3

All the problems orginates from the fact that I want to go back to the default lmodern, and that lmodern doesn't support Greek. I compared lmodern and cm-unicode and found that the differences are pretty minor. If I don't want to deal with the mess mentioned above, and don't mind using a slightly different font than lmodern, then I could just change the default to cm-unicode:

\documentclass{article} \usepackage{fontspec} \setmainfont{CMU Serif} \setsansfont{CMU Sans Serif} \setmonofont{CMU Typewriter Text} \begin{document} \section{Testing Greek Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας} \begin{itemize} \item Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας \end{itemize} testing \end{document}

Answer 3

"Default" means 'minimum required to run the app'; it doesn't mean "the recommended setting". You choose what main font(s) you want to use as a default for your document. Look in the TexMF tree, under fonts. The Latin Modern families of fonts are available as OpenType fonts (under \fonts\opentype\public\lm\). Nowadays, also, Unicode system fonts are available, not just local Tex fonts. For example, Noto Serif contains Latin, Greek and Coptic, and Cyrillic, and that means you don't have to use transition packages to manage these scripts:

(Greek and Coptic scripts go together as a pair, because of their large overlap.)

MWE

\documentclass[12pt]{article} 
\usepackage{fontspec}
\setmainfont{Noto Serif}
\newfontface\fgla{Fedorovsk Unicode TT}
\begin{document}
\section{Testing Greek} Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας
\begin{itemize}
\item Πέτρος ἀπόστολος Ἰησοῦ Χριστοῦ ἐκλεκτοῖς παρεπιδήμοις διασπορᾶς Πόντου, Γαλατίας, Καππαδοκίας, Ἀσίας καὶ Βιθυνίας
\item Петр, Апостол Иисуса Христа, пришельцам, рассеянным в Понте, Галатии, Каппадокии, Асии и Вифинии, избранным,

from:
1-е послание Петра 1 глава – Библия: https://bible.by/syn/46/1/
\end{itemize}

A Glagolitic font in the Tex distribution:

{\fgla \huge Петр, Апостол Иисуса Христа, пришельцам,}


\end{document}

Question here describes the various ways to get back to the fontspec default settings (including Latin Modern; it used to be CM), and also how to switch font encodings if desired.