5

I am having trouble creating an hyphenation dictionary for the Khmer language with patgen. I keep getting errors like, "Bad representation" and "Bad character" but I am not sure what I am doing wrong. Khmer is UTF-8.

Part of my khmer.dic (I tried UTF-8 on the first line but that didn't help):

ខិត-ខំ
ប្រឹង-ប្រែង
យក-ចិត្ត-ទុក-ដាក់
ព្រះយេស៊ូវ-គ្រីស្ទ
កណ្ឌ-គម្ពីរ
សញ្ញា-ថ្មី

But I'm not sure what to use for the translation file. I've seen this tutorial as well as read this and this but I still can't figure out what to do. Can anyone give me more specific instructions?

Khmer doesn't have upper or lower case (all the same), so I am not sure what to do with the translation file (khmer.tra). Should I include all the Khmer alphabet? Here's what I have now:

 2 3
 ក
 ខ
 គ
 ឃ
 ង
 ច
 ឆ
 ជ
 ឈ
 ញ
 ដ
 ឋ
 ឌ
 ឍ
 ណ
 ត
 ថ
 ទ
 ធ
 ន
 ប
 ផ
 ព
 ភ
 ម
 យ
 រ
 ល
 វ
 ឝ
 ឞ
 ស
 ហ
 ឡ
 អ
  ា
  ិ
  ី
  ឹ
  ឺ
  ុ
  ូ
  ួ
  ើ
  ឿ
  ៀ
  េ
  ែ
  ៃ
  ោ
  ៅ
  ំ
  ា
  ះ
 ឥ
 ឦ
 ឧ
 ឨ
 ឩ
 ឪ
 ឫ
 ឬ
 ឭ
 ឮ
 ឯ
 ឰ
 ឱ
 ឳ
 ឲ
  ្

I am using the command: patgen khmer.dic khmer.pat khmer.log khmer.tra in Ubuntu and using (even though I don't fully understand what these are for): hyph_start: 1 hyph_finish: 2 pat_start: 2 pat_finish: 4 good weight: 1 bad weight: 1 threshold: 1

Community
  • 1
Nathan
  • 271
  • i think it highly unlikely that patgen would be able to support utf-8 without modification. consider this: patgen was created in the early 1980s. utf-8 didn't appear until at least 1990. this mail message may or may not be accurate, but the dates make sense, and the message itself makes for amusing reading: https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt – barbara beeton Oct 08 '14 at 16:00
  • Thanks @barbarabeeton - Do you know of any program that can automatically generate of hyphenation rules (I want to use it with Hunspell) that would work with UTF-8? Quite an interesting story there! – Nathan Oct 09 '14 at 00:50
  • A nice tutorial written by Yannis Haralambous may be found at https://ctan.org/pkg/patgen2-tutorial – LEo Jan 19 '22 at 21:51

2 Answers2

6

First of all, whatever you are going to achieve, it won't work with ‘classical’ TeX. This is due to a design decision of Knuth – today we know that this was unfortunate, but at the time of writing TeX this was far less obvious: Hyphenation patterns are applied to glyph indices and not to input character codes. Since there are more than 256 Khmer ligature glyphs, the standard hyphenation algorithm can't be applied.

Today, this design problem can be circumvented natively by luatex only, which allows manipulation of the input data stream by applying lua programs or filters. I don't know whether someone has written the necessary lua code of such ‘early’ hyphenation already, however.

Now back to your problem. The patgen program is completely agnostic of what it processes; the only limitation is that it cannot handle more than 243 entities: The 8bit range of 256 characters minus the digits 0-9 and characters ‘.’, ‘-’, and ‘*’ (which can be mapped to different characters if necessary). Since the number of Khmer characters is less than 128, patgen can be used to create patterns.

First, you have to define an ad-hoc mapping from Unicode to an 8bit encoding (and providing converters from and to this mapping). A simple choice might be to simply strip off the high byte from the Unicode values, this is, mapping U+1780-U+17FF to 0x80-0xFF. Then set up the translation file – exactly in the same was as you tried above with Unicode, but this time using the 8bit encoding –, run patgen, and convert back.

You might look up the ‘wortliste’ repository for more information on how to set up shell scripts and the like to facilitate the process of pattern creation:

http://repo.or.cz/w/wortliste.git/tree

Of particular interest could be the script skripte/make-full-pattern.sh.

lemzwerg
  • 599
  • Thanks. Still a little over my head, but I think I am starting to get the picture. I also found this for Thai: http://www.ctan.org/tex-archive/language/thai/thailatex/hyphen which looks like they might have done something similar with conv-utf8-hex.sed – Nathan Oct 09 '14 at 07:35
  • Note that Thai is different since all necessary glyphs nicely fit into the available 8bit range of TeX, so you can have native hyphenation support for this language. – lemzwerg Oct 09 '14 at 16:45
2

I used patgen for Georgian. Georgian UTF-8 is 3 bytes long. so when patgen asks for Number of chars I use 3*N1, 3*N2 Where N1 and N2 are for one byte chars what patgen expects :) I had no succes with opatgen (unicode patgen)/was unable copile it for windows. This is nice program and library. Unfortunately not updated long time/

another way is to convert your words(actually chars) to 1 byte encoding , than use patgen as described and convert generated patterns back to utf-8. But as i mentioned way above worked for me. after that I just converted generated paterns for 1byte encoding for georgian tex (T8M). Those 2 sets of generated patterns work for 1byte and utf-8 respectivelly. pattern loading is handled with hyph-utf8 package

Levan Shoshiashvili
  • 916