13

When I use an input encoding to (La)TeX that allows me to directly input characters such as é and ä (such as by declaring \usepackage[utf8]{inputenc}), how is a token defined? Specifically,

  • is it possible to write things like \^ä; and for this case, would it matter to (La)TeX whether I represented this character
    • as U+00E4 or
    • with the combining sequence U+0061 U+0308 ,
  • or must I use \^{ä} ?

(This question was inspired by a comment from user @doncherry to this question on accented characters.)

Community
  • 1
Lover of Structure
  • 22,025
  • As a further question I'd like to add: What is the definition in Xe/LuaTeX? – Caramdir Jul 18 '12 at 01:28
  • @Caramdir It depends from which packages you load. If you don't load xunicode (maybe via fontspec), there's nothing special in the characters the engines see. And of course, they won't be printed in most cases, as they correspond to inexistent font positions. – egreg Jul 18 '12 at 09:24

1 Answers1

10

Standard (pdf)LaTeX based on pdftex is 8-bit orientated.

If you use an 8-bit encoding like ansinew, ä is encoded as 11100100. This means it is 8 bits long and so for pdftex it's a single input "entity". In this case \^ä can work (if you use inputenc, ä is a command and what happens depends on the font encoding which sets the actual definition of ä).

If you use utf8, ä is normally encoded as 1100001110100100. So it is at least 2x8 bits long and pdftex sees two input "entities". As inputenc makes the first bit active and so ä is actually a rather complicated command, a combination like \^ä will normally break.

The engines XeTeX and LuaTeX are unicode orientated.

For them a UTF-8 encoded U+00E4 ä is a single entity. With these engines there is no longer a difference between "normal" ASCII and "special" non-ASCII chars: ä will be handled like an a and \^ä will work fine. If the ä is encoded as U+0061 U+0308 then there are also for these engines two input entities. A command like \^ä will normally not break in such a case but the output could be different from \^{ä}. Again what happens depends on font and on the actual definition of \^ (normally set by xunicode with these engines).

Alan Munn
  • 218,180
Ulrike Fischer
  • 327,261
  • 1
    I'd add that the U+0061 U+0308 combinations are not supported by pdflatex – egreg Jul 18 '12 at 13:56
  • @egreg Where does this restriction derive from? Are all Unicode combining sequences unsupported by pdflatex? – Lover of Structure Jul 19 '12 at 08:56
  • 1
    @user14996: Roughly inputenc handles utf8 by making the first octett active (a command) and this commands grabs the following pieces as arguments to build the char. But inputenc can't make ASCII chars like U+0061 active. This would break all command names with an a in it. So it would have to define U+0308 so that it looks back, removes the already printed a and replace it by something else: That's not easy. – Ulrike Fischer Jul 19 '12 at 09:05
  • 2
    @user14996 pdftex only works with "one-byte" input. The support for multibyte UTF-8 characters works by looking at the first byte, which bears the information on how many bytes the character is made of, and performing appropriate actions. Supporting also combining characters would require a look ahead that's quite complicated to accomplish in full generality. This is an inherent limitation of eight bit engines, unfortunately. – egreg Jul 19 '12 at 09:10