2

What's the point of using both fontenc and inputenc in the same preamble? Why would anyone want to specify an input encoding (with inputenc) and an output encoding (with fontenc) that don't coincide? And if they do coincide, why use both?

This question is not a duplicate of this one. In the answer given to that question two scenarios are described:

Scenario 1

  1. When you type ä in an editor set up for Latin-1, the machine stores character number 228.

  2. When TeX reads the file it finds the character number 228 and the macros of inputenc transform this into \"a.

  3. Now fontenc comes into action; the command \" has an associated table of the known accented characters the font has available, and ä is among these, so the sequence \"a is transformed into the command "print character 228" in the current (T1-encoded) font.

In this case the two coincide.

Scenario 2

for instance, of ß:

  1. The machine stores character number 223

  2. The macros of inputenc change this into \ss

  3. fontenc transforms this into "print character 255" (where T1 encoded fonts have a ß character).

In the first scenario what's the point in specifying both inputenc and fontenc if they coincide (e.g. they both associate the character number 228 with the string \"a.

In the second scenario, why would you want to convert the character number 223 to the character number 225? Why can't you stick to 223?

Community
  • 1
Evan Aad
  • 11,066

5 Answers5

10

OK I'll try to take a different slant than the answers in the duplicate question.

A document (let's say it is saved in UTF-8)

\documentclass{article}

%\usepackage[utf8]{inputenc}
%\usepackage[T1]{fontenc}

\begin{document}

Söméţhìñĝ

\end{document}

This produces

enter image description here

Because tex does not decode the UTF-8 input and all the accented letters make multi-byte sequences using characters above 127 and the default font encoding (OT1) has no characters in those slots so the log says

Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!
Missing character: There is no � in font cmr10!

Note the number of missing character relates to the number of bytes in the the internal UTF8 representation.

If we use inputenc to declare the encoding

you get

enter image description here

which looks OK but if you have a long text you will find that (a) hyphenation is not working and (b) you can not search for this text in the resulting pdf as the font encoding OT1 has no accented letters the ö is rendered as a normal o with an accent " positioned over it.

If now we declare that we want to use a different font encoding say T1 and uncomment that line then you get

enter image description here

which looks slightly different if you look closely but most importantly it (mostly) uses pre-constructed glyphs in the font so is searchable and hyphenation works.

Note however you can still input (thousands) of characters in UTF-8 input that do not relate to characters in the declared font encoding (which only has 256 slots) so depending on the instructions used latex will either need to "construct" something as it does for OT1 or give an error that the character is not supported or as Ulrike showed you can use different font encodings in the same document for the same input encoding to cover different ranges.

Going back to the original document save it as latin1 (iso-8859-1) encoding 

\documentclass{article}

%\usepackage[latin1]{inputenc}
%\usepackage[T1]{fontenc}

\begin{document}

SöméThìñG

\end{document}

Note I had to drop two characters that can not be encoded in this input encoding even though I could typeset them with the 7-bit OT1 tex encoding above.

enter image description here

Declaring the latin1 input encoding to latex gets us

enter image description here

But as before this is constructed characters so no searching.

If you declare the input encoding to be latin1 and the font encoding to be T1 (which is not the same in general) then

enter image description here

David Carlisle
  • 757,742
8

font encoding and input are quite unrelated. By changing the font encoding you can map the same input to different output:

\documentclass{article}

\usepackage[LSF,LGR,T3,T1]{fontenc}

\begin{document}

K Q N

\fontencoding{LSF}\selectfont 

K Q N 

\fontencoding{LGR}\selectfont

K Q N 

\fontencoding{T3}\selectfont

K Q N

\end{document}

enter image description here

Why not only one encoding?

pdflatex can handle only fonts with 256 chars, so even if you use a large input encoding like utf8 you can't use it on the font side.

And while with lualatex you can have "unicode" on both side and so have much less need for special font encodings: unicode doesn't cover all possible symbols. E.g. from the symbols needed to typeset chess (figures, boards and commentary symbols) only around the half are actually in unicode.

Ulrike Fischer
  • 327,261
  • This doesn't answer my question, it only makes my question more pertinent: why would anyone want to map the same input to different output? – Evan Aad May 18 '17 at 07:55
  • How else would you be able to output e.g. chess symbols or flowers or ducks or greek if you have only ascii as input? (The question doesn't change if you have a larger range of symbols as input like with ut8, there is always something missing). – Ulrike Fischer May 18 '17 at 07:57
  • @EvanAad see my comment to your question for a frequently happening example! – Skillmon May 18 '17 at 07:57
  • I didn't say that you have only ascii as input. I asked why you need to specify two different encodings: one for input and one for output. Why can't you specify one encoding which will be good for both input and output? Under what circumstances would you want the output generator to interpret your file differently to the input parser? – Evan Aad May 18 '17 at 08:01
  • @EvanAad how would you propose to tell your editor to use the encoding used in computer modern as its input (and how then would you input accented characters as computer modern font has none) – David Carlisle May 18 '17 at 08:12
  • @DavidCarlisle: If the computer modern font has no accented characters and you wish to use accented characters, why would you specify it? – Evan Aad May 18 '17 at 08:15
  • @EvanAad: I added a comment regarding the "one encoding". – Ulrike Fischer May 18 '17 at 08:19
  • @EvanAad because that is the way computers work. No font has every character. – David Carlisle May 18 '17 at 08:25
  • @DavidCarlisle: But you're going to typeset your document somehow, right? So the font that you're going to use for that will have all the characters that you're interested in, won't it? So use this font as the input and output encoding. – Evan Aad May 18 '17 at 08:27
  • @EvanAad You'd then need separately-encoding font files for each input encoding. – Joseph Wright May 18 '17 at 08:32
  • @EvanAad: My example above uses 4 different fonts. So no, "the" font doesn't need to have all characters as a document doesn't use only one font. – Ulrike Fischer May 18 '17 at 08:32
  • @EvanAad No why do you assume the font has all the characters – David Carlisle May 18 '17 at 08:34
  • @UlrikeFischer: Your example feels contrived to me. If I really wanted to generate the four rows of glyphs and there was no single font encoding that allowed for all of them, then I would specify a suitable input encoding for every row, not one inadequate input encoding at the top of the file and then four different output encodings for every row. It doesn't make sense to me, except as a kind of 'party trick' or 'hack', not if you want to write readable and reasonable code. – Evan Aad May 18 '17 at 09:14
  • A change of input encoding wouldn't change the need to use four fonts. I can above use utf8-encoding and use ♔ as input. I would still have to change to a chess font and to tell pdflatex on which position the king is to print it. – Ulrike Fischer May 18 '17 at 09:20
  • 1
    Btw: I personally wouldn't want to have to use ♔ in the input of a chessgame. It may look good but is a pain to type. "K" is much faster and easier, so I'm quite gratefull that one can remap the input to different glyphs. – Ulrike Fischer May 18 '17 at 09:32
8

Very simple.

1. No inputenc and no fontenc

Let's assume first you don't load inputenc.

If you don't load fontenc, the fonts are chosen following the OT1 encoding; upon finding ß in the input there are two main scenarios:

  1. the file uses a single byte encoding (such as Latin-1)
  2. the file uses UTF-8

In case 1, you'll get the message Missing character ^^df; in case 2 the message will tell you that two characters are missing, namely ^^c3 and ^^df.

2. With inputenc and no fontenc

If you load inputenc with the option corresponding to the encoding of the file, ß will be mapped to \ss. The OT1 encoding definitions will choose the right character to output. On the other hand, hyphenation of words with accented characters will usually be wrong or massively incomplete. For French and German it would be a stopper.

3. No inputenc, but load fontenc

Suppose you load \usepackage[T1]{fontenc} and no inputenc. Again, two scenarios like before. In the case of Latin-1, ß will print ý; in the case of UTF-8, ß will print Ãý.

4. With both inputenc and fontenc

ß will print ß

Conclusion

Which among the four options is good?

egreg
  • 1,121,712
  • 1
    answer to the question in the conclusion, option 1 and write in English? – David Carlisle May 18 '17 at 08:42
  • In scenario 2 you wrote: "ß will be mapped to \ss". What do you mean by '\ss`? The string consisting of the three characters BACKSLASH, ESS, ESS or some number (a byte) that TeX associates with the escape sequence '\ss'? – Evan Aad May 18 '17 at 08:56
  • 1
    @EvanAad The token (control sequence) \ss, which is the LICR for Eszett. – Joseph Wright May 18 '17 at 08:57
  • In scenario 2 you wrote that "hyphenation of words with accented characters will usually be wrong or massively incomplete". Why is that? – Evan Aad May 18 '17 at 08:59
  • @EvanAad The control sequence \ss, of course. For the problem with hyphenation, see https://tex.stackexchange.com/questions/725/ – egreg May 18 '17 at 10:31
7

Files, both for input and for fonts, have a limited set of slots available which can be addressed. Classically (TeX82), both were 7-bit so allowed 128 separate slots in any one file. TeX90 is 8-bit so extends this to 256. So fonts encoded for TeX use may have either 128 or 256 separate glyphs available (ignoring empty slots) in any one file. To use more than 256 symbols we therefore need to pick more than one font file: Ulrike's answer demonstrates this nicely.

In the input file, again if we start from a 7- or 8-bit situation we need to be able to make best use of the slots available. There are a range of 8-bit input encodings which have been used over time, often related to the language(s) they are used for. However, these encodings have been developed separately from the font encodings used to place glyphs in output. As such, there are two stages we need to carry out:

  • Read the input and covert from the input encoding to some fixed internal representation (this is called the LICR in LaTeX).
  • Select the correct output glyph for each LICR point, taking account of the fact that there can be more LICR slots than font slots (LICR uses what we can think of as 'descriptions' of slots, so is not limited to a fixed number of positions).

As the two stages are separate and essentially independent, the approach of having two separate LaTeX packages is appropriate. Note that for stability reasons, LaTeX2e with pdfTeX always starts from the assumption of ASCII input encoding and a 7-bit output font encoding.

Notably, the encodings which have been widely used are stable: no changes in input or output positions can be made. This is essential if documents/data are to continue to be usable. In particular, the original Knuth 'OT1' encoding is somewhat 'interesting', whilst T1 does not entirely align with the Latin-1 input encoding structure.

Some confusion may arise as with Unicode the situation is somewhat less acute. Unicode specifies a large number of available slots with a fixed set of meanings, and these map directly between input and output for a the large number of cases that have been codified. As such, for a Unicode TeX engine used with a Unicode font, the need for inputenc/fontenc is removed (largely). However, even with this large range of standardised positions there are still cases that are not covered: one still needs mechanisms to deal with 'non-standard' or 'new' meanings. (Unicode is also 'stable' in the sense that some less-than-ideal decisions cannot be reversed as the encoding must remain usable into the future.)

A separate but important point here is that although we have Unicode engines nowadays, pdfTeX (8-bit) is very stable and many workflows are perfectly acceptable using it. This means that whilst for some cases (say mixing Cyrillic and English) one could use a Unicode engine and not need separate encodings, there are many reasons that users prefer to stick to tried-and-tested methods. (Font availability in particular is not a certainty for Unicode work: no font covers the entire range and many very good fonts are available only in classical TeX format.)

Joseph Wright
  • 259,911
  • 34
  • 706
  • 1,036
  • Thanks. This answer is quite clear. One thing, though. You wrote that "the two stages are separate and essentially independent", but if I understand correctly, while they may be separate, they're anything but independent, because the input encoding must translate the input file to a sequence of tokens that would make sense to the output encoding. – Evan Aad May 18 '17 at 09:55
  • 1
    @EvanAad They are essentially independent in LaTeX as the path input-> LICR - > output goes via a fixed intermediate step. Conversion of input to LICR needs no details of the output encoding, and vice verse. – Joseph Wright May 18 '17 at 10:23
6

If I understand your final question correctly, you're asking why we couldn't use the same "numbers" everywhere, for both input encoding and font encoding.

The short answer is that the two sets to which we want to give "numbers" are different:

  • A font is a collection of glyphs (shapes): it describes appearance. It's up to the font designer to put any shape at any position, though they'll usually be sensible and try to follow certain conventions (among many) where possible. With fontenc, you tell TeX where to look up certain shapes it wants to find.

  • Your input is a collection of characters, seen by TeX as a stream of bytes. With inputenc, you tell TeX which (sequences of) bytes in your input correspond to which characters.

Examples of input characters: The lowercase letter A (looks like a), the space character, the newline character, the Greek letter pi (looks like π), the Devanagari letter ka (looks like क), the Angstrom sign (looks like Å), the letter A with a dotted circle (looks like Å).

Examples of shapes: Anything you can draw.

How do they differ? Let me count the ways…

A font contains shapes, not characters.

  • A font may contain multiple shapes for a character (e.g. different ways of writing lowercase a).
  • A font may contain the same shape for multiple characters (e.g. Å and Å).
  • A font may contain shapes for non-characters (e.g. "the bowl to the left of the vertical bar in the Devanagari letter क").
  • A font may omit shapes for a character (e.g. the newline character). And of course any font will omit most characters and support only a very limited range.

In short: there's no unique "number" that can be given to both characters and shapes, because the two sets are different and cannot be mapped to each other.

However, there are limited cases where font encodings and input encodings may agree:

  • if you use (say) only characters A–Z, a–z, 0–9 (and a few more), then indeed the input convention can agree to put the letter A at position 65 (this is called ASCII), and the font convention can agree to put its shape A at position 65 as well. This already happens by default and you don't have to specify anything.

  • With Unicode (which currently has over 120,000 characters), quite a large number of characters have been given numbers, and fonts that contain only shapes for those characters can agree to use the same numbers. This would mean that a font designed for the Brahmi script (and which contained shapes precisely for individual characters) would have to put its characters in positions 69632 to 69743. Such large "position numbers" work only with Unicode-aware engines XeTeX and LuaTeX. Instead if the font simply used positions (say) 10 to 122, it could potentially work even without those engines.

In all other (and typical) cases, fonts and characters simply cannot be given the same numbers. Or simply happen not to have been given the same numbers. If you always insisted that they had to use the "same" encoding (over whatever limited domain such a statement is meaningful), you would be able to use only certain fonts with certain input encodings: for instance if you changed the way you input ä, then in your hypothetical world you may suddenly find yourself needing to use a different font. This would be silly.

ShreevatsaR
  • 45,428
  • 10
  • 117
  • 149