4

In chapter 8 "The Characters You Type", the TeXbook (20th printing, Addison-Wesley 1991) describes TeX's tokenizer, whose input is a stream of characters, and whose output is a stream of tokens. The tokenizer is described as a state-machine with three states:

  • N - Beginning a new line
  • M - Middle of a line
  • S - Skipping blanks

Near the end of this chapter (on p. 47), it is described how the primitive \input is dealt with by the tokenizer:

If TEX has nothing more to read on the current line, it goes to the next line and enters state N. However, if \endinput has been specified for a file being \input, or if an \input file has ended, TEX returns to whatever it was reading when the \input command was originally given.

It is unclear to me:

  1. what state change, if any, happens when TeX starts reading an \input file.

  2. what state change, if any, happens when TeX resumes reading the main file after finishing reading the \input file.

  3. whether the end of the \input file is considered the end of the last line of that file.

Johannes_B and Joseph Wright gave good answers to questions 1 and 3, but the answer to question 2 is still not clear to me. Both Johannes_B and Joseph Wright claim that when the input file has been read, the tokenizer reverts to its former state. But at what point was this former state established? Before or after the \input primitive was read? Before or after the file name was read? Before or after the spaces after the file name were consumed?

Evan Aad
  • 11,066
  • 3
    I have the feeling it would be more efficient to read the documented TeX code and the texbook rather than asking the different questions here and people that have read those documents answer. – Johannes_B Aug 27 '17 at 10:17
  • 4
    I'm wondering what you need to actually find out here: what issue have you run into with the \input primitive that prompts this question? (I can't think of an obvious one ...) – Joseph Wright Aug 27 '17 at 10:23
  • 1
    @JosephWright: The issue I ran into was that I failed to understand how TeX's tokenization algorithm handled the \input primitive, as I was reading chapter 8 of the TeXbook. – Evan Aad Aug 27 '17 at 10:28
  • I just saw how the parens while opening and closing a file are created. I also saw state:=new_line;. Now i feel like i am in the matrix. – Johannes_B Aug 27 '17 at 10:41
  • @Johannes_B: Are you saying that the state is set to N just before the beginning and just after the end of the \input file? – Evan Aad Aug 27 '17 at 10:45
  • If i understand what i just read correctly: 1) state changes to new line. 2) goes back to the state it had before (\input does not have to be on a line of its own) 3) the end of the file is considered as the end of the file, even if it is not. \endinput can be before the end of the line, but TeX doesn't bother. It closes the file. and resumes its previous state. – Johannes_B Aug 27 '17 at 10:46
  • @Johannes_B: OK, thanks. I need to mull over this information. Either it makes sense, or else hopefully I'll be able to come up with an example to help pinpoint what I don't understand. – Evan Aad Aug 27 '17 at 10:49
  • You really should have a look at http://ftp.uni-erlangen.de/ctan/systems/knuth/dist/tex/tex.web to get the information first hand. – Johannes_B Aug 27 '17 at 10:54
  • Let us continue this discussion in chat. – Evan Aad Aug 27 '17 at 15:56

2 Answers2

8

When the \input primitive is encountered, TeX starts reading the new file and starts a new line for the beginning: thus the state changes from 'whatever it was' to N. It then proceeds under the normal rules until the end of the file (or \endinput). TeX will then return to whatever state it was in before.

In most cases, this is all somewhat academic as \input is typically used on a line on it's own with 'well-behaved' source files. Thus the outcome is the same as if the lines of the file input were simply copy-pasted into the main source. If we deliberately arrange that this is not the case we can do tests to show that the behaviour is as described. For example, with file main.tex reading 

\def\world{world}
\def\bar{bar}
Hello \world \input foo \bar
\bye

and foo.tex reading

foo

we get output

Hello worldfoo bar

On the other hand, if we change foo.tex to read

foo%

then we get output

Hello worldfoobar

This shows that as described TeX has returned to skipping spaces after the end of the \input primitive. In the first case, there is a newline at the end of foo.tex which turns into a space, and so shows up. In the second case we don't have that space, and the one after \input foo is ignored so we don't see it. Notice that that space does delimit the file name picked up by \input: without it we need a `\relax.)

If the input file contains \endinput then all lines after that one are ignored, but not anything on the same line but after \endinput. Again, altering our demo foo.tex to

foo \endinput baz%
oops

we get

Hello worldfoo bazbar

or if we omit the %

Hello worldfoo baz bar

Joseph Wright
  • 259,911
  • 34
  • 706
  • 1,036
  • Thank you very much! I'm always enlightened by your answers. A couple questions, please. Firstly, you wrote: "In the first case, there is a newline at the end of foo.tex which turns into a space, and so so shows up." However, to go by appearance, the file foo.tex ends abruptly, without a newline. So can I infer that TeX treats the last line of the file as though it ends with a newline, even if this isn't so? – Evan Aad Aug 27 '17 at 12:45
  • @EvanAad Remember that from the macro layer point-of-view all lines end in a newline char unless they are deliberately cut off with %. TeX works on a line-by-line basis: the macro layer never sees the 'raw' file at all (remember some operating systems use(d) fixed-length line records with no explicit newline char at all). – Joseph Wright Aug 27 '17 at 12:48
  • I see. So this answers my 3rd question in the affirmative. Thanks. Secondly, you wrote: "TeX will then return to whatever state it was in before." Before what? Before the \input primitive was read, or before the scanner switched attention from the main file to the \input file (so, after the \input file's name has been read)? – Evan Aad Aug 27 '17 at 12:57
  • Why is the space after the \input foo ignored in the second case? – Evan Aad Aug 27 '17 at 14:44
  • @EvanAad Hello \world \input foo \bar the space after foo is eaten here as it terminates the filename. There is also no space between the \world control sequence and the first string of file foo which is string foo. The space in the first example is the end of the line space of the inputted file called foo. In the second example, that line is cut off with the % sign. I have to say though, the names could have been a bit more distinctive. – Johannes_B Aug 27 '17 at 14:56
  • @Johannes_B: Why is a space after a filename eaten? – Evan Aad Aug 27 '17 at 15:03
  • @EvanAad Try out without that space. – Johannes_B Aug 27 '17 at 15:04
  • From the answer: "Notice that that space does* delimit the file name picked up by \input: without it we need a \relax."* – Johannes_B Aug 27 '17 at 15:19
  • @Johannes_B: OK. But this eating of spaces after a file name is not a result of the rules of tokenization described in chapter 8, right? So it must be caused by the actions of the \input primitive itself, which the tokenizer is not aware of, right? – Evan Aad Aug 27 '17 at 15:19
  • @EvanAad Each control sequence terminates with a space or a non-letter. Compare \LaTeX is cool. The space between LaTeX and is will be gone. – Johannes_B Aug 27 '17 at 15:21
  • Let us continue this discussion in chat. – Evan Aad Aug 27 '17 at 15:22
0

The following is meant to complement Joseph Wright's answer by offering a clear answer to question no. 2 in the original post, namely what state change, if any, happens when TeX resumes reading the main file after finishing reading the \input file. This answer is speculative, in that I don't base it on the source code, the TeXbook, or any other authoritative source, but on my own thoughts and observations. It is essentially what Joseph Wright wrote in his answer, except I don't claim that once the input file has been read, TeX will "return to whatever state it was in before". Other than that, it's basically a restatement of his answer.

I will assume a grammatically correct TeX manuscript. I will also assume that the input filename is specified explicitly rather than via a macro, and that it is ended with whitespace or some non-expandable control-sequence.

When the TeX "stomach" receives the token \input from the tokenizer, the characters that gave birth to this token have been consumed, and so has the following whitespace, so that the next character on the input stream marks the file name's first character.

Assuming the token \input has the default value (i.e. assuming it has not been \let to some other value), the next thing to happen is that the file name will be consumed, until the next character on the input stream will be whitespace, or some non-expandable control-sequence.

The tokenizer will then switch to state N (as pointed out in Joseph Wright's answer), the current input stream will be cached, and then the current input stream pointer will be directed to the input stream engendered by the input file.

When the latter input stream has been consumed, the tokenizer's state will change to S, and the current input stream pointer will be redirected to the cached input stream. Thus, any whitespace following the original input file name will now be consumed (as pointed out in Joseph Wright's answer).

Evan Aad
  • 11,066