Why does a newline character still disappear even with other/active catcode set?

Question

For example, with this code

%! TEX program = lualatex
\documentclass{article}
\begin{document}
\ExplSyntaxOn
\char_set_catcode_other:N %
\tl_set:Nn\var{%}
\tl_analysis_show:N\var
\ExplSyntaxOff
\end{document}

The output is

The token list \var contains the tokens:
>  % (the character %).

as expected, and it also works with all the other special characters. However when I try it with the newline character (which is encoded with the byte with value 10 on UNIX)

\char_set_catcode_other:n {10}
\tl_set:Nn\var{
}
\tl_analysis_show:N\var

the output is

The token list \var is empty

Why is that happening? (also ^^J or ^^M results in the same output, while a literal newline results in the output below)

The token list \var contains the tokens:
>    (the character  ).

See also: tex core - New lines and TeX: difference between ^^J and ^^M - TeX - LaTeX Stack Exchange — user202729, Dec 27 '21 at 03:01
Also: for the same reason, \<literal new line character> is not the same thing as \^^J or \^^M. The latter is useful when e.g. you want to have the escape sequence for passing to \string or \regex_replace_all:NNn to replace something with literal new line character. — user202729, Dec 28 '21 at 09:14
See also: spacing - Why doesn't a trailing tab typeset? - TeX - LaTeX Stack Exchange – caused by the same reason. — user202729, Dec 30 '21 at 08:03

score 6 · Accepted Answer · answered Dec 15 '21 at 15:17

Inside the scope of \ExplSyntaxOn the value of \endlinechar is set to 32 (space) and the category code of the space is set to 9 (ignored).

Thus your code is equivalent to \tl_set:Nn \var {} and the token list is indeed empty.

Setting the category code of the Unix newline (character 10) is irrelevant, because this character appears nowhere: when TeX reads a record (a line of input) it immediately discards the end-of-record operating system specific character and replaces it with the \endlinechar.

Why does a newline character still disappear even with other/active catcode set?

1 Answers1

Linked