Why are my newlines turned into \Omega?

Question

If I use ^^J or \iow_newline:, it prints a \Omega instead of a newline. Any idea what I got wrong?

\documentclass{article}
\ExplSyntaxOn
\NewDocumentCommand{\printTwiceWithNewline}{v}{
  \str_clear_new:N \l_test_str
  \str_gput_right:Nx \l_test_str {\tl_to_str:N{#1}^^J\tl_to_str:N{#1}}
  \str_gput_right:Nx \l_test_str {\iow_newline:\tl_to_str:N{#1}}
  \begin{flushleft}\ttfamily
    \l_test_str
  \end{flushleft}
}
\ExplSyntaxOff
\begin{document}
\printTwiceWithNewline{Test}
\end{document}

@egreg but I'm missing the alpha's at the begin of the lines ;-) — Ulrike Fischer, Aug 30 '23 at 13:15
@egreg ?? How come a newline is expected to be displayed as \Omega?? What if I have the (maybe weird?) will to display newlines as newlines? ^^' — tobiasBora, Aug 30 '23 at 13:16
I’m afraid you’re misunderstanding what ^^J is for. And this looks very much like an XY-question. — egreg, Aug 30 '23 at 13:35
If you have \documentclass{article} \begin{document} ^^J \end{document} you'd get upper case omega as the output. L3 is not really part of the "problem". — Willie Wong, Aug 30 '23 at 13:43
@egreg So basically this is supposed to be a debug function to write exactly the verbatim content of a latex string in a document. This string might contain any text, including python code etc… So using \par instead of ^^J would likely forbid that file from being compiled in python. So do you suggest I should replace all occurrences of ^^J with \\ before printing? If so is there any other characters that might be poorly printed? — tobiasBora, Aug 30 '23 at 14:15
@DavidCarlisle Do you mean I should replace ^^J with \par when printing the string only? — tobiasBora, Aug 30 '23 at 14:19
@tobiasBora it is not clear what is your intention of adding ^^J above, ^^J is set up in latex to give a line break in \typeout but it is never used anywhere else. — David Carlisle, Aug 30 '23 at 14:32
@DavidCarlisle So I basically want \l_test_str to contain a string with newlines (e.g. python code) that I should be able to write to some files to compile later. How \l_test_str is created does not really matters (here I just manually add ^^J that should hopefully write a \n character in the string, but it might come from different sources). Then, I also want to be able to print in LaTeX the python code that I will run: for this I need to be able to print the newlines in \l_test_str as real new lines. — tobiasBora, Aug 30 '23 at 14:36
No that is mistaken: a real character 10 in the pdf will give you an omega, you need to the text to be printed on two lines in the pdf (by using \par) so your pdf reader if extracting that as plain text will use \n characters. You do not want \n characters in the pdf. — David Carlisle, Aug 30 '23 at 15:03
this is what happens with a classic verbatim environment: every line is marked by \par. — David Carlisle, Aug 30 '23 at 15:05

Willie Wong · Accepted Answer · 2023-08-30T14:53:07.173

4

To address your needs, basically you want to be able to change ^^J to \par between the two uses. An easy way is to define a temporary stand-in macro for the would-be line break. Redefine it before writing to file, and redefine it before printing in PDF.

Here's a LaTeX2e example:

\documentclass{article}
\newcommand\fakeLF{}
\newcommand\teststring{Test\fakeLF Test\fakeLF Test}
\begin{document}
\renewcommand\fakeLF{^^J}
\message{\teststring}
\renewcommand\fakeLF{\par}
\teststring
\end{document}

For the literal question you asked about Omegas: that is because ^^J generates the unicode 0x000a line-feed character, and with the default OT1 font encoding this happens to generate the upper case Omega character. The behavior is different, for example, if you switch to \usepackage[T1]{fontenc}.

edited Aug 30 '23 at 14:53

answered Aug 30 '23 at 14:44

Willie Wong

24,733
8
74
106

Thanks, but I tried this, but it is not working in my usecase: the string are created via: \ior_str_map_inline:Nn \g_robExt_read_ior {\str_gput_right:Nx \l_robExt_tmp {\tl_to_str:N{##1}\fakeLF }}, but it seems like \fakeLF is replaced with the empty string during the \str_gput_right call. – tobiasBora Aug 30 '23 at 15:23
2

you need a tl not a str to hold a \par token @tobiasBora – David Carlisle Aug 30 '23 at 15:27
Yeah… but I wanted to avoid using tl, exactly because str are more predictable ^^' – tobiasBora Aug 30 '23 at 15:39
Finally I chose a solution where I use tl only when printing as it is easier to do it this way in my code. I also needed to do a special case for spaces, hopefully there are no other edge cases. – tobiasBora Aug 30 '23 at 18:52

campa · Answer 2 · 2023-08-30T17:14:01.833

(Just to explain what is happening.)

The TeX book explains (p. 45)

If the character following ^^ has an internal code between 64 and 127, TEX subtracts 64 from the code; if the code is between 0 and 63, TEX adds 64.

(See also this answer for an explanation of the double caret notation.)

The uppercase J has a character code of 74 (as you can find out by typing \number`J, or looking into an ASCII table), so by the above rule typing ^^J is the same as requiring the character in slot 74-64=10 in the actual font. This clearly depends on the current font encoding:

\documentclass{article}
\begin{document}
\number`J
OT1: ^^J (same as \char10)
\fontencoding{T1}\selectfont
T1: ^^J (same as \char10)
\end{document}

The LaTeX kernel does \newlinechar`\^^J but this has an effect only within \writes.

score 1 · Answer 3 · answered Aug 30 '23 at 18:49

I actually chose the solution of converting to tl the string, and replacing ^^J with \par, and spaces to \ :

\documentclass{article}
\ExplSyntaxOn
\NewDocumentCommand{\printTwiceWithNewline}{v}{
  \str_clear_new:N \l_test_str
  \str_gput_right:Nx \l_test_str {\tl_to_str:N{#1}^^J\space\space\space\tl_to_str:N{#1}}
  \str_gput_right:Nx \l_test_str {\iow_newline:\tl_to_str:N{#1}}
  %%% replace
  \tl_set_eq:NN \l_robExt_tmp_str \l_test_str
  \tl_replace_all:Nnn \l_robExt_tmp_str {^^J} { \par }
  \tl_replace_all:Nnn \l_robExt_tmp_str { ~ } { \  }
  \begin{flushleft}\ttfamily
    \l_robExt_tmp_str
  \end{flushleft}
}
\ExplSyntaxOff
\begin{document}
\printTwiceWithNewline{Test}
\end{document}

Why are my newlines turned into \Omega?

3 Answers3

Linked