3

In other words,

  • How can I allow breaks after any visible character? ...[such that lines are broken when the sum of the lengths of horizontal boxes equals \linewidth]

  • How can I disable or influence the TeX spacing/line-breaking algorithm to allow for breaks whenever the line is full?

This is a follow-up to a question asked earlier How can I make LaTeX to recognize spaces in my macro (catcode 10)?.

I ran into two problems:

  1. Dealing with the active character
  2. Dealing with memory

The active character causes issues (as expected). David Carlisle mentioned that I could use \noexpand to properly handle the active character. I tried, but to no avail. Furthermore, because the entire text is fed into the macro, the whole thing is stored in memory before it is actually shipped out.

How this thing works:

  1. \treateachletterasword sends its entire input to a scanner (this is the memory hog)
  2. The scanner, \xscan, sends a single token to \xxscan by grabbing a token with \afterassignment
  3. \xxscan tests for catcodes and works accordingly
  4. Finally, \xxscan calls \xscan on the next token (this loop breaks when the equivalent of \relax is encountered-may or may not be the best solution)

Code

(David Carlisle mentioned that instead of \hskip 0pt plus 1sp minus 1sp, I should use \hskip 0pt, or even better, \penalty0 to inprove efficiency. This way, TeX does not need to calculate as much for each line.)

\documentclass{article}
\usepackage{fontspec}


\newfontfamily\monofont{Inconsolatazi4-Regular.otf}% let's pretend this is not a mono font for practice
%\makeatletter % or \catcode"0040=11
\long\def\treateachletterasword#1{\xscan #1\relax}% calls initial \xscan on first char
\def\xscan{\afterassignment\xxscan\let\token= }% assign next single token to \token and send it to \xxscan
\def\xxscan{%
\monofont% apply mono font
\ifx\token\relax\normalfont\else%test for end-of-line or end of group and switch back to normal font
  \ifcat\token\space%
    \token% token is catcode 10
    \spaceskip=.5em% remove glue from space for fixed-width space and precise control
    \xspaceskip=.5em% remove glue from space for fixed-width space and precise control
  \else%
    \ifcat\token\active% deal with active character backslash
      %\noexpand\token
      \textbackslash% for lack of a solution, this macro should be expanded
    \else
      \token\hskip 0pt plus 1sp minus 1sp% add glue to any non-catcode 10 (space)
    \fi
   \fi
\expandafter\xscan%calls subsequent xscan until next char representd by \token = \relax
\fi}
%\makeatother % or \catcode"0040=12
\parindent=0pt % remove firstpar autoindent
\obeylines% insert \par after each end-of-line (^^M)


\begin{document}
\section{SomeTeXFile.tex}\subsection{Typesetting Failure}
\treateachletterasword{
This is XeTeX, Version 3.14159265-2.6-0.99996 (TeX Live 2016) (preloaded format=xelatex 2016.6.27)  11 AUG 2017 13:15
entering extended mode
 restricted \char"005C{}write18 enabled.
 file:line:error style messages enabled.
 \%\&-line parsing enabled.
**ThirdPartyLicenses.tex
(./ThirdPartyLicenses.tex
LaTeX2e <2016/03/31> patch level 2
Babel <3.9r> and hyphenation patterns for 83 language(s) loaded.
(../../DocumentClass.tex (/usr/local/texlive/2016/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/local/texlive/2016/texmf-dist/tex/latex/base/size10.clo
File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
)
\char"005C{}c@part=\char"005C{}count79
\char"005C{}c@section=\char"005C{}count80
\char"005C{}c@subsection=\char"005C{}count81
\char"005C{}c@subsubsection=\char"005C{}count82
\char"005C{}c@paragraph=\char"005C{}count83
\char"005C{}c@subparagraph=\char"005C{}count84
\char"005C{}c@figure=\char"005C{}count85
\char"005C{}c@table=\char"005C{}count86
\char"005C{}abovecaptionskip=\char"005C{}skip41
\char"005C{}belowcaptionskip=\char"005C{}skip42
\char"005C{}bibindent=\char"005C{}dimen102
}
\end{document}

Output

Note that

  1. By adding glue to every non-catcode 10 (except active char) and removing glue from every catcode 10 (space), TeX fills lines with as many boxes as it takes to fill the line without stretching spaces. This effectively disables the TeX beautifying mechanism.
  2. Macros should be expanded, but they are not. See \"005C, for example.
  3. Here is another point I forgot to mention: The new lines are ignored despite \obeylines. This is because both } and \par (from ^^M) would yield \relax. This is undesired.

enter image description here

Jonathan Komar
  • 13,297
  • What is your end goal supposed to look like? For example, does \obeylines\obeyspaces\ttfamily instead of \treateachletterasword{} suffice? – Steven B. Segletes Aug 11 '17 at 12:19
  • @StevenB.Segletes I added the output. And no. – Jonathan Komar Aug 11 '17 at 12:25
  • 1
    what is a "letter" here? is å a letter, or ﭴ ? they are multiple tokens to pdftex (which is exactly the issue that listings has when it tries to do exactly this and space out "characters") – David Carlisle Aug 11 '17 at 12:35
  • why are you adding invisible glue? that amount of stretch will never be seen, it would be equivalent to just add 0pt or more simply, if the intention is to add linebreaking, to not add glue but to add \linebreak[0] (which is \penalty0) – David Carlisle Aug 11 '17 at 12:37
  • also space in a monospaced font doesnt stretch so it isn't clear why you need the spaceskip settings. – David Carlisle Aug 11 '17 at 12:39
  • @DavidCarlisle Aha. Well, I intend to avoid the issues encountered by the listing package. I hope to do this by ensuring that everything is pre-escaped by a script before it is put into the TeX file. This means that the active character catcode 13, letters and punctuation catcodes 11 and 12, and spaces catcode 10 will be run through the scanner only. Let's pretend the monospace font is not there :) (using xelatex). I have another big problem. The first test tests for \relax. This means that new lines are ignored despite \obeylines, since } and \par (from ^^M) would yield a \relax. – Jonathan Komar Aug 11 '17 at 12:41
  • 2
    last question are you using luatex or xetex? (both have features that probably make this easier) lua callbacks or xetex inter-character classes respectively. (The xetex character classes were specifically designed to add inter-character glue to allow line breaking in asian languages that don't use space, which seems to be very close to this) – David Carlisle Aug 11 '17 at 12:42
  • @JonathanKomar but that didn't answer my question about what you mean by a letter an accented e is two characters one active, one not, the arabic one would be three, and if you add space between those tokens they won't work at all. (but actually I since noticed that you are using fontspec, so a unicode tex so none of that applies and characters won't be encoded as multiple tokens) – David Carlisle Aug 11 '17 at 12:44
  • 1
    \ifcat\token\active does not test if the token is active, if the token is expandable it tests the first two tokens in its expansion have the same catcode, otherwise it tests if the token is not a character (as all other non expandable tokens, including \active test as equal to \ifcat) – David Carlisle Aug 11 '17 at 12:52
  • @DavidCarlisle Yes, I am using xelatex. I do not see a difference between using invisible glue and \penalty0. Is there some performance benefit to using one or the other? – Jonathan Komar Aug 11 '17 at 12:53
  • almost certainly, but also the code is much clearer as unless you happen to notice how small the space is it seems like you are trying to add white space for inter-character justification (which is what listings does) but as it is you are making tex work quite a bit harder as you are adding loads of stretch glue in to the line so tex has to work out how much to stretch it. You know in advance there isn't enough stretch to make any difference , but it doesn't... 0pt would be easier to understand (and more efficient) than -1sp but a penalty is much clearer – David Carlisle Aug 11 '17 at 12:56
  • @DavidCarlisle Note about the spaces: I considered this, and concluded that even if a mono font is used, it still makes sense to remove the glue if the goal is to precisely control the width of the spaces. – Jonathan Komar Aug 11 '17 at 13:04
  • `\documentclass{article}

    \begin{document} \def\token{abc}

    \ifcat\token\space \token (token is catcode 10) \fi

    \ifcat\token\active \token (token is active) \fi \end{document}`

     – David Carlisle Aug 11 '17 at 13:15

1 Answers1

3

I was able to solve this with a concept unique to XeTeX: The Character and Intercharacter Classes as suggested by David Carlisle. Will Robertson and Khaled Hosny did a good job documenting this functionality here, http://mirror.unl.edu/ctan/info/xetexref/xetex-reference.pdf.

There seem to be two ways to set delimiters for the injection of characters between tokens using \XeTeXinterchartoks:

  1. Between character slots and intercharacter classes. e.g. Use \XeTeXcharclass to assign `\A to a \macro and finally define the injection as \XeTeXinterchartoks \macro 0 = {\penalty0\relax}. This adds a penalty of 0 after every occurance of the "A" character when it is followed by a character of class 0.
  2. Between intercharacter classes. e.g. An intercharacter class is a number from 0 to 255 that represents a group of related characters. e.g. \XeTeXinterchartoks 0 0 = {\penalty0\relax}. Unfortunately, this part of the document was unclear to me, so I did a quick search and found another document that sheds more light on the intercharacter classes: https://github.com/zohooo/interchar/blob/master/interchar.tex.

Code

I removed the token scanner in this version. It is no longer necessary, perhaps because the \XeTeXinterchartoks functionality takes place before or during the assignment of category codes. I am not sure exactly how it works. The only downside is that I lost control over spaces. There might be a way to implement this control again using character classes.

\documentclass{article}
\usepackage{fontspec}
\usepackage{lipsum}

\newfontfamily\monofont{Inconsolatazi4-Regular.otf}% let's pretend this is not a mono font for practice

\long\def\treateachletterasword{%
  \bgroup
    \XeTeXinterchartokenstate=1% Enable Character Classes (unique to xelatex) 0=off 1=on
    \XeTeXinterchartoks 0 0 = {\penalty0\relax}% Set token to be inserted between interchar class 0 and interchar class 0
    \monofont% Set monospaced font
    \setlength{\parindent}{0pt}% rm new par indents
    \obeylines% \catcode`\^^M\active \let ^^M\par
}%
\def\endtreateachletterasword{\egroup}

\begin{document}
\section{SomeTeXFile.tex}\subsection{Typesetting Failure}
\begin{treateachletterasword}
This is XeTeX, Version 3.14159265-2.6-0.99996 (TeX Live 2016) (preloaded format=xelatex 2016.6.27)  11 AUG 2017 13:15
entering extended mode
 restricted \char"005C{}write18 enabled.
 file:line:error style messages enabled.
 \%\&-line parsing enabled.
**ThirdPartyLicenses.tex
(./ThirdPartyLicenses.tex
LaTeX2e <2016/03/31> patch level 2
Babel <3.9r> and hyphenation patterns for 83 language(s) loaded.
(../../DocumentClass.tex (/usr/local/texlive/2016/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/local/texlive/2016/texmf-dist/tex/latex/base/size10.clo
File: size10.clo 2014/09/29 v1.4h Standard LaTeX file (size option)
)
\char"005C{}c@part=\char"005C{}count79
\char"005C{}c@section=\char"005C{}count80
\char"005C{}c@subsection=\char"005C{}count81
\char"005C{}c@subsubsection=\char"005C{}count82
\char"005C{}c@paragraph=\char"005C{}count83
\char"005C{}c@subparagraph=\char"005C{}count84
\char"005C{}c@figure=\char"005C{}count85
\char"005C{}c@table=\char"005C{}count86
\char"005C{}abovecaptionskip=\char"005C{}skip41
\char"005C{}belowcaptionskip=\char"005C{}skip42
\char"005C{}bibindent=\char"005C{}dimen102
\end{treateachletterasword}
\end{document}

Output

enter image description here

Be careful! The \relax after the \penalty0 is necessary else consecutive numbers will be gobbled up by TeX as shown in the output image. I have since corrected this mistake in the code. Also, I have since added \setlength{\parindent}{0pt} to remove the new paragraph indentation after each new line of code caused by \obeylines.

Jonathan Komar
  • 13,297