8

I am trying to output several megabytes of UTF8 text into a printable format. As enscript and friends don't like Unicode, this Unix SE question soon turned to LaTeX.

Summary of requirements: processing of several MB of Chinese, Latin and special chars (i.e. invalid LaTeX code), consisting of long and short lines, into a tightly packed, multi-column format. I know LaTeX has the ability to control font size, spacings, margins and so on, as well as headings and page numbers, so I am not too worried about that.

The problem I am having so far is getting both wrappable and UTF8-capable verbatim input of the file.

Is there a way to get both?

With the fancyvbr package, for example, I get good rendition of the Chinese text, but no line breaking:

\documentclass{article}
\usepackage{CJKutf8}
\usepackage{ucs}
\usepackage{multicol}
\usepackage{verbatim}
\usepackage[encapsulated]{CJK}
\newcommand{\myfont}{bsmi} % or {stheiti}, etc

\begin{document}

\begin{CJK}{UTF8}{\myfont}

    \begin{multicols}{2}
        \VerbatimInput[fontfamily=cmr]{file.txt}
    \end{multicols}
\end{CJK}
\end{document}

I have so far been unable to get a listings-type environment to deal with the Unicode in the file.

Example of the type of thing I'm feeding it. This to to format thousands of lines of multi-lingual chat logs. Basically 100,000++ lines of this kind of thing:

###### 2013-10-26.223938+0000GMT.html ######
**** user@example.com/ (jabber) ****
(00:00:00) Lorem ipsum
(00:00:01) 車檢作畫病得星定局而是作的所由次園又此對這一馬的生故他試……外由懷黃客建時常嚴在位以說其配場戲回有部結一自法就生機，定的被各世皮全空！也地傳現他重城，書照展商直起眾家不思政國林年八計出地口早體故失離際們層氣，簡他廣集義，四便入的只了極。
(00:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet definiebas eos ad. Te eos nihil populo vivendum, vix iusto noster peri
(00:00:02) odesset ullamcorper quo. Cu adip
(02:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet d
(02:00:01) 車檢作畫病得星定局而是作的所由次園又此對這一馬的生故他試……外由懷黃客建時常嚴在位以說其配場戲回有部結一自法就生機，定的被各
(02:00:02) ok
(00:01:02) 病
(00:00:02) ok
(00:01:02) 

###### 2013-10-26.223938+0000GMT.html ######
**** user@example.com/ (jabber) ****
(00:00:01) 車檢作畫病得星定局而是作展商直起眾家不思政國林年八計出地口早體故失離際們層氣，簡他廣集義，四便入的只了極。
(00:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet definiebas eos ad. Te eos nihil populo vivendum, vix iusto noster peri
(00:00:02) odesset ullamcorper quo. Cu adip
(02:00:02) odesset ullamcorper quo.

I'm ideally looking for the most general solution possible, as the input text is not necessarily in a well-defined machine-readable format.

Edit also tried the following using the plain verbatim package:

\usepackage{verbatim}

\makeatletter
\def\@xobeysp{ }
\makeatother

with this in the document body:

\verbatiminput{file.txt}

This works for normal text and Chinese, but fails to break lines containing very long words, URLs or long strings of letters, all of which occur in the file.

Community
  • 1
diwhyyyyy
  • 235

4 Answers4

6

enter image description here

You can tell xetex to linebreak Chinese and not to worry about characters that are normally special. (You probably need to make more characters catcode 12, this shows the basic idea)

\documentclass{article}

\usepackage{fontspec}

\setmainfont{SimSun}


\newcommand\showlog[1]{{%
\catcode`\#=12
\catcode`\^=12
\catcode`\$=12
\obeylines
\raggedright
\XeTeXlinebreaklocale "zh"
\input{#1}}}

\begin{document}


\showlog{\jobname.txt}

\end{document}
David Carlisle
  • 757,742
6

As you have already used, verbatim package is easy to configure to get proper line breaking and escape all the special characters.

For CJK text, xeCJK is your friend. And there are some options of xeCJK to control the behavior for verbatim CJK text.

I hacked into xeCJK package to tune the linebreak. It seems better, but not perfect.

% -*- coding: utf-8 -*-
% Compile with XeLaTeX
\documentclass{article}

\usepackage{etoolbox}

\usepackage{verbatim}
\makeatletter
\def\@xobeysp{\ }% Or just a space, with a different result
\let\verbatim@nolig@list\empty
\appto\verbatim@font{\raggedright}
\makeatother

\usepackage{xeCJK}% You'd better use the latest version
\setCJKmonofont{SimSun}
\xeCJKsetup{Verb=false}
\normalspacedchars{}
\ExplSyntaxOn
% Hack into xeCJK package if you want to allow linebreaks after (almost) any character.
% Delete this if you don't want that.
\tex_chardef:D \c_fifty = 50 ~
\xeCJK_inter_class_toks:nnn { Default } { Default } { \tex_penalty:D \c_one_hundred }
\xeCJK_inter_class_toks:nnn { Default } { HalfLeft } { \tex_penalty:D \c_fifty }
\xeCJK_inter_class_toks:nnn { Default } { HalfRight } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfLeft } { Default } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfLeft } { HalfLeft } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfLeft } { HalfRight } { \tex_penalty:D \c_one_thousand }
\xeCJK_inter_class_toks:nnn { HalfRight } { Default } { \tex_penalty:D \c_fifty }
\xeCJK_inter_class_toks:nnn { HalfRight } { HalfLeft } { \tex_penalty:D \c_fifty }
\xeCJK_inter_class_toks:nnn { HalfRight } { HalfRight } { \tex_penalty:D \c_one_thousand }
\ExplSyntaxOff

\begin{document}

\verbatiminput{test.log}

\end{document}

enter image description here

The test log file with long lines:

###### 2013-10-26.223938+0000GMT.html ######
**** user@example.com/ (jabber) ****
(00:00:00) Lorem ipsum
(00:00:01) 車檢作畫病得星定局而是作的所由次園又此對這一馬的生故他試……外由懷黃客建時常嚴在位以說其配場戲回有部結一自法就生機，定的被各世皮全空！也地傳現他重城，書照展商直起眾家不思政國林年八計出地口早體故失離際們層氣，簡他廣集義，四便入的只了極。
(00:00:02) odesset ullamcorper quo. Cu adipisci assentior eam, debet definiebas eos ad. Te eos nihil populo vivendum, vix iusto noster peri
(00:00:02) `~!@#$%^&*()_+-={}[]|\:;"'.?/
(02:00:02) LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLONG LINE
Leo Liu
  • 77,365
  • This is the closest answer to what I want to happen, and it generates output that I can use right now. The only (minor) issue is that the line breaking doesn't prefer to break on a space or special char like, say, "?" in a URL. Is there a way to do this? – diwhyyyyy Nov 21 '13 at 17:31
  • @Inductiveload: You'd better specify how to break lines in detail. BTW, TeX don't understand the text, it can hardly recognize URLs. – Leo Liu Nov 22 '13 at 02:49
  • The way the script I posted does it is in two steps. There are a set of "breakable chars" that it looks for, so far "&;.?!", as this covers most URLs and file names. First it adds "-" after the first of these chars in each run of chars. Then, it adds more soft-hyphens after every 12 sequences of characters that don't include an existing soft-hyphen or whitespace. This way, it will break on "nice" chars and spaces first, and then split long chunks up as best it can. Of course, there is no awareness of hyphenation correctness, but that is really not an issue, as 99% are done well. – diwhyyyyy Nov 22 '13 at 11:12
  • @Inductiveload: Maybe you have noticed, I edited my solution and tried my best to provide a reasonable linebreak routine. My solution exploited TeX's linebreaking algorithm via setting \penalty arguments, which may be smarter than your perl script. – Leo Liu Nov 22 '13 at 13:31
  • Wonderful! The breaking is indeed nice and very sensible looking! One final thing: any possibility of \raggedright in this kind of \verbatiminput environment? Currently, it is throwing a lot of underfull hboxes (when it can only fit one chunk on a line). – diwhyyyyy Nov 22 '13 at 17:57
  • @Inductiveload: Done. – Leo Liu Nov 23 '13 at 05:14
  • Brilliant! This is also useful as a reference to see how you can adjust things like this in future. – diwhyyyyy Nov 23 '13 at 16:24
3

you do not have any special character except the # then you can use a simple \input. My example uses xelatex which makes live easier with CJK languages. 

\documentclass{article}
\usepackage[margin=1cm]{geometry}
\usepackage{fontspec}
\setmainfont{Code2000}
\setmonofont{DejaVu Sans Mono}%% has no chines characters
\usepackage{pdflscape}
\newcommand\Listing[1]%
  {\begin{landscape}
   \catcode`\#=12%
   \obeylines
%   \ttfamily%%%    only possible with a mono type font with chinese characters
   \footnotesize
   \input{#1}%
  \end{landscape}}

\begin{document}
foo

\Listing{xxx.tex}%% THE EXTERNAL FILE

bar 
\end{document}

enter image description here

  • 1
    I have no guarantees about what is and is not in the text. Other than #, I can see at least ^, , $, / in the file, there could be anything else in there. – diwhyyyyy Nov 20 '13 at 18:39
  • no problem, it can also escaped. And, of course, you can use \VerbatimInput from package fancyvrb instead –  Nov 20 '13 at 18:49
  • The problem is that doesn't wrap lines (though you can hack normal verbatim to break on spaces, long words, URLs, etc are unbroken and stick out of the right hand side of the column). – diwhyyyyy Nov 20 '13 at 19:14
  • @Inductiveload: So what do you want to break the lines? Is it allowed to break after any character? – Leo Liu Nov 21 '13 at 11:43
  • Yes, pretty much. So far I am using a bash script to insert a \- after special chars like ,.&?!, and then breaking any run of text without \- or whitespace with \-. – diwhyyyyy Nov 21 '13 at 11:49
1

My current best effort at this is based on the concept by @Gilles at Unix SE. It is not a pure *TeX implementation, and it is not perfect, but it is the one that has allowed me to compress my first 1.4MB of text onto 36 pages: around 40kB and 1400 lines per page.

I use perl to pre-process the input text to escape LaTeX special characters, and also add soft-breaks at certain special characters (like ?!/\$, etc), as well as artificially breaking up long words and meaningless strings into maximum-length runs with more soft-hyphens. Obviously, this won't respect "good" places to hyphenate, but that is an incredibly small detail, as in nearly all the cases, it either breaks on a space, special char, or the string itself doesn't have a good place to break anyway.

This is then processed with XeLaTeX and CJK to handle the Chinese using xeCJK. It is placed in a multicols with very narrow columns because there are a lot of very short lines, and I don't want to waste space.

#! /bin/bash

maxLength=12
breakableChars='\&\;\/\.\?\!\='

# break singly-occuring breakable chars, and the first one in runs with a soft hyphen
# break any un-broken runs of any ascii with a soft hyphen
# replace any backslashes with \textbackslash{}
# replace braces (except the ones in \textbackslash{})
# escape #$%&_ with backslashes
# escape ^ | and ~ with \textTHING{}
# line break after all non-empty lines
cat $1 | 
perl -pe 's/(?<![\s'$breakableChars'])(['$breakableChars'])(?!\s)/\1\\-/g;' | 
perl -pe 's/(((?<!-)[!-~]){'$maxLength'}(?!\s))/\1\\-/g;' |
perl -pe 's/\\(?!-)/\\textbackslash{}/g;' | 
perl -pe 's/(?<!\\textbackslash){/\\{/g;' | 
perl -pe 's/(?<!\\textbackslash\{)\}/\\}/g;' | 
perl -pe 's/([\#\%\$\&\_])/\\\1/g;' | 
perl  -pe 's/\^/\\textasciicircum{}/g;'| 
perl -pe 's/\|/\\textbar/g;' | 
perl -pe 's/~/\\textasciitilde{}/g;' | 
perl -pe 's/^(.*\S.*)$/\1\\\\/g;'  > $2

cat << EOF > driver.tex
\documentclass{article}

\usepackage{fontspec}
\usepackage{multicol}

\usepackage[margin=1cm]{geometry}
\usepackage{setspace} 
\usepackage{xeCJK}

\setstretch{0.5}
\setlength{\columnsep}{0.1cm}
\setmainfont{Nimbus Sans L Regular Condensed}
\setCJKmainfont{WenQuanYi Micro Hei}

\newcommand\showlog[1]{{%
\obeylines
\raggedright
\XeTeXlinebreaklocale "zh"
\input{#1}}}

\begin{document}
    \raggedright
    \fontsize{4}{5}\selectfont

    \begin{multicols}{9}
      \showlog{h.txt}
    \end{multicols}

\end{document}
EOF

/opt/texbin/xelatex driver.tex

The main remaining issue then is that non-Chinese, non-Latin-ish characters not in either the CJK font or the main font don't appear. Using ucharclasses and Code2000 works better, but it is horribly slow.

diwhyyyyy
  • 235