9

I'm trying to import some logs files using \verbatiminput to do a automatic report from our logs. I got some encoding problem.

My objective: import some logs files with various special char (included hidden one) as verbatim.

  • I believed the file to be in ISO-8859-1. (the server OS is windows)
  • I got a LOT of file. So the usual solution "rewrite it"/"look for the faulty char" is not an option.
  • I run pdflatex from a mac osx.
  • I converted the files with (notice the -c option to drop faulty char)

.

for file in ./test*/*.txt
do
  iconv -c -f ISO-8859-1 -t utf8 $file  > $file.n1
  mv -f $file.n1 $file 
done

My tex header:

\usepackage{verbatim}
\usepackage{lmodern}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

error message I got :

! Package inputenc Error: Unicode char \u8:° not set up for use with LaTeX.

To solve the "degree" problem, I added 

cat $file.n1 | tr '°' 'º' > $file.n2

then I got an other error on an other file, this time with an invisible char.

! Package inputenc Error: Unicode char \u8: not set up for use with LaTeX.

I first tried to not convert the file and import them as T1/latin1 or LY1/ascinew, but I got the error

! Package inputenc Error: Keyboard character used is undefined
(inputenc)                in inputencoding `latin1'.

on different files.

My question:

  • Is there a way to ignore/replace ALL the faulty char?
  • from LaTeX ? from the shell ?
  • using "tr" or "iconv" ?
  • Am I doing something wrong when importing the files / doing the conversion?
  • with which encoding will I have less problem ?

any help is welcome, as I am blocked on this since two days.

PS: My problem is not with the degree symbole in particular, but with unicode char that LaTeX cannot printout.

lockstep
  • 250,273
Loda
  • 265
  • Very strange, but, at the end, I just remove the \usepackage[utf8]{inputenc} and the iconv then everything works fine. (using pdflatex). Thanks a lot for the answers, I did learn useful stuffs. – Loda Jan 04 '12 at 14:35
  • Well if your files are not utf8-encoded than if you use \usepackage[utf8]{inputenc} you are lying to LaTeX - and LaTeX doesn't like lies ;-). But you shouldn't remove inputenc completly (unless you are using an unicode engine like xetex). If you do it non-ascii-chars will only work partly. Load inputenc with the correct option (your encoding), and if you load files with another encoding than tell this LaTeX with the \inputencoding command. – Ulrike Fischer Jan 04 '12 at 15:29

3 Answers3

4

Yes you can replace the faulty char. You could replace ° with \ensuremath{^\circ}

or add to your latex header:

\newcommand{\degree}{\ensuremath{^\circ}}
\DeclareUnicodeCharacter{B0}{\degree}

Or you could use xelatex then there is no need for \usepackage[utf8]{inputenc} because it has full utf8 support build in.

The encoding is all right Latex just doesn't know what to do with the unicode symbol :D

someonr
  • 8,531
  • as I include the files as verbatim, changing the faulty char in the imported file will not help, no? I will look at the xelatex... – Loda Jan 04 '12 at 11:48
  • No this would be hard inside of verbatim. XeLatex should be the easiest option. Just remove the \usepackage[utf8]{inputenc} from your header and use xelatex instead of pdflatex. If you want to see the unicode characters make sure that you use a font that supports them. (The standard cmtt10 hasn't support) – someonr Jan 04 '12 at 11:58
  • You can use \usepackage{xeCJK} to get such a font. – Martin Ueding Mar 03 '12 at 11:26
4

The character ° is actually not included in the definitions make by the utf8 module for inputenc.

\usepackage{textcomp} % for \textdegree
\usepackage{newunicodechar} % for \newunicodechar
\newunicodechar{°}{\textdegree}

Alternatively, you can resort to a "hand made symbol"

\newunicodechar{°}{\ensuremath{{}^\circ}}

that will work both in text and math mode (and textcomp is not needed).

The advantage of newunicodechar is that it doesn't require knowing the Unicode point of the character we want to define a command for.

Of course the "degree" case is an example: you can add all the characters you need.

However, if your files are Latin-1 encoded (ISO 8859-1), you can try switching to this encoding before inputting them:

\usepackage[latin1,utf8]{inputenc} % utf8 is default
\newcommand{\latinoneverbatiminput}{%
  \begingroup\inputencoding{latin1}\verbatiminput{#1}\endgroup}

Then \latinoneverbatiminput{file} will use Latin-1 for file.

egreg
  • 1,121,712
  • nice package. I always looked at a Unicode table for this. – someonr Jan 04 '12 at 11:55
  • @someonr That's why I wrote it: I'm lazy. :) – egreg Jan 04 '12 at 11:58
  • about the "degree" : it's working: I removed the extra "tr" from my .sh. But How do I solve the invisible faulty char ? – Loda Jan 04 '12 at 12:00
  • 1
    You can look at the log file, for example with LESSCHARSET=ascii less file: search for u8 and you'll find the UTF-8 representation of the character; for example the degree is <C2><B0>. Add it to your question, so that a recipe for dealing with it can be suggested. – egreg Jan 04 '12 at 12:08
  • I will keep in my notes the "\newunicodechar". nice function! – Loda Jan 04 '12 at 14:36
1

You don't need to reencode your files. But you must declare/change the encoding of the files before loading them:

\begingroup
\inputencoding{whatever}
\verbatiminput{file}
\endgroup

I don't know what value you should use for "whatever". On windows I would at first try "ansinew" (latin1 has a less chars declared).

Ulrike Fischer
  • 327,261