21

I want to convert a .tex file into something that is possible to open in MS Word (.doc, .docx, .rtf, ...) by means of the pandoc command line software. A minimum .tex file is attached below. I use the following command in a standard command window when trying to convert :

pandoc -o MyFile.tex MyFile.rtf

This produces the following error message: enter image description here

What does "Invalid UTF-8 stream" mean, and what can I do about it? I am using MikTeX 2.9 and TexWorks.

My latex file MyFile.tex is like this:

% !TeX program    = pdflatex
% !TeX encoding   = ISO-8859-1
% !TeX spellcheck = nb_NO

\documentclass[10pt,a4paper,twosided]{article}

\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amsthm} 
\usepackage{amssymb}
%\usepackage[dvips]{graphicx}
\usepackage{graphicx}

%\usepackage[french,english]{babel}
\usepackage[english]{babel}
%\usepackage[latin1]{inputenc}

\usepackage[usenames,dvipsnames]{color}
\usepackage{subfigure}
\usepackage{natbib}
\usepackage{dcolumn}

\usepackage{bm}
\usepackage{url}

\usepackage{alltt}
\usepackage{xspace}
\usepackage{enumerate}
\usepackage{textcomp}
\usepackage{etoolbox}
\usepackage{longtable}
\usepackage{pdflscape} 
%\usepackage[absolute,showboxes]{textpos}
\usepackage[absolute]{textpos}

\usepackage{fancyhdr}
\usepackage{array}
\usepackage{tabularx}
\usepackage{xfrac}

\usepackage{fmtcount}
\usepackage{mfirstuc}
\usepackage[width=0.95\textwidth,font={bf,small}]{caption}[2011/08/06]

\usepackage{booktabs}
\usepackage[table]{xcolor}

\usepackage{tikz}
\usepackage{pgfplots}
  \pgfplotsset{compat=newest}
  \pgfplotsset{plot coordinates/math parser=false}

\usepackage{filecontents}
\usepackage[pagewise,displaymath,mathlines]{lineno}
\usepackage{layout}
\usepackage[margin=1in]{geometry}
\usepackage{changepage}
\usepackage{placeins}
\usepackage{paralist}
\usepackage{refcount}

\usepackage[pdftex]{hyperref}


\hypersetup{pdfauthor = Espen Donali}

\hypersetup{pdfdisplaydoctitle=true}
\hypersetup{bookmarks=true}
\hypersetup{pdfpagemode=UseOutlines}
\hypersetup{pdfpagelayout=TwoColumnRight}

\hypersetup{pdfmenubar=true}
\hypersetup{pdftoolbar=true}
\hypersetup{dvips}
\hypersetup{hyperfigures}
\hypersetup{pdfstartview=FitV}

\hypersetup{backref=page}
\hypersetup{pagebackref=true}

\hypersetup{colorlinks=true}
\hypersetup{allcolors=blue}
\hypersetup{linkcolor=blue}
\hypersetup{citecolor=blue}
\hypersetup{urlcolor=blue}
\hypersetup{anchorcolor=red}
\hypersetup{menucolor=red}
\hypersetup{runcolor=red}
\hypersetup{hyperfootnotes=true}



\begin{document}

my test of pandoc latex to word

\end{document}

enter code here
myotis
  • 936

3 Answers3

14

You have told your editor that the file is stored as latin1 encoding

% !TeX encoding   = ISO-8859-1

But pandoc thinks it is reading a UTF-8 encoded file so can not decode it. These encodings are the same for ascii letters but differ for anything else.

Pandoc may have an option to tell it the file encoding or if not, you should get your editor to save the file as utf-8.

David Carlisle
  • 757,742
  • Thanks :) By changing "ISO-8859-1" to "UTF-8" in my minimum example pandoc functioned ! BUT, when doing the same on my real file, from which the minimum example were made from, I get this error: pandoc: Cannot decode byte '\x72' : Data.Text.Encoding.decodeUtf8: Invalid UTF-8 stream. Is there some mismatch between UTF-8 and any of the packages I have used? – myotis Feb 12 '13 at 12:54
  • Follow up question: It seems like pandoc needs all latex code to be contained within the sinlge .tex file being processed. But I have numerous occurences of \inpu{} in my masterfile. Are there any functionality in latex letting me construct a new masterfile where all the input files are copied into the masterfile instead of being referred to with the \input{} command ? – myotis Feb 12 '13 at 13:25
  • Have look at Martin Scharrers answer to this question: http://tex.stackexchange.com/questions/21838/replace-inputfilex-by-the-content-of-filex-automatically – DG' Nov 26 '13 at 14:07
  • @DavidCarlisle I want to pdf https://hea-www.harvard.edu/~fine/Tech/vi.html document using pandoc -o vi.pdf vi.html that produce pandoc: Cannot decode byte '\xd7': Data.Text.Encoding.Fusion.streamUtf8: Invalid UTF-8 stream error. But, as it is html file how do I know the encoding? may be ascii. – alhelal Feb 12 '18 at 02:29
  • @DavidCarlisle I download this page by right clicking and save as html. – alhelal Feb 12 '18 at 02:51
10

Pandoc uses the UTF-8 character encoding for both input and output. If your local character encoding is not UTF-8, you should pipe input and output through iconv:

iconv -t utf-8 input.txt | pandoc | iconv -f utf-8
dontmentionthebackup
  • 201
  • 2
  • 2
  • 1
    I get iconv: template.html:411:54: cannot convert – incandescentman Mar 21 '17 at 05:10
  • 1
    when using vi/vim to view the .tex file use :set fileencoding=utf-8 and save ist. to show the current fileencoding in vim just type :set fileencoding oder :set fileencoding? Additionally maybe look whats inside :set fileencodings, too – MacMartin Aug 11 '17 at 09:46
  • 1
    @incandescentman I figured how to avoid the "iconv: cannot convert" error from the command line. I just tried iconv -f ISO-8859-1 -t UTF-8 template.html and it worked fine. It seems iconv is also not correctly reading the input encoding, so we have to specify it manually. – Luke Davis Aug 18 '18 at 06:02
4

Check JabRef encoding

In my case, I bumped into a similar error when converting Pandoc Markdown to XHTML. The culprit were a set of BibTeX citations which JabRef had encoded by default in ISO8859_1.

This default JabRef behaviour can be changed once and for all by setting Default encoding: to UTF8 in JabRef's Options > Preferences > General menu.

Community
  • 1
Serge Stroobandt
  • 4,252
  • Good thought, but I don't know enough about how pandoc is handling references. I just used iconv on the .bbl and .aux file too but that didn't seem to do it. – Joanna Bryson May 15 '17 at 19:27