How to prevent extendedchars=\true from producing a blank line?

Question

My previous question How to suppress BOM effect in the output? has been solved by @Vaulty. Enabling extendedchars=\true, however, produces an unnecessary blank line at the first line as shown below.

\documentclass{article}
\usepackage{xcolor}
\usepackage{listings}

\lstset
{ 
        breaklines=true,
        tabsize=3, 
        showstringspaces=false,
        extendedchars=\true,%<======= SOURCE OF PROBLEM
        language={[Sharp]C},   
        frame=single,   
        rulecolor=\color{red}%   
}

\begin{document}

\lstinputlisting{Program.cs}

\end{document}

Program.cs, that is generated by Visual Studio, is always prefixed with a BOM (Byte Order Mark). If you have no Microsoft Visual Studio installed, you can download a test file named Default.aspx.cs from the official ASP.NET website securely. It is worth mentioning that the backslash in \true is NOT a typo.

The question is how to prevent extendedchars=\true from producing a blank line?

Answer 1

The correct syntax is extendedchars=true without the backslash. But it doesn't make sense without inputenc. And with inputenc and/or fontenc+T1 you don't get the wanted result as then the input will give characters.

If you use a command as value this inserts after the BOM a new line in the code. That's why it looks as if is it a solution to your problem. But actually it will break again if you load inputenc or fontenc.

\documentclass{article}
\usepackage{xcolor}
\usepackage{listings}
\usepackage[T1]{fontenc}
\usepackage[ansinew]{inputenc}

\lstset
{
        breaklines=true,
        tabsize=3,
        showstringspaces=false,
        extendedchars=\blub,%<======= SOURCE OF PROBLEM
        language={[Sharp]C},
        frame=single,
        rulecolor=\color{red}%
}

\begin{document}
abc
\lstinputlisting{test-bom.txt}

\end{document}

And here a suggestion how to get around the BOM-problem when your main document is utf8 and the listing is (with the exception of the BOM) pure ASCII:

\documentclass{article}
 \usepackage[T1]{fontenc}
 \usepackage{listings}
 \usepackage[utf8]{inputenc}

 \lstset{%
 language={[Sharp]C}}

\begin{document}

\begingroup  
 \inputencoding{ansinew}
 \lstset{
   literate={^^ef^^bb^^bf}{}0
 }
 \lstinputlisting{test-bom.txt}
\endgroup
\end{document}

Answer 2

There is no empty line with

\lstset{ 
        breaklines,
        tabsize=3, 
        showstringspaces=false,
        extendedchars,
        language={[Sharp]C},   
        frame=single,   
        rulecolor=\color{red}%   
}

Answer 3

The extendedchars option of the listings package is relevant if you want to typeset source code that includes non-ASCII characters using a TeX engine without native Unicode support such as pdfTeX. Obviously, the listings package has to deal a lot with category codes and, specifically, the extendedchars option addresses collaboration with the inputenc package which traditionally enables non-ASCII encoding of LaTeX input files via character activation.

By current design, the collaboration of the listings package with the inputenc package is restricted to single-byte encodings such as ISO-8859-1 and ISO-8859-15. As a consequence, the default value of the extendedchars option (i.e. true) is no use when the source code to be typeset is encoded in UTF-8, independently of whether the utf8 or the utf8x module is loaded by the inputenc package. In either case, you get errors and faulty output.

If the extendedchars option is set to false (or \true, \chapter, \documentclass etc. ;-) ) the listings package doesn’t attempt to collaborate with the inputenc package. Depending on the combination of active input encoding and UTF-8 character, this may accidentally result in pseudo-correct output (as is the case with the combination utf8x + BOM). In general, however, you get errors and/or faulty output once again.

To sum up: When the source code to be typeset is encoded in UTF-8 playing around with the extendedchars option doesn’t make sense. But there are two other possibilities:

• Resort to the literate option as suggested by Ulrike Fischer.

• Instead of the listings package, use the listingsutf8 package in conjunction with the extended inputencoding option:

  ...
  \usepackage{listingsutf8}
  \lstset{%
    inputencoding=utf8/ascii,
    breaklines=true,
    tabsize=3,
    showstringspaces=false,
    language={[Sharp]C},
    frame=single,
    rulecolor=\color{red}%
  }
  ...
  

  Actually, the listingsutf8 package is only capable of handling UTF-8 characters that can be translated to some single-byte encoding. UTF-8 characters such as the BOM that can’t be translated to any single-byte encoding are silently ignored. This may or may not be desirable. In case of the BOM, it fortunately is.

Answer 4

This issue only happens in MikTeX. Without using utf8x encoding and/or extendedchars option, this issue disappears if you use TeXLive.

Answer 5

Here is the solution that has been actually answered by Vaulty. Unfortunately, I did not realize that \usepackage[utf8x]{inputenc} is needed.

\documentclass{article}
\usepackage{xcolor}
\usepackage[utf8x]{inputenc}%<========= MANDATORY
\usepackage{listings}

\lstset
{ 
        breaklines=true,
        tabsize=3, 
        showstringspaces=false,
        extendedchars=\true,%<======= MANDATORY, it is not a typo :)
        language={[Sharp]C},   
        frame=single,   
        rulecolor=\color{red}%   
}

\begin{document}

\lstinputlisting{Default.aspx.cs}

\end{document}