Spacing issue with literate in listings

Question

I answered this question and OP showed me a problem with the answer. The code:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{listings}

\lstset{
  % keepspaces, % Apparently this works...
  literate=
    {à}{{\`a}}1
}

\begin{document}

\begin{lstlisting}[mathescape=true]
  Adicionar v à lista.
  Adicionar $v$ à lista.
  Adicionar $v$ onde?
  R: à lista.
  Yoda style: à lista, $v$ adicionar.
  Adicionar $v$ à lista e à outra lista.
\end{lstlisting}

\end{document}

prints:

and the space after the à is eaten.

From the four examples above, I concluded that the problem is isolated at the first literated character in lines after mathescaped text.

• The second and sixth lines show the issue.
• The fourth line shows that the effect from the third line stops after the line break.
• The fifth line shows that the effect starts only after the mathescaped text
• The sixth line shows that the problem does not propagate to the following literated characters.

Using the keepspaces option apparently solves the problem.

From the look of it I think it's a bug. How to fix this?

---

I also concluded that v was definitely added to the list :)

Answer 1

(August 12, 2018—it's Sunday and I don't have the nerve for delving deeply into the source code of the listings package.)

But when skimming over that source code, I got the impression that

1. the macro underlying the replacement-process for the things provided via the literate key is called \lst@Literate.

2. there is a switch \lst@ifwhitespace which indicates whether the last processed thingie has been a space which has been printed as a whitespace.

3. the code of \lst@Escape—\lst@Escape seems to underlie all those thingies for escaping to LaTeX—does set that switch to false in case of being used due to mathescape being true. So with mathescape=true the switch is false after $...$. This leads to processing the space following $...$ as whitespace which is not to be discarded but to be printed. This processing of the next space in turn leads to setting the switch to true again. Now the literate-thingie is processed. The underlying macro \lst@Literate does not set the switch to false. Thus due to the space before the literate-thingie being processesd as printed whitespace, the "machinery" after processing the literate-thingie still "thinks" that it has just processed and printed a whitespace and thus should discard following spaces.

This "thinking" of the machinery seems wrong in cases where the literate-thingie did as last item of output produce something other than whitespace .

What can be done about this?

Perhaps \lst@Literate should be redefined to set that switch to false.

Could look like this:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{listings}

\lstset{
  literate=
    {à}{{\`a}}1
}

\makeatletter
\def\lst@Literate#1#2#3{%
    \ifx\relax#2\@empty\else
        \lst@CArgX #1\relax\lst@CDef
            {}
            {\let\lst@next\@empty
             \lst@ifxliterate
                \lst@ifmode \let\lst@next\lst@CArgEmpty \fi
             \fi
             \ifx\lst@next\@empty
                 \ifx\lst@OutputBox\@gobble\else
                   \lst@XPrintToken \let\lst@scanmode\lst@scan@m
                   \lst@token{#2}\lst@length#3\relax
                   \lst@XPrintToken
                   \lst@whitespacefalse %!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
                 \fi
                 \let\lst@next\lst@CArgEmptyGobble
             \fi
             \lst@next}%
            \@empty
        \expandafter\lst@Literate
    \fi}
\makeatother

\begin{document}

\begin{lstlisting}[mathescape=true]
  Adicionar v à lista.
  Adicionar $v$ à lista.
  Adicionar $v$ onde?
  R: à lista.
  Yoda style: à lista, $v$ adicionar.
  Adicionar $v$ à lista e à outra lista.
\end{lstlisting}

\end{document}

(Addition in August 18, 2018—it's Saturday and I still don't have the nerve for delving deeply into the source code of the listings package. ;-) )

But when doing so, we loose the possibility of having the literate-mechanism produce material whereafter a following space-token does not lead to producing more whitespace.

Thus perhaps it is better to just set the \lst@whitespace-switch to false within the literate-directive whenever that directive is used for having LaTeX typeset non-whitespace.

Could look like this:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{listings}

\makeatletter
\lstset{
  literate=
    {à}{{\`a\lst@whitespacefalse}}1
}
\makeatother

\begin{document}

\begin{lstlisting}[mathescape=true]
  Adicionar v à lista.
  Adicionar $v$ à lista.
  Adicionar $v$ onde?
  R: à lista.
  Yoda style: à lista, $v$ adicionar.
  Adicionar $v$ à lista e à outra lista.
\end{lstlisting}

\end{document}

I can absolutely not guarantee that my analysis of the situation is correct.
Maybe it is a starting point.

Seems that things work out correctly when

• either adding \lst@whitespacefalse to the definition of \lst@Literate

• or applying \lst@whitespacefalse inside the literate-directive.

Again: I can absolutely not guarantee that my analysis of the situation is correct.
Maybe it is a starting point.

In case setting \lst@whitespacefalse inside the literate-directive is to be considered good practice, both this should be mentioned somewhere in the manual, and for the end-user there should be a possibility of setting that switch without using \makeatletter..\makeatother beforehand.

In case you come to the same conclusion, feel free to hand in a bug report. ;-)