autounindent cannot work together with hyperref and hypcap

Question

This question is related to my previous question answered by Jubobs here. I compiled the given code below with both TeX Live 2013 and TeX Live 2014 pretest. Both produce the same result.

Case 1

The expected result with autounindent is as follows.

enter image description here

Case 2

When I load hyperref, the autounindent no longer works as follows.

enter image description here

Case 3

The worst case happens when I load both hyperref and hypcap, I got some errors as follows.

enter image description here

MWE

\documentclass[12pt]{article}

%TODO
% - handle tabs

\usepackage{accsupp}
\newcommand*{\noaccsupp}[1]{\BeginAccSupp{ActualText={}}#1\EndAccSupp{}}

\usepackage{xcolor}
\usepackage{listings}

\makeatletter

% -------- ugly details --------

% custom key
\lst@Key{autounindent}f[t]{\lstKV@SetIf{#1}\lst@ifautounindent}

% --- accumulation of tokens ---
\toks@={
    \lst@DefSaveDef{`\ }\lsts@myspace{\processmyspace}
    %\lst@DefSaveDef{`\^^I}\lsts@mytab{\processmytab}
}


% --- egreg's helper function for accumulating tokens ---
\def\add@savedef#1#2{%
  \begingroup\lccode`?=#1\relax
  \lowercase{\endgroup
  \edef\@temp{%
    \noexpand\lst@DefSaveDef{\number#1}%
    \expandafter\noexpand\csname lsts@?\endcsname{\noexpand#2% % I had to swap things around to fix a bug here;
      \expandafter\noexpand\csname lsts@?\endcsname}%          % otherwise, comment highlighting would be broken.
  }}%
  \toks@=\expandafter{\the\expandafter\toks@\@temp}%
}

\count@=33
\loop
    \add@savedef\count@\processchar
  \ifnum\count@<127
  \advance\count@\@ne
\repeat

% --- to keep track of the state ---
\newcount\spacesToGobble
\newcount\spacesGobbledSoFar
\newif\ifafterindent
\newif\ifonBasisLine
\newif\ifPostponeCountToNextLine


\lst@AddToHook{Init}
{%
  \lst@ifautounindent%
    % patch hook macros
    \let\@ddedToInitVarsBOLhook\@@ddedToInitVarsBOLhook%
    \let\@ddedToEOLhook\@@ddedToEOLhook%
    % initialise things
    \global\onBasisLinetrue%
    \global\afterindentfalse%
    \global\spacesToGobble=0%
    \lst@ifincluderangemarker%
      \ifnum9999999=\lst@firstline% (i.e. if firstline option not used)
        \global\PostponeCountToNextLinetrue%
      \fi
    \fi
  \fi
}


\lst@AddToHook{SelectCharTable}
{%
    \lst@ifautounindent%
        \lst@lAddTo\lst@DeveloperSCT{\the\toks@}%
    \fi
}


\lst@AddToHook{InitVarsBOL}{\@ddedToInitVarsBOLhook}
\newcommand\@ddedToInitVarsBOLhook{}
\newcommand\@@ddedToInitVarsBOLhook
{%
  \global\afterindentfalse%
}

\lst@AddToHook{EOL}{\@ddedToEOLhook}
\newcommand\@ddedToEOLhook{}
\newcommand\@@ddedToEOLhook
{%
  \ifonBasisLine%
    \ifPostponeCountToNextLine%
    \else
      \global\onBasisLinefalse%
    \fi
  \fi
  \global\spacesGobbledSoFar=0%
  \global\PostponeCountToNextLinefalse%
}

\lst@AddToHook{DeInit}
{%
  \lst@ifautounindent%
    % undo patches
    \def\@ddedToInitVarsBOLhook{}%
    \def\@ddedToEOLhook{}%
  \fi
}


% --- helper macros ---
\newcommand\processchar
{%
  \global\afterindenttrue%
}

\newcommand\processmyspace
{%
  \ifafterindent%
    \lsts@myspace%
  \else
    \ifonBasisLine%
      \global\advance\spacesToGobble\@ne\relax%
    \else
      \ifnum\spacesGobbledSoFar<\spacesToGobble\relax%
        \global\advance\spacesGobbledSoFar\@ne\relax%
      \else
        \lsts@myspace%
      \fi
    \fi
  \fi
}

\makeatother

% ---------- end of ugly details ----------

\usepackage{filecontents}
\begin{filecontents*}{Program.cs}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Delegate
{
    class Program 
    {
        // start
        sdfgsd fgdsfg  static void Main(string[] args)
        {
            for (int x = 0; x < 10; x++)
                Console.WriteLine(x);
        }
        /* sdflkjhsdf */
        // stop
    }
}
\end{filecontents*}

% --- general style definition ---
\lstdefinestyle{Common}
{   
    language={[Sharp]C},
    numbers=left,
    numbersep=1em,
    numberstyle=\tiny\noaccsupp,
    frame=single,
    framesep=\fboxsep,
    framerule=\fboxrule,
    rulecolor=\color{red},
    xleftmargin=\dimexpr\fboxsep+\fboxrule,
    xrightmargin=\dimexpr\fboxsep+\fboxrule,
    breaklines=true,
    breakindent=0pt,
    tabsize=2,
    columns=flexible,
    %includerangemarker=false,
    rangeprefix=//\ ,
}

\lstdefinestyle{A}
{
    style=Common,
    backgroundcolor=\color{yellow!10},
    basicstyle=\scriptsize\ttfamily,
    keywordstyle=\color{blue}\bf,
    identifierstyle=\color{black},
    stringstyle=\color{red},
    commentstyle=\color{green},
}

%\usepackage[colorlinks,bookmarksnumbered,bookmarksopen]{hyperref} % makes  autounindent no longer work
%\usepackage[all]{hypcap}% causes errors.

\begin{document}


\section*{Using \texttt{linerange} with \texttt{autounindent}}
\lstinputlisting[style=A,linerange=start-stop,autounindent]{Program.cs}


\end{document}

Questions

How to fix these issues?

I've managed to isolate the bug (it has do to with my modification of the character table), but I don't know how to solve it. I hope Heiko will see your question and chime in... — jub0bs, Jun 11 '14 at 20:11

score 5 · Accepted Answer · answered Jun 14 '14 at 16:40

The problem is not the package order. The code that patches listings uses the token register \toks@, a temporary scratch register, as permanent token register. Thus it happens that the token register \toks@ is also used by other packages like hyperref or hypcap and that they can leave garbage afterwards. Then this garbage is called inside lstlisting by the "autounindent" feature causing unwanted surprising effects like error messages.

\toks@ is a temporary scratch token register, which can be used by any package or command. And if the register is cleared before usage, it will not help either, because the settings of the token register in the preamble would be lost for the "autounindent" feature.

The following fix replaces \toks@ by a fresh allocated token register:

\documentclass[12pt]{article}

%TODO
% - handle tabs

\usepackage{accsupp}
\newcommand*{\noaccsupp}[1]{\BeginAccSupp{ActualText={}}#1\EndAccSupp{}}

\usepackage{xcolor}
\usepackage{listings}

\makeatletter

% -------- ugly details --------

\newtoks\mylsttoks

% custom key
\lst@Key{autounindent}f[t]{\lstKV@SetIf{#1}\lst@ifautounindent}

% --- accumulation of tokens ---
\mylsttoks={
    \lst@DefSaveDef{`\ }\lsts@myspace{\processmyspace}
    %\lst@DefSaveDef{`\^^I}\lsts@mytab{\processmytab}
}

% --- egreg's helper function for accumulating tokens ---
\def\add@savedef#1#2{%
  \begingroup\lccode`?=#1\relax
  \lowercase{\endgroup
  \edef\@temp{%
    \noexpand\lst@DefSaveDef{\number#1}%
    \expandafter\noexpand\csname lsts@?\endcsname{\noexpand#2%
      % I had to swap things around to fix a bug here;
      \expandafter\noexpand\csname lsts@?\endcsname}%
      % otherwise, comment highlighting would be broken.
  }}%
  \mylsttoks=\expandafter{\the\expandafter\mylsttoks\@temp}%
}

\count@=33
\loop
    \add@savedef\count@\processchar
  \ifnum\count@<127
  \advance\count@\@ne
\repeat

% --- to keep track of the state ---
\newcount\spacesToGobble
\newcount\spacesGobbledSoFar
\newif\ifafterindent
\newif\ifonBasisLine
\newif\ifPostponeCountToNextLine


\lst@AddToHook{Init}
{%
  \lst@ifautounindent%
    % patch hook macros
    \let\@ddedToInitVarsBOLhook\@@ddedToInitVarsBOLhook%
    \let\@ddedToEOLhook\@@ddedToEOLhook%
    % initialise things
    \global\onBasisLinetrue%
    \global\afterindentfalse%
    \global\spacesToGobble=0%
    \lst@ifincluderangemarker%
      \ifnum9999999=\lst@firstline% (i.e. if firstline option not used)
        \global\PostponeCountToNextLinetrue%
      \fi
    \fi
  \fi
}


\lst@AddToHook{SelectCharTable}
{%
    \lst@ifautounindent%
        \lst@lAddTo\lst@DeveloperSCT{\the\mylsttoks}%
    \fi
}


\lst@AddToHook{InitVarsBOL}{\@ddedToInitVarsBOLhook}
\newcommand\@ddedToInitVarsBOLhook{}
\newcommand\@@ddedToInitVarsBOLhook
{%
  \global\afterindentfalse%
}

\lst@AddToHook{EOL}{\@ddedToEOLhook}
\newcommand\@ddedToEOLhook{}
\newcommand\@@ddedToEOLhook
{%
  \ifonBasisLine%
    \ifPostponeCountToNextLine%
    \else
      \global\onBasisLinefalse%
    \fi
  \fi
  \global\spacesGobbledSoFar=0%
  \global\PostponeCountToNextLinefalse%
}

\lst@AddToHook{DeInit}
{%
  \lst@ifautounindent%
    % undo patches
    \def\@ddedToInitVarsBOLhook{}%
    \def\@ddedToEOLhook{}%
  \fi
}


% --- helper macros ---
\newcommand\processchar
{%
  \global\afterindenttrue%
}

\newcommand\processmyspace
{%
  \ifafterindent%
    \lsts@myspace%
  \else
    \ifonBasisLine%
      \global\advance\spacesToGobble\@ne\relax%
    \else
      \ifnum\spacesGobbledSoFar<\spacesToGobble\relax%
        \global\advance\spacesGobbledSoFar\@ne\relax%
      \else
        \lsts@myspace%
      \fi
    \fi
  \fi
}

\makeatother

% ---------- end of ugly details ----------

\usepackage{filecontents}
\begin{filecontents*}{Program.cs}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Delegate
{
    class Program 
    {
        // start
        sdfgsd fgdsfg  static void Main(string[] args)
        {
            for (int x = 0; x < 10; x++)
                Console.WriteLine(x);
        }
        /* sdflkjhsdf */
        // stop
    }
}
\end{filecontents*}

% --- general style definition ---
\lstdefinestyle{Common}
{   
    language={[Sharp]C},
    numbers=left,
    numbersep=1em,
    numberstyle=\tiny\noaccsupp,
    frame=single,
    framesep=\fboxsep,
    framerule=\fboxrule,
    rulecolor=\color{red},
    xleftmargin=\dimexpr\fboxsep+\fboxrule,
    xrightmargin=\dimexpr\fboxsep+\fboxrule,
    breaklines=true,
    breakindent=0pt,
    tabsize=2,
    columns=flexible,
    %includerangemarker=false,
    rangeprefix=//\ ,
}

\lstdefinestyle{A}
{
    style=Common,
    backgroundcolor=\color{yellow!10},
    basicstyle=\scriptsize\ttfamily,
    keywordstyle=\color{blue}\bf,
    identifierstyle=\color{black},
    stringstyle=\color{red},
    commentstyle=\color{green},
}

\usepackage[colorlinks,bookmarksnumbered,bookmarksopen]{hyperref}
\usepackage[all]{hypcap}

\begin{document}
  \section*{Using \texttt{linerange} with \texttt{autounindent}}
  \lstinputlisting[style=A,linerange=start-stop,autounindent]{Program.cs}
\end{document}

That's great. I had a feeling that \toks@ was the culprit. I think egreg's answer should probably be amended accordingly. What do you think? — jub0bs, Jun 14 '14 at 17:12

score 3 · Answer 2 · answered Jun 11 '14 at 15:56

3

Change the order: if you start your document with

\documentclass[12pt]{article}
\usepackage[colorlinks,bookmarksnumbered,bookmarksopen]{hyperref}
\usepackage[all]{hypcap}

and leave the rest as it is, you should get the expected result.

answered Jun 11 '14 at 15:56

Clément

5,591

But those packages must be loaded last, according to their manual. :D – kiss my armpit Jun 11 '14 at 15:57
Yep, I was trying to find where it was mentionned in the hyperref manual. I agree this solution can mess up in a larger document, but for your MWE, it works like a charm. – Clément Jun 11 '14 at 15:59
@FifaEarthCup2014 (your nickname chages every day !) : could you specify why this "charming" solution is not enough ? – Clément Jun 14 '14 at 16:25
1

Ok, Heiko Oberdiek made it clear that the problem was deeper… – Clément Jun 14 '14 at 17:02
It is of course not enough for the case where the order does matter. – kiss my armpit Jun 14 '14 at 19:09

autounindent cannot work together with hyperref and hypcap

Case 1

Case 2

Case 3

MWE

Questions

2 Answers2

Linked