\lstinputlisting always parses entire source file

Question

I'm using a lot of \lstinputlisting commands to display snippets from a large source file.

For example:

\lstinputlisting[firstline=8244,lastline=8250]{source.c}
\lstinputlisting[firstline=9244,lastline=9250]{source.c}

and so on.

Compile time is terrible and I'm pretty sure its because listings is parsing the entire file up to the firstline for each \lstinputlisting command. Compile time is fine when I do something like this:

\lstinputlisting[firstline=14,lastline=34]{source.c}
\lstinputlisting[firstline=24,lastline=44]{source.c}

Is there a way of avoiding parsing the entire file each time?

Answer 1

I try a simple benchmark, use TeX to copy the desired range to a temp file each time then use lstinputlisting to input that file is actually slower.

%! TEX program = lualatex
\documentclass{article}
\usepackage{listings}
\usepackage{l3benchmark}
\ExplSyntaxOn
\NewDocumentCommand \CopyPartialFile {mmmm} {
\ior_open:Nn \g_tmpa_ior {#1}
\iow_open:Nn \g_tmpa_iow {#2}
\prg_replicate:nn {#3} {
    \ior_str_get:NN \g_tmpa_ior \l_tmpa_str
}
\prg_replicate:nn {#4} {
    \ior_str_get:NN \g_tmpa_ior \l_tmpa_str
    \iow_now:Nx \g_tmpa_iow {\l_tmpa_str}
}
\ior_close:N \g_tmpa_ior
\iow_close:N \g_tmpa_iow
}
\NewDocumentCommand \Repeat {mm} { \prg_replicate:nn {#1} {#2} }
\ExplSyntaxOff
\begin{document}
\ExplSyntaxOn
\benchmark_tic:
\ExplSyntaxOff
\Repeat{100}{%
\lstinputlisting[firstline=5001,lastline=5020]{source.c}%
}
\ExplSyntaxOn
\benchmark_toc:
\ExplSyntaxOff
\ExplSyntaxOn
\benchmark_tic:
\ExplSyntaxOff
\Repeat{100}{%
\CopyPartialFile{source.c}{tmp.c}{5000}{20}%
\lstinputlisting{tmp.c}%
}
\ExplSyntaxOn
\benchmark_toc:
\ExplSyntaxOff
\end{document}

(which makes it likely that listings is not actually parse these lines, it only read and skip through them)

TeX does not have the feature to "seek to line X of a file" (this is actually fundamentally impossible), so an obvious solution here is to store the whole file into memory and extract the relevant lines when needed.

This solution defines one control sequence for each source code file line to store the line content, then when needed write the content to a temporary file and \lstinputlisting it.

%! TEX program = lualatex
\documentclass{article}
\usepackage{listings}
\usepackage{l3benchmark}
\ExplSyntaxOn
% ======== this block of code read the file source.c and store it into several control sequences ========
\ior_open:Nn \g_tmpa_ior {source.c}
\int_zero:N \l_tmpa_int
\ior_str_map_variable:NNn \g_tmpa_ior \l_tmpa_str {
    \int_incr:N \l_tmpa_int
    \str_set_eq:cN {l__robert_fileline_ \int_use:N \l_tmpa_int _str} \l_tmpa_str
}
\ior_close:N \g_tmpa_ior
\cs_new:Npn __robert_use_file_line:n #1 {
    \use:c {l__robert_fileline_ #1 _str} ^^J
}
% ======== this defines a function \WritePartialToFile{targetfilename.c}{firstline}{lastline}
% which writes lines firstline-lastline of source.c to targetfilename.c ========
\NewDocumentCommand \WritePartialToFile {mmm} {
    \iow_open:Nn \g_tmpa_iow {#1}
    \iow_now:Nx \g_tmpa_iow { \int_step_function:nnN {#2} {#3} __robert_use_file_line:n }
    \iow_close:N \g_tmpa_iow
}
% ======== auxiliary function for benchmarking ========
\NewDocumentCommand \Repeat {mm} { \prg_replicate:nn {#1} {#2} }
\ExplSyntaxOff
\begin{document}
\ExplSyntaxOn
\benchmark_tic:
\ExplSyntaxOff
% ======== example usage of \WritePartialToFile to input a part of the file ========
\Repeat{100}{%
\WritePartialToFile{tmp.c}{5001}{5020}%
\lstinputlisting{tmp.c}%
}
\ExplSyntaxOn
\benchmark_toc:
\ExplSyntaxOff
\end{document}

It's about 3× faster than the original code. (because of hash collision it's not as fast as it could be, but good enough)