Difference in compilation time between two slightly different versions

Question

This question is a byproduct of earlier questions: Creating a list of labels automatically for use by datatool and Automatically creating a table from datatool using references in the text.

Here is a LaTeX file using datatool. There are two slightly different versions of this: I'll call these VERSION 1 and VERSION 2. VERSION 1 is due to egreg, VERSION 2 is due to DavidCarlisle.

For reasons that are not at all clear to me, VERSION 2 is much slower to compile than VERSION 1. VERSION 1 consistently takes around 10 seconds. VERSION 2 varies, but takes at least twice that, upwards of 20 seconds. Superficially the two versions look very similar.

The different VERSIONs correspond to two blocks in the following file, each starting with \makeatletter and makeatother.

When testing, remember to remove the aux file when switching versions.

What is causing this significant difference in compilation speed?

#########################################################################

\documentclass{letter}
\usepackage[verbose]{datatool}
\usepackage{datagidx}
\usepackage{longtable}
\usepackage{lipsum}
\usepackage{array}
\usepackage{etoolbox}
\usepackage{url}
\setlength{\oddsidemargin}{-0.10in}
\setlength{\evensidemargin}{-0.10in}
\setlength{\topmargin}{-0.3in}
\setlength{\headsep}{0.2in}
\textheight=8.8in
\textwidth=6.7in
\newcounter{tabenum}\setcounter{tabenum}{0}
\newcommand{\colhead}[1]{\multicolumn{1}{>{\bfseries}l}{#1}}
\newcommand{\nextnuml}[1]{\refstepcounter{tabenum}\thetabenum.\label{#1}}

% VERSION 1

\makeatletter
\let\oldref\ref
\def\ref#1{%
  \immediate\write\@auxout{%
    \string\gappto\string\ReferencedIDs{#1,}%
  }%
  \oldref{#1}%
}
\def\ReferencedIDs{}
\makeatother

% VERSION 2

% \makeatletter
% \AtBeginDocument{\providecommand\ReferencedIDs{}}
% \AtEndDocument{\immediate\write\@auxout{\gdef\string\ReferencedIDs{\ReferencedIDs}}}
% \let\oldref\ref
% \def\ref#1{%
%   \g@addto@macro\ReferencedIDs{,#1}%
%   \oldref{#1}%
% }
% \makeatother

\usepackage{fouriernc}
\address{Some Address\\ Some Place\\Email: foo@bar.com}
\signature{(Somebody)}
\newcommand*{\checkmissing}[1]{\DTLifnull{#1}{}{#1}}

\newcommand{\PrintDocTableParekh}[3][]{%
 % #1 = list of rowIDs
 % #2 = database to search
 % #3 =caption
  \begin{longtable}{r l p{1.5in} c c p{2.5in}}
    \caption{#3}\\
   & \colhead{Date} & \colhead{Filename} & \colhead{From} & \colhead{To} & \colhead{Subject}\\\hline\endhead
    \DTLforeach
    [\ifblank{#1}{\boolean{true}}{\DTLisSubString{#1}{\RowID}}]
    % [%
    % \(%
    % \equal{#1}{}\AND\DTLisSubString{\ReferencedIDs}{\RowID}%
    % \)%
    % \OR\(
    % \DTLisSubString{#1}{\RowID}\AND\DTLisSubString{\ReferencedIDs}{\RowID}%
    % \)%
    % ]
    {#2}{%
      \RowID=RowID,%
      \Date=Date,%
      \Filename=Filename,%
      \From=From,%
      \To=To,%
      \Subject=Subject%
    }{%
      \nextnuml{\RowID} & \Date & {\bfseries\expandafter\url\expandafter{\Filename} } & \checkmissing{\From} & \checkmissing{\To} & \Subject \\
    }%
  \end{longtable}
}%

\makeatletter
\newcommand{\AddToTable}[2]{%
\DTLforeach*{#2}{\newDBRowID=RowID,\newDBDate=Date,\newDBFilename=Filename,\newDBFrom=From,\newDBTo=To,\newDBSubject=Subject}{
  \DTLnewrow{#1}
  {\let\DTLnewdbentry\relax % Avoid expansion of \DTLnewdbentry when using \protected@xdef below
   \protected@xdef\insertnewdbentry{%
     \DTLnewdbentry{#1}{RowID}{\newDBRowID}%
     \DTLnewdbentry{#1}{Date}{\newDBDate}%
     \DTLnewdbentry{#1}{Filename}{\newDBFilename}%
     \DTLnewdbentry{#1}{From}{\newDBFrom}%
     \DTLnewdbentry{#1}{To}{\newDBTo}%
     \DTLnewdbentry{#1}{Subject}{\newDBSubject}%
  }}\insertnewdbentry
}}
\makeatother

\begin{filecontents*}{newa.csv}
smyt.bs.ie,Mar 2004-2013 (incl.),something.txt,,,,subject
ct.bs.ie,Mar 2004-2013 (incl.),something.txt,,,,subject
cf.bs.ie,Mar 2004-2013 (incl.),something.txt,,,,subject
ismail.bs.ie,Mar 2004-2013 (incl.),something.txt,,,,subject
mh.bs.ie,Mar 2004-2013 (incl.),something.txt,,,,subject
smyt.it,2007-2013,something.txt,,,,subject
ismail.it,2008-2013,something.txt,,,,subject
familytree,26 Feb 2014,something.txt,,,,subject
trusts,26 Feb 2014,something.txt,,,,subject
1952.02.19,19 Feb 1952,something.txt,,,,subject
1994.03.26,26 Mar 1994,something.txt,,,,subject
1998.09.14,14 Sep 1998,something.txt,FROM,TO,,subject
1998.09.16,16 Sep 1998,something.txt,FROM,TO,,subject
2001.05.23,23 May 2001,something.txt,FROM,TO,,subject
2007.06.10,10 Jun 2007,something.txt,FROM,TO,,subject
2010.11.04,04 Nov 2010,something.txt,,,,subject
2011.05.09,09 May 2011,something.txt,FROM,TO,Y,subject
2011.09.09.slm,09 Sep 2011,something.txt,FROM,TO,Y,subject
2011.09.26.fm,26 Sep 2011,something.txt,FROM,TO,Y,subject
2011.09.26.slm,26 Sep 2011,something.txt,FROM,TO,Y,subject
2011.09.26.zhm,26 Sep 2011,something.txt,FROM,TO,,subject
2011.11.21.slm,21 Nov 2011,something.txt,FROM,TO,Y,subject
2011.11.21.fm,21 Nov 2011,something.txt,FROM,TO,Y,subject
2011.11.22,22 Nov 2011,something.txt,FROM,TO,Y,subject
2011.11.23,23 Nov 2011,something.txt,FROM,TO,Y,subject
2012.01.25,25 Jan 2012 (?),something.txt,FROM,TO,,subject
2012.12.07,07 Dec 2012,something.txt,FROM,TO,,subject
2013.01.14,14 Jan 2013,something.txt,FROM,TO,,subject
2013.08.29,29 Aug 2013,something.txt,FROM,TO,Y,subject
2013.09.09.fm,09 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.09.rm,09 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.10.slm,10 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.10.fm,10 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.10.slm2,10 Sep 2013,something.txt,FROM,TO,,subject
2013.09.11.slm1,11 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.11.fm,11 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.11.slm2,11 Sep 2013,something.txt,FROM,TO,Y,subject
2013.09.16,16 Sep 2013,something.txt,FROM,TO,,subject
2013.09.17,17 Sep 2013,something.txt,FROM,TO,,subject
2013.09.23,23 Sep 2013,something.txt,FROM,TO,,subject
2013.10.05.powai,05 Oct 2013,something.txt,,,,subject
2013.10.05.kanjur,05 Oct 2013,something.txt,,,,subject
2013.10.06,06 Oct 2013,something.txt,FROM,TO,,subject
2013.10.14.kma,14 Oct 2013,something.txt,FROM,TO,,subject
2013.10.14.ks,14 Oct 2013,something.txt,FROM,TO,,subject
2013.10.16,16 Oct 2013,something.txt,FROM,TO,,subject
2013.10.17,17 Oct 2013,something.txt,FROM,TO,,subject
2013.10.18,18 Oct 2013,something.txt,FROM,TO,,subject
2013.10.22,22 Oct 2013,something.txt,FROM,TO,,subject
2013.10.23,23 Oct 2013,something.txt,FROM,TO,,subject
2013.10.24,24 Oct 2013,something.txt,FROM,TO,,subject
2013.10.25.zhm,25 Oct 2013,something.txt,FROM,TO,,subject
2013.10.25.fm1,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.25.rm1,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.25.fm2,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.25.rm2,25 Oct 2013,something.txt,FROM,TO,Y,subject
2013.10.28,28 Oct 2013,something.txt,FROM,TO,,subject
2013.10.28.zhm,28 Oct 2013,something.txt,,,,subject
2013.10.30,30 Oct 2013,something.txt,FROM,TO,,subject
2013.11.11,11 Nov 2013,something.txt,FROM,TO,,subject
2013.11.12,12 Nov 2013,something.txt,FROM,TO,,subject
2013.11.13,13 Nov 2013,something.txt,FROM,TO,,subject
2013.11.25,25 Nov 2013,something.txt,FROM,TO,,subject
2013.12.05.ta,05 Dec 2013,something.txt,FROM,TO,,subject
2013.12.05.slm,05 Dec 2013,something.txt,FROM,TO,,subject
2014.01.15,15 Jan 2014,something.txt,FROM,TO,,subject
2014.02.24,24 Feb 2014,something.txt,FROM,TO,,subject
2014.02.26,26 Feb 2014,something.txt,,,,subject
2014.04.02,02 Apr 2014,something.txt,FROM,TO,,subject
2014.04.10,10 Apr 2014,something.txt,FROM,TO,,subject
2014.04.15,15 Apr 2014,something.txt,FROM,TO,,subject
2014.07.18.tm,18 Jul 2014,something.txt,FROM,TO,,subject
2014.07.18.ta,18 Jul 2014,something.txt,FROM,TO,,subject
2014.07.24,24 Jul 2014,something.txt,FROM,TO,,subject
2014.09.05.tm,05 Sep 2014,something.txt,FROM,TO,,subject
2014.09.05.ta,05 Sep 2014,something.txt,FROM,TO,,subject
2014.09.05.slm,05 Sep 2014,something.txt,FROM,TO,,subject
2014.10.17,17 Oct 2014,something.txt,FROM,TO,,subject
2014.11.12.tm,12 Nov 2014,something.txt,FROM,TO,,subject
2014.11.14,14 Nov 2014,something.txt,FROM,TO,,subject
2014.11.17,17 Nov 2014,something.txt,FROM,TO,,subject
2015.01.06,06 Jan 2015,something.txt,FROM,TO,,subject
\end{filecontents*}

\begin{document}

\def\today{9th January, 2014}
\begin{letter}{
    Someone\\
    Somewhere\\
    Subject: Some stuff
}

  \opening{Dear Someone}

  \lipsum[1]. Here is a ref - [\ref{2014.11.14}].

  \lipsum[2]. Here is a ref - [\ref{2014.11.14}].

  \lipsum[3] Here is a ref - [\ref{2014.11.17}].

  \lipsum[4]


  \lipsum[5] Here is a ref - [\ref{2014.11.14}].

  \lipsum[6] Here is a ref - [\ref{2014.07.18.tm}].

  \lipsum[7]

  \lipsum[8] Here is a ref - [\ref{2014.11.12.tm}].

  \lipsum[9]. See references
    ~[\ref{2013.10.05.kanjur}], [\ref{2013.10.05.powai}].
    [\ref{2013.10.06}], [\ref{2013.10.28.zhm}],
    [\ref{2013.10.17}], [\ref{2013.10.22}],

    \lipsum[10]. See references
    [\ref{2013.10.14.ks}], [\ref{2013.10.06}]
    [\ref{2013.10.17}], [\ref{2013.10.22}], [\ref{2013.10.24}],

    \lipsum[10]. See references
    [\ref{2013.10.28}], [\ref{2013.10.30}]

    \lipsum[10]. See references [\ref{2013.11.11}]

    \lipsum[11]. See references [\ref{2014.09.05.tm}]

    \lipsum[12]. See references
    [\ref{2014.07.18.tm}] [\ref{2014.11.12.tm}]

   \lipsum[13]. See references
    [\ref{2014.11.17}], [\ref{2015.01.06}], [\ref{2015.01.06}]

\closing{Yours Sincerely,}

\DTLloaddb[noheader,keys={RowID,Date,Filename,From,To,Email,Subject}]{newaDB}{newa.csv}
\PrintDocTableParekh[\ReferencedIDs]{newaDB}{Documents}\renewcommand\ReferencedIDs{}

\end{letter}
\end{document}

score 3 · Accepted Answer · answered Feb 12 '15 at 00:40

3

You need to clear the list but it's inside the group from letter so replace

\renewcommand\ReferencedIDs{}

by

\gdef\ReferencedIDs{}

then the times are the same

answered Feb 12 '15 at 00:40

David Carlisle

757,742

Yes, this works, but I don't understand the point at all. Can you elaborate, please? In particular, what does "it's inside the group from letter" mean? What is "it", and what group? I assume "letter" refers to the letter class. – Faheem Mitha Feb 12 '15 at 00:49
compare the aux file after several runs, if you only locally clear it (with my version) the list gets longer and longer every time, I mean the resetting to {} is inside the local group from letter not the class the environment alll environments (except document) are a group like {...} – David Carlisle Feb 12 '15 at 00:54
Yes, I see, You mean the \renewcommand\ReferencedIDs{} isn't actually resetting the \renewcommand\ReferencedIDs{} because it is only redefining it locally? So there is some kind of namespace in effect here? – Faheem Mitha Feb 12 '15 at 01:06
@FaheemMitha not really a namespace just local scope if you go {\renewcommand\foo{}} then it has no effect as the value is restored at the } – David Carlisle Feb 12 '15 at 01:08
So, this raises two issues. First, the timing is obviously very sensitive to the length of \ReferencedIDs. This in turn suggests that the macro \DTLisSubString is very slow. However, since all this macro does is do a substring check, it should be possible to replace it with something that is not so slow. Second, neither of these methods ensure that the set of strings in \ReferencedIDs is unique, which presumably further slows down the compilation time. Should I post these issues as a separate question? – Faheem Mitha Feb 12 '15 at 07:18
(from the Datatool manual), \DTLisSubString{hstringi}{hsubstringi} This checks if hsubstringi is contained in hstringi – Faheem Mitha Feb 12 '15 at 07:20
Possibly. I haven't looked at the code there to see if it could be improved. – David Carlisle Feb 12 '15 at 09:39

Difference in compilation time between two slightly different versions

1 Answers1

Linked