Long List of Files Names with Paths

Question

I need to keep a list of files (max of around 500) along with the path.

Based on the fact that each file name and path could be up to 100 characters, should I just save it in a comma separated list using \g@addto@macro, or should I use the datatool package, or something else?

My main concern is any limitations I might run into.

In case it matters, the list processing that I am contemplating is to simply iterate through each file via pgf's \foreach to

1. Count the types of files.
2. Perhaps keep create separate sub-lists based on the file attributes/contents.
3. Generate a PDF with links to the files.

References

• How keep a running list of strings and then process them one at a time

Answer 1

It is generally more efficient to arrange that the list is executable rather than use a comma separated list (which needs to be parsed), traditionally LaTeX uses \@elt for this but any macro will do, the idea basically is that instead of

\def\foo{aaa,bbb,ccc,ddd}
\foreach \tmp in \foo ...{\mymacro{\tmp}}

you do

\def\foo{\mymacro{aaa}\mymacro{bbb}\mymacro{ccc}\mymacro{ddd}}

so it takes a little bit more storage space, but to execute it you just do

\foo

and to get different processing you locally define \mymacro so

\count@=0
\def\mymacro#1{\advance\count@ 1 }
\foo

will leave a count of the number of items in \count@

Answer 2

(David can obviously type faster than me! But then, he had less to type.)

Here's a possible method using datatool:

\documentclass{article}

\usepackage{etoolbox}
\usepackage{datatool}
\usepackage{hyperref}

\DTLnewdb{files}

\newcommand*{\extlist}{}

\makeatletter
\newcommand*{\getfileext}[2]{%
 % Get the file extension
 \def\thisfile{#2}%
 \filename@parse\thisfile
 \let#1\filename@ext
}

\newcommand*{\addfile}[1]{%
 \getfileext{\thisfileext}{#1}%
 % Has a count register been defined?
 \ifcsundef{\thisfileext @count}%
 {%
   \expandafter\global
   \expandafter\newcount\csname \thisfileext @count\endcsname
   \listeadd{\extlist}{\thisfileext}%
 }%
 {}%
 \expandafter\global
 \expandafter\advance\csname \thisfileext @count\endcsname by 1\relax
 \DTLnewrow{files}%
 \dtlgetrow{files}{\DTLrowcount{files}}%
 \dtlappendentrytocurrentrow{File}{#1}%
 \dtlappendentrytocurrentrow{Ext}{\thisfileext}%
 \@onelevel@sanitize\thisfile
 \dtlappendentrytocurrentrow{Link}{\noexpand\url{\thisfile}}%
 \dtlrecombine
}
\makeatother


\addfile{/home/me/documents/paper1/paper1.tex}
\addfile{/home/me/documents/paper1/images/img1.png}
\addfile{/home/me/documents/paper1/images/img2.jpg}
\addfile{/home/me/documents/paper1/images/img2.png}
\addfile{/home/me/documents/paper2/paper2.tex}
\addfile{/home/me/documents/paper2/images/img1.png}
\addfile{/home/me/documents/paper2/images/img2.jpg}
\addfile{/home/me/documents/paper2/images/img2.png}

\begin{document}

File types:

\renewcommand{\do}[1]{Extension: #1. 
Count: \number\csname#1@count\endcsname\par}
\dolistloop{\extlist}

\DTLforeach*{files}{\File=File,\Ext=Ext,\Link=Link}{%
  File: \File\par
  Ext: \Ext\par
  Link: \Link\par
}

\end{document}

This produces:

Notes:

1. I sanitized the path name using \@onelevel@sanitize for use in \url, but this probably isn't needed, unless you have funny characters in your file name, but that might cause a problem for the database as well, so it's probably overkill.
2. If you want to add a row to a database, it's faster to use the \dtlgetrow method than using \DTLnewdbentry. (Since you have 500 entries, you may prefer the faster less tidy option that the slower neater option.)

Answer 3

I made a file list of /usr/local/texlive/2013/texmf-dist/tex/latex/base, the first lines of which look like

/usr/local/texlive/2013/texmf-dist/tex/latex/base/alltt.sty
/usr/local/texlive/2013/texmf-dist/tex/latex/base/ansinew.def
/usr/local/texlive/2013/texmf-dist/tex/latex/base/applemac.def
/usr/local/texlive/2013/texmf-dist/tex/latex/base/article.cls
/usr/local/texlive/2013/texmf-dist/tex/latex/base/article.sty
/usr/local/texlive/2013/texmf-dist/tex/latex/base/ascii.def
/usr/local/texlive/2013/texmf-dist/tex/latex/base/bezier.sty

I saved the list in the file filelist.txt. Then I prepared this document:

\documentclass{article}

\usepackage{xparse}

\ExplSyntaxOn

\NewDocumentCommand{\MakeFileList}{mm}
 {%1 = list name, #2 = file name
  \grill_make_file_list:nn { #1 } { #2 }
 }

\NewDocumentCommand{\CountItemsInList}{ O{} m }
 {
  \grill_count_items_in_list:nn { #1 } { #2 }
 }

\ior_new:N \l_grill_file_list_stream
\seq_new:N \l__grill_file_list_temp_seq

\cs_new_protected:Npn \grill_make_file_list:nn #1 #2
 {
  \seq_new:c { g_grill_file_list_#1_seq }
  \ior_open:Nn \l_grill_file_list_stream { #2 }
  \ior_map_inline:Nn \l_grill_file_list_stream
   {
    \seq_gput_right:cn { g_grill_file_list_#1_seq } { ##1 }
   }
 }

\cs_new_protected:Npn \grill_count_items_in_list:nn #1 #2
 {
  \seq_clear:N \l__grill_file_list_temp_seq
  \seq_map_inline:cn { g_grill_file_list_#2_seq }
   {
    \tl_if_in:nnT { ##1 } { #1 }
     {
      \seq_put_right:Nn \l__grill_file_list_temp_seq { ##1 }
     }
   }
  \seq_count:N \l__grill_file_list_temp_seq
 }

\ExplSyntaxOff

\begin{document}

\MakeFileList{LaTeX}{filelist.txt}

There are \CountItemsInList{LaTeX} files in total.

There are \CountItemsInList[.tex]{LaTeX} \texttt{.tex} files.

There are \CountItemsInList[.def]{LaTeX} \texttt{.def} files.

There are \CountItemsInList[.ltx]{LaTeX} \texttt{.ltx} files.

There are \CountItemsInList[.sty]{LaTeX} \texttt{.sty} files.

\end{document}

There are of course other methods for building the list. A finer control over the file types could be obtained with regular expressions. The same method used for counting the items could be used for building new lists based on the file extension or for creating links.

Here's the result.

Just to prove the correctness of the result, here's a set of shell commands:

> ls /usr/local/texlive/2013/texmf-dist/tex/latex/base/ | wc -l
     149
> ls /usr/local/texlive/2013/texmf-dist/tex/latex/base/*.tex | wc -l
      12
> ls /usr/local/texlive/2013/texmf-dist/tex/latex/base/*.def | wc -l
      32
> ls /usr/local/texlive/2013/texmf-dist/tex/latex/base/*.ltx | wc -l
       6
> ls /usr/local/texlive/2013/texmf-dist/tex/latex/base/*.sty | wc -l
      30