Implementation of property list in expl3

Question

For some preprocessing task, I would like to use an expl3 property list. Since the number of items in that list would be in the thousands - index entries for a textbook - I was wondering how time-efficient this might be. So, I did some testing:

\documentclass{article}

\usepackage{expl3,xparse, fp}

% timing adapted from http://tex.stackexchange.com/questions/9087
\def\startTimer{\pdfresettimer}
\def\stopTimer{%
    \FPdiv\result{\the\pdfelapsedtime}{65536}
    \FPround\result{\result}{3}\par 
    Time (seconds): \result \par}

\ExplSyntaxOn

\prop_new:N \sg_user_mapping
\cs_generate_variant:Nn \prop_gput:Nnn { Nxx }

\int_new:N  \i_target
\int_new:N  \i_counter

\NewDocumentCommand{\dostuff}{m} {
    Adding\ #1\ keys\ to\ user\ mapping\par
    %\prop_gclear:N \sg_user_mapping
    \startTimer
    \int_add:Nn \i_target {#1}
    \int_zero:N \i_counter % Hopefully obvious!

    \int_while_do:nn { \i_counter < \i_target }
    {
        \edef\wurst{\int_to_arabic:n \i_counter}
        \prop_gput:Nxx \sg_user_mapping {hans\wurst} {0}
        \int_incr:N \i_counter
    }
    \stopTimer\par
}

\ExplSyntaxOff

\begin{document}

\ttfamily

\dostuff{10} 
\bigskip

\dostuff{100} 
\bigskip

\dostuff{1000} 
\bigskip

\dostuff{10000} 

\end{document}

Here is what I get:

So, that is a clear case of O^2; I'm guessing that the entire list gets scanned before a new entry is inserted.

I'm wondering if it would be viable to use some behind-the-scenes trickery in order to exploit pdftex's built-in hashtables. For example, for each key-value pair,

1. sanitize the key by stripping out any characters that can't be part of a command name: #foo%^bar!- baz? becomes foobarbaz

2. create a property list (as currently implemented) under the name \pl_userland_foobarbaz

3. store #foo%^bar!- baz? => whatever value in that list.

I actually tried that, and the code, while obviously slower for small numbers of keys, does scale with O^1 as expected, and it reaches parity with the straight, simple property list at about 1000 keys. I hesitate to post that code here, since I'm only beginning with expl3, and the pros could probably do a better job of it; it was only an experiment.

Of course my suggestion brings up the question how many such keys and property list instances pdfTeX will be able to accommodate before starting to throw the toys out of the pram? Alternatively, it may be possible to implement a proper hash table in TeX itself; but I'm not up to snuff in that regard.

Answer 1

First off, it's important to emphasis that only the interfaces are documented for expl3 and that the implementation may change with no notice provided the interfaces are unchanged. (Over my time on the team we've had a couple of different implementations here, and there was at least one more in the past.) The current implementation uses a single macro to hold the entire property list (the previous one uses a single toks, and before that there was some csname oddness: before my time!).

As discussed in detail in How to implement (low-level) arrays in TeX, the most efficient way to implement an array for fast lookup in TeX is to use the hash table and make sure your control sequence names don't cause too many clashes. (Basically, don't use numbers!)

However, this was not the key design driver in the design of the current prop data type in expl3. The code has developed since before the release of LaTeX2e (1994) and there certainly was another worry: memory. Today there are many more csnames available in a standard TeX binary, so the pressure to preserve them is reduced (though it is still possible to run out, depending on the use case). Twenty years ago the pressure was much more severe: famously, there were then only about 50 'spare' names after loading the LaTeX2e kernel when using TeX on MS-DOS. That factor is one of those here: the implementation has altered over time, but not simply burning through names remains one factor.

There are also other considerations. Whilst random lookup is fast using a hash table approach, copying a property list is easier when 'behind the scenes': it's a one-shot not a loop. There's also the question of mapping over each entry in a list: for the current set up, we can expand the underlying macro and then work through the entries on the stack. Using separate macros for each entry makes that process more tricky.

The team have discussed using different implementations, but to date there's been no strong push to do so from the point of view of 'use cases' (certainly for the team's own code). Balancing up the various use cases is non-trivial, and we have discussed an 'object' type which would have a more than one 'back end' and would switch between them (perhaps via a user setting).