Built-in Mathematica data: are they cached? how to speed up the loading?

Question

Mathematica has numerous built-in data for science and math. Every first call to a data function in a new Mathematica session will "re-initialize indices" (download the data). Is there a way to store this data locally to save time on the next data call after Mathematica is restarted?

======== Edit: further thoughts with Leonid's answer ========

After Leonid posted his answer, I decided to update this question – the title and content. I would ask about this anyway – so may as well do it here. The point is not in whether data were cached or not, but really in a method to speed up the load process. I just confused caching and loading. I do not want to take away “best answer” from Spartacus, because he did answer what I asked ideally, thanks. But thank you very much Leonid for posting your answer too. +1 to both of you.

Answer 1

Please note that parts of the explanations and initialization code is shown together with the main function code, as a single large code block. I will appreciate any help on this matter - I am quite confused, perhaps overlooking something obvious here.

---

Preamble

While the question has been answered already, the delays with loading built-in data are a pretty serious problem, to my mind. This question prompted me to write a tiny framework, which will considerably speed it up, pretty much for arbitrary built-in data. The techniques involved will be a mix of memoization, some meta-programming, Block trick, and .mx files (DumpSave - Get).

The code

The first ingredient is the symbol-cloning functionality, which is described here (the function clone and related). I will reproduce this here to have a self-contained answer:

Clear[GlobalProperties];
GlobalProperties[] :=
  {OwnValues, DownValues, SubValues, UpValues, NValues, FormatValues, 
      Options, DefaultValues, Attributes};


Clear[unique];
unique[sym_] :=
 ToExpression[
    ToString[Unique[sym]] <> 
       StringReplace[StringJoin[ToString /@ Date[]], "." :> ""]];


Attributes[clone] = {HoldAll};
clone[s_Symbol, new_Symbol: Null] :=
  With[{clone = If[new === Null, unique[Unevaluated[s]], ClearAll[new]; new],
        sopts = Options[Unevaluated[s]]},
     With[{setProp = (#[clone] = (#[s] /. HoldPattern[s] :> clone)) &},
        Map[setProp, DeleteCases[GlobalProperties[], Options]];
        If[sopts =!= {}, Options[clone] = (sopts /. HoldPattern[s] :> clone)];
        HoldPattern[s] :> clone]]

Here are the functions of the "framework". The following is just a helper function to create a file name string:

ClearAll[makeFileName];
Options[makeFileName] = {
   TargetFileName :> Automatic,
   TargetDirectory :> $TemporaryDirectory
 };
makeFileName[dataFunction_Symbol, opts : OptionsPattern[]] :=
  With[{dir = OptionValue[TargetDirectory], fname = OptionValue[TargetFileName]},
      FileNameJoin[{
          dir, 
          If[fname =!= Automatic, 
             fname, 
             (* else *)
             "MemoizedData_" <> ToString[dataFunction] <> ".mx"
          ]}
      ]];

This is the main function. It will create dynamic environment (it returns a closure), inside which the values of a given data-holding function will be memoized. More details below.

ClearAll[generateMemoEnvironment];
SetAttributes[generateMemoEnvironment, HoldFirst];
Options[generateMemoEnvironment] = {
    StorageSymbol :> memoData,
    Sequence @@ Options[makeFileName]
 };
generateMemoEnvironment[env_Symbol, dataFunction_, opts : OptionsPattern[]] :=
   With[{memoSymbol = dataFunction /. clone[dataFunction],
      storageSymbol = OptionValue[StorageSymbol]
   },
   With[{fullname = makeFileName[ dataFunction, opts]},
       storageSymbol /: Save[storageSymbol[dataFunction]] :=
          DumpSave[fullname, {env, memoSymbol, storageSymbol}];
   ];
   env = 
     Function[
        code,
        Block[{dataFunction},
           dataFunction[args___] :=  storageSymbol[dataFunction][args];
           storageSymbol[dataFunction][args___] :=
               storageSymbol[dataFunction][args] = memoSymbol [args];
           code
        ],
        HoldAll]
];

How it works

What happens here is that, when we call generateMemoEnvironment, first the clone for a given symbol (e.g. for ChemicalData) is created (a clone is a symbol with identical global properties. Try f[x_]:=x;f[x_,y_]:=(x+y);clone[f,g] and look at definitions for g, to see what it does. The tricky point here is that the line dataFunction /. clone[dataFunction] calls dataFunction, which allows it to auto-load first. Otherwise, the clone would be empty. The main idea is that now, since we cloned the symbol, we can use Block to Block the main (original) symbol, and temporarily make it memoizing inside Block. Memoization is however done via an intermediate symbol memoSymbol. The main symbol which is kind of a "handle" for everything is given by the StorageSymbol option (I made it default to memoData) - it can be the same for all types of data.

Illustration and workflow

Let me now illustrate how to use this beast. Assume that you loaded the above code on a fresh kernel. Now, we create our dynamic environment:

generateMemoEnvironment[withMemoChemicalData, ChemicalData];

We can check that now the symbol withMemoChemicalData holds a pure function (closure):

 withMemoChemicalData

 (*
  ==>  Function[code$,Block[{ChemicalData},ChemicalData[args$___]:=
       memoData[ChemicalData][args$];memoData[ChemicalData][args$___]:=
      memoData[ChemicalData][args$]=ChemicalData$568201222222151718750[args$];
      code$],HoldAll]
 *)

Now, we execute some code within it (twice):

withMemoChemicalData[
   res1 = ChemicalData[#,"MolecularWeight"]&/@ChemicalData[]
];//Timing

(*
  ==> {20.375,Null}
*)

and again:

withMemoChemicalData[
    res2 = ChemicalData[#,"MolecularWeight"]&/@ChemicalData[]
];//Timing

(*
 ==> {0.125,Null}
*)

The timing difference reflects memoization at work. Note that, you can execute arbitrary code involving ChemicalData inside the environment, and memoization will work!

res1===res2

(*
 ==> True
*)

At the same time, because we used Block, the function ChemicalData did not receive any global definitions (which is easy to check) - which was one of the goals. This means, that our local modifications inside withMemoChemicalData present no danger whatsoever for the rest of the system, and / or other code which may be using ChemicalData from the outside of our environment.

To save the memoized values, you just call Save (I may get flamed for overloading it to work with a single argument, but that can be easily avoided if so desired):

Save[memoData[ChemicalData]];//Timing

(*
  ==> {0.454,Null}
*)

Now, here comes the main point: once you saved it once, you no longer need generateMemoEnvironment - you just need may be makeFileName, to construct the file name automatically. Let us now quit the kernel:

Quit

Now, we execute on a fresh kernel:

Get[makeFileName[ChemicalData]]

and we are ready to go:

withMemoChemicalData[
   res1 = ChemicalData[#,"MolecularWeight"]&/@ChemicalData[]
];//Timing


(*
 ==> {0.125,Null}
*)

Length[res1]

(*
 ==> 43987
*)

Moreover, you can now keep calling properties you did not call before, and those will be also memoized automatically - just wrap your code in withMemoChemicalData. All you have to do is to call Save[memoData[ChemicalData]] periodically, to update the file with newly memoized definitions.

Summary

I presented a tiny framework which may allow hundred-fold speed-ups when working with built-in data. The main ideas involved dynamic environments, metaprogramming, memoization, encapsulation, Block trick, and using .mx files to back up memoized values.

Comments and suggestions welcome!

Answer 2

Initializing is not the same as downloading:

I believe you are witnessing the data being unpacked for use.