Clever use of DownValues, or failure to understand the "Mathematica Way."

Question

I needed to look at the changes in the values of about 4000 items from one year to the next. I have a table with identifiers and values for each year. Not all of the items are in each list. The tables contain the following rows, with more data to the right. enter image description here
So I did the following:

govnew = Select[data[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
govold = Select[dataold[[2 ;;, {2, 3, 4, 6}]], #[[4]] > 0 &];  
intersection = Intersection[govnew[[All, 1]], govold[[All, 1]]];   
Clear[both];  
Map[(both[#] = 1) &, intersection];  
govnew2 = Select[govnew, both[#[[1]]] == 1 &];   
govold2 = Select[govold, both[#[[1]]] == 1 &];

Which reduced the two lists to the intersection of the identifiers in both lists. It worked. But it reduces a list of Down Values to a dictionary. Doesn't use a Mathematica function in a secret and powerful way. Is there a more Mathematica like approach.

Could you please provide some dummy (or actual) data? btw, Scan is more typical and cleaner instead of your Map above. — rm -rf, Jul 23 '13 at 14:35
@rm-rf FWIW I'm using {data, dataold} = List @@ RandomInteger[12, {2, 20, 6}]; — Mr.Wizard, Jul 23 '13 at 14:44
+1 - Great use of DownValues. Nice! P.S., How's life? Time to actually get together. Best. — Jagra, Jul 23 '13 at 14:57
@rm-rf I never use scan. I'll find out what it does. Thanks for the suggestion. — George Wolfe, Jul 23 '13 at 17:23
Seems like a perfectly fine use of DownValues. An alternative might be to join the lists, then GatherBy[combinedlist,First] or something along those lines. Now discard all singletons. — Daniel Lichtblau, Jul 23 '13 at 22:41

Mr.Wizard · Accepted Answer · 2013-07-25T18:29:35.060

13

This appears to be a perfectly legitimate use of DownValues. These are often used by experienced users as a hash table.

There are some ways you might improve this.

First, you could use the value True directly, and it's arguably better to Scan than to Map, but I've used the latter often enough myself as it rarely matters.

Scan[(both[#] = True) &, intersection];

govnew2 = Select[govnew, both[#[[1]]] &]

This DownValues hash table is often very fast as should not be dismissed. You can however do this more simply with Alternatives, e.g.:

pat = Alternatives @@ intersection;

govnew2 = Cases[govnew, {pat, __}]
govold2 = Cases[govold, {pat, __}]

Regarding the first part of your code, Pick can usually be faster though in this case also longer:

govnew = Pick[data[[2 ;;, {2, 3, 4, 6}]], Sign @ data[[2 ;;, 6]], 1]

A note regarding Alternatives

As mentioned above it is important to know that Alternatives is not a replacement for the DownValues. Alternatives are checked in order and therefore the time taken for a match will be proportionate to the number of alternatives. For applications involving a large number it will be much faster to use either DownValues or a Rule table optimized with Dispatch. For example:

range = Range[1*^5];
nums = RandomSample[range, 5000];
alts = Alternatives @@ nums;
Scan[(test[#] = True) &, nums];

Select[range, test] // Timing // First
Cases[range, alts]  // Timing // First

0.049

5.289

edited Jul 25 '13 at 18:29

answered Jul 23 '13 at 14:35

Mr.Wizard

271,378
34
587
1,371

I see that True makes sense. Then I could just Select[govnew, both[#[[1]]] &], right? And I don't think I've ever used Alternatives. That would be closer to pure "Mathematica" code. – George Wolfe Jul 23 '13 at 17:25
1

PS. I find myself wanting to use the word "Mathematical" to mean more like a Mathematica way of doing it, but it doesn't mean that. Any suggestions? – George Wolfe Jul 23 '13 at 17:30
@rm-rf Thanks, everyone!! – George Wolfe Jul 23 '13 at 17:32
Mathematicaish? (Maybe not..) – Daniel Lichtblau Jul 23 '13 at 22:43
@George How about Wolfish? :o) – Mr.Wizard Jul 24 '13 at 00:30
1

@Mr.Wizard I just used Alternatives[] in another application. It's elegant and clean. It's, it's ... Mathematicalicious! – George Wolfe Jul 24 '13 at 21:32
@George I added a short warning about Alternatives that you should read. Please don't let it dissuade you from using this function, but rather be aware of when it is and isn't appropriate. – Mr.Wizard Jul 25 '13 at 18:30
@Mr.Wizard Thanks for pointing this out. – George Wolfe Jul 25 '13 at 20:01

score 9 · Answer 2 · answered Jul 24 '13 at 15:41

The example below indicates what I had in mind with a comment to use Join and GatherBy. It also shows that this is roughly comparable to using DownValues.

Set up an example:

n = 6;
l1 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l2 = RandomInteger[10^n, {7*10^(n - 1), 2}];
l1b = Map[First, GatherBy[l1, First]];
l2b = Map[First, GatherBy[l2, First]];

Using Join and GatherBy:

Timing[
 l3 = Select[GatherBy[Join[l1b, l2b], First], Length[#] == 2 &];]

(* Out[178]= {2.360000, Null} *)

Using DownValues:

govnew = l1b;
govold = l2b;
Clear[both];
Timing[intersection = 
  Intersection[govnew[[All, 1]], govold[[All, 1]]]; 
 Scan[(both[#] = 1) &, intersection]; 
 govnew2 = Select[govnew, both[#[[1]]] == 1 &]; 
 govold2 = Select[govold, both[#[[1]]] == 1 &];]

(* Out[189]= {3.780000, Null} *)

Quick check:

Sort[l3][[All, 1, 1]] === Sort[govold2][[All, 1]] === 
 Sort[govnew2][[All, 1]]

(* Out[207]= True *)

score 6 · Answer 3 · answered Nov 07 '15 at 00:15

This is another problem that used to be kind of a pain back before Mathematica 10, but is now dramatically simplified by Associations and the related functions:

CompareRows[tables:{___List}] := MapThread[SortBy, 
   {(Apply[Join]) @* Values /@ KeyIntersection[GroupBy[First] /@ 
       tables], PositionIndex /@ tables}];

The SortBy/PositionIndex combination makes sure the rows come out in their original order. Then it's an easy matter to reproduce your results:

{govold2, govnew2} = CompareRows[{data, dataold}];

Even better, this new version works with any number of data sets for comparison, and doing so required zero extra effort on my part.

Clever use of DownValues, or failure to understand the "Mathematica Way."

3 Answers3

Linked