2

I have two huge matrices m1 and m2 (length 25000), and for simplicity let us assume they are:

m1 = {{1, 1, -56}, {1.3, 2, 3.06}, {2, 0, -30.02}, {3, 
   1, -7.291}, {3.5, 2, 1.93}, {4, 0, 0}, {5, 0, 0}, {5.5, 
   1, -356.4}, {6, 1, 9.945}, {7, 0, -7.512}};
m2 = {{1, 1, -56}, {1, 2, 3.06} + .2, {2, 
0, -30.02}, {3, 1, -7.291} + .3, {3, 2, 1.93}, {4, 0, 0}, {5, 0, 
0}, {5, 1, -356.4} + .4, {6, 1, 9.945}, {7, 0, -7.512}};

I wrote the following code with the help of this forum to pick those rows of both matrices that have the same first value. However, the code is very slow 

picker1 = m1[[All, 1]];
picker2 = m2[[All, 1]];
newM1 = Cases[m1, x_ /; MemberQ[picker2, x[[1]]]]
newM2 = Cases[m2, x_ /; MemberQ[picker1, x[[1]]]]

I know the code is not an efficient and I appreciate if you can help me with more efficient and fast code.

qahtah
  • 1,397
  • 6
  • 14
  • Pick[m1, MemberQ[picker2, #] & /@ picker1] and Pick[m2, MemberQ[picker1, #] & /@ picker2]? – kglr Apr 26 '18 at 22:17
  • qahtah, your application could be boosted immensely if you can achieve to have matrices with either all integers or all real numbers, so that they can be packed. However, MemberQ has certain problems with floating point numbers. Do the first entries in each row in your huge matrices happen to be integers? – Henrik Schumacher Apr 26 '18 at 22:25
  • @HenrikSchumacher no they are real. – qahtah Apr 26 '18 at 23:44
  • @kglr too slow. – qahtah Apr 26 '18 at 23:46

1 Answers1

2

Using MemberQ is not a good idea, performancewise.

Let's create some example data

n = 25000;
m = 1000;
p1 = RandomInteger[{1, m}, n];
p2 = RandomInteger[{1, m}, n];
a1 = RandomReal[{-1, 1}, {n, 2}];
a2 = RandomReal[{-1, 1}, {n, 2}];

Note that we separated the "pickers" p1 and p2 from the data a1, a2. So m1[[i]] would correspond to Join[{p1[[i]]},a1[[i]]]. This way, p1, p2, a1, and a2 are all packed arrays.

For each list of pickers p1, p2, I create an Association with values all 1.

dict1 = AssociationThread[p1, 1];
dict2 = AssociationThread[p2, 1];

This way, dict1[x] will return 1 (read as True) if x is a member of p1 and Missing["KeyAbsent", x] otherwise. Lookup can convert Missing[...] to a default value; we use 0 (read as False) in the following. We create new lists of pickers newp1 and newp2 along with the new data matrices newa1 and newa2:

With[{b1 = Developer`ToPackedArray[Lookup[dict2, p1, 0]]},
   newp1 = Pick[p1, b1, 1];
   newa1 = Pick[a1, b1, 1];
   ]; // AbsoluteTiming
With[{b2 = Developer`ToPackedArray[Lookup[dict1, p2, 0]]},
   newp2 = Pick[p2, b2, 1];
   newa2 = Pick[a2, b2, 1];
   ]; // AbsoluteTiming

0.002823

0.002927

Here, we use Pick with 1 as third argument to pick all rows from a1 for which the corresponding entry in b1 is 1. Doing the anologous by constructions involving MemberQ takes about 5 seconds on my machine.

One has to be careful if the pickers are floating point numbers: This construction with Association might break down due to rounding errors since Associations are very literal about the keys. But that's the same with MemberQ.

Henrik Schumacher
  • 106,770
  • 7
  • 179
  • 309