Performance of select from a 3d list - Mathematica slower than Python

Question

I am creating a random 3d data set in Matematica 12.1. Then I am selecting all points that are in a certain range of one axis.

The same I am doing in Python (same computer, Python 3.8.5, numpy 1.19.2)

RESULT: It seems that Python is able to select much faster (1.7 sec) than Mathematica (5.2 sec). What is the reason for that? For selection in Mathematica I used the fastest solution, which is by Carl Woll (see here at bottom).

SeedRandom[1];
coordinates = RandomReal[10, {100000000, 3}];
selectedCoordinates = 
   Pick[coordinates, 
    Unitize@Clip[coordinates[[All, 1]], {6, 7}, {0, 0}], 
    1]; // AbsoluteTiming
{5.16326, Null}
Dimensions[coordinates]
{100000000, 3}
Dimensions[selectedCoordinates]
{10003201, 3}

PYTHON CODE:

import time
import numpy as np
np.random.seed(1)
coordinates = np.random.random_sample((100000000,3))*10
start = time.time()
selectedCoordinates = coordinates[(coordinates[:,0] > 6) & (coordinates[:,0] < 7)]
end = time.time()
print(end-start)
print(coordinates.shape)
print(selectedCoordinates.shape)
1.6979997158050537
(100000000, 3)
(9997954, 3)

Is your ultimate goal to select or to count? – yarchik Feb 19 '21 at 23:34 — yarchik, Feb 19 '21 at 23:34

score 6 · Accepted Answer · answered Feb 20 '21 at 19:44

I would argue that it's not a fair comparison unless you consider the performance with NumericArray, as the default Mathematica list-of-lists has many other features regarding numeric stability, etc. that are not present in a merely list of Real32 numbers.

Let me demonstrate: In your code you do something like:

SeedRandom[1];
coordinates = RandomReal[10, {100000000, 3}];
Clear[f]
f[coordinates_] := (Pick[coordinates, 
     Unitize@Clip[coordinates[[All, 1]], {6, 7}, {0, 0}], 1];) // 
  AbsoluteTiming
f[coordinates] (4.0478 s on my MacBook)

Now let's convert coordinates into a NumericArray (similar to what is being used internally in numpy):

coordinates32 = NumericArray[coordinates, "Real32"];
f[coordinates32] (* 1.09 s on my MacBook *)

This gives us a 3.7x speedup, comparable to the performance gains you observe in Python.

In the end, 5.2 sec is an excellent performance as the elements could be numbers, but also complex numbers, symbols, functions, etc. Try to do this with Python with no change in your code! — Denis Cousineau, Feb 20 '21 at 22:30

Performance of select from a 3d list - Mathematica slower than Python

1 Answers1