Fast way to subtract the mean from a dataset

Question

I have a dataset, say:

data = 
  {{10, 8.04}, {8, 6.95}, {13, 7.58}, {9, 8.81}, {11, 8.33}, {14, 9.96}, 
   {6, 7.24}, {4, 4.26}, {12, 10.84}, {7, 4.82}, {5, 5.68}}

and I want to subtract the mean from it, producing:

correcteddata = 
  {{1, 0.539091}, {-1, -0.550909}, {4, 0.0790909}, {0, 1.30909}, {2, 0.829091}, 
   {5, 2.45909}, {-3, -0.260909}, {-5, -3.24091}, {3, 3.33909}, {-2, -2.68091}, 
   {-4, -1.82091}}

However, I'm looking to do this on a much larger data-set as efficiently as possible.

I've gone through several attempts:

This was very slow, but simple:

correcteddata = # - Mean[data] & /@ data

This was significantly faster, but involves adding another variable:

m = Mean[data];
correcteddata = # - m & /@ data

The fastest/best I've found so far is this:

correcteddata = Transpose[# - Mean /@ # &[Transpose[data]]]

But I'm wondering if there is something better.

I think the notation of the question is a bit confusing. You do not have a dataset ... you have sample bivariate $(X_i,Y_i)$ data. And rather than the stated desire to 'subtract the mean from it, you wish to subtract the sample means from the data. — wolfies, Jul 02 '17 at 08:26

Jens · Accepted Answer · 2017-07-01T15:53:59.237

14

Do this:

Standardize[data, Mean, 1 &]

Here are the timings for comparison, with f1 and f2 as defined in Mr.Wizard's answer:

big = RandomReal[{0, 50}, {1*^7, 2}];

f1[big] // RepeatedTiming // First
f2[big] // RepeatedTiming // First
Standardize[big, Mean, 1 &] // RepeatedTiming // First

0.274

0.232

0.12

edited Jul 01 '17 at 15:53

answered Jul 01 '17 at 15:46

Jens

97,245
7
213
499

1

What version and platform are you using? In 10.1 under Windows this is much slower than either f1 or f2 in my post at 0.85 seconds on the same test. – Mr.Wizard Jul 01 '17 at 15:49
1

Strange - on my Mac with MMA 11.0.1 my answer is twice as fast as yours... – Jens Jul 01 '17 at 15:51
On the same big data? If so I guess this function received a needed upgrade. – Mr.Wizard Jul 01 '17 at 15:53
2

Yes, I added the timings with your method. – Jens Jul 01 '17 at 15:54

Mr.Wizard · Answer 2 · 2017-07-02T06:15:42.800

11

This is a modest improvement on your code:

f2[data_] := (data\[Transpose] - Mean[data])\[Transpose]

With your code as a function f1 for reference:

f1[data_] := Transpose[# - Mean /@ # &[Transpose[data]]]

And now with Simon Woods's f3 calling the internal function used by Standardize in v11:

f3[data_] := 
  Module[{a = data}, Statistics`Library`MatrixRowTranslate[a, -Mean[a]]; a]

Timings:

big = RandomReal[{0, 50}, {1*^7, 2}];

f1[big] // RepeatedTiming // First
f2[big] // RepeatedTiming // First
f3[big] // RepeatedTiming // First

0.193
0.15
0.0936

f1[big] === f2[big] === f3[big]

True

In Mathematica 10.1 under Windows x64 Standardize performs rather poorly. I get:

Standardize[big, Mean, 1 &] // RepeatedTiming // First

0.852

I am curious to know if this is version or platform dependent, or perhaps both.

edited Jul 02 '17 at 06:15

answered Jul 01 '17 at 15:29

Mr.Wizard

271,378
34
587
1,371

4

In V10.4.1 and V9.0, Standardize took 1.6 sec and in V11.1.1 it took 0.14 sec. – Michael E2 Jul 01 '17 at 16:19
3

Try f3[data_] := Block[{a = data}, Statistics`Library`MatrixRowTranslate[a, -Mean[data]]; a] - this is what Standardize does in 11.1.1 – Simon Woods Jul 01 '17 at 21:28
@Simon Thanks, that gives the performance one would have anticipated for a built-in! (I used Module as I feel that is appropriate.) – Mr.Wizard Jul 02 '17 at 06:16
@Simon beat me to it. I figured that out, which led to this answer, but haven't had time to come back here. – Michael E2 Jul 03 '17 at 00:34
@Michael would you please use that to answer https://mathematica.stackexchange.com/q/103527/121 ? – Mr.Wizard Jul 03 '17 at 08:27
@Mr.Wizard Done. -- Thanks. – Michael E2 Jul 03 '17 at 15:11

Fast way to subtract the mean from a dataset

2 Answers2