Incorrect Timing of Total

Question

For demonstrating how fast C-compiled functions can be, in one of my courses I use the following function for finding the sum of a list of reals:

myTotal = Compile[{{lst, _Real, 1}},Module[{s=0.}, Do[s=s+z, {z, lst}];s], CompilationTarget->"C"];

In Mathematica 8 and 9, this function is about as fast as the built-in function Total.

In Mathematica 10.0.0, there was a minor bug in Total, making Total three times slower than myTotal. That bug is repaired now. Even better, while the function myTotal is as fast as it was, Total now is almost two times faster:

lst=RandomReal[{0,1}, {2 10^7}];

Do[myTotal[lst], {100}] // RepeatedTiming
Do[Total[lst], {100}] // RepeatedTiming

(* {1.919,Null}
   {1.05,Null} *)

RepeatedTiming works fine. However, in Mathematica 10.3, Timing for Total does not work well:

Do[myTotal[lst], {100}] // Timing
Do[Total[lst], {100}] // Timing

(* {1.93441,Null}
   {0.,Null} *)

Do[myTotal[lst], {100}] // AbsoluteTiming
Do[Total[lst], {100}] // AbsoluteTiming

(* {1.92955,Null}
   {1.05932,Null} *)

Do[Total[lst], {1000}] // Timing

(* {0.0312002,Null} *)

This looks like a minor bug to me. Is this bug restricted to Windows, or is it a 'general' bug?

I can reproduce it on Windows, but not on OS X. It is indeed specific to Total. Tagging it as bug. — Szabolcs, Nov 02 '15 at 09:12
This is not specific to Total. For example, Do[Exp[lst];, {50}] // Timing returns {0., Null}, whereas Do[Exp[lst];, {50}] // AbsoluteTiming returns {2.14685, Null} on my PC with Win 10 and Mma 10.3. — Karsten7, Nov 02 '15 at 11:34
I normally never use Timing, as I consider it meaningless. Or as the documentation puts it: "it may ..." and "on some operating systems ...". — Karsten7, Nov 02 '15 at 11:45
@Karsten7. Timing was originally meant to measure total CPU time in the main kernel process. It is far from meaningless in principle, and if it worked reliably, it would probably be more broadly useful than AbsoluteTiming. But, I agree that more effort needs to be spent to make its operation reliable and consistent enough to trust it. — Oleksandr R., Nov 02 '15 at 11:59
Do[Total[lst], {1000}]; // Timing // AbsoluteTiming returns {10.0118, {0.046875, Null}}. Probably most of the calculation is just done in a way that "on some operating systems, Timing may ignore". — Karsten7, Nov 02 '15 at 16:20
By the way, Total is more intelligent then the direct sum. — ybeltukov, Nov 02 '15 at 18:56
Why is this considered a bug? Timing behaves as documented. It's just not necessarily the timing function one would like to have. — Karsten7, Nov 02 '15 at 19:40
@Karsten7. agreed; it shouldn't be a bug. I'll remove the tag. I would still argue strongly against this being a useful implementation of Timing, though. In the past the problem wasn't obvious because few functions were multithreaded. Now, as the rest of Mathematica improves, Timing is becoming increasingly useless. — Oleksandr R., Nov 02 '15 at 22:55
@ybeltukov so is rolling your own function! http://mathematica.stackexchange.com/questions/79174/compilation-of-total-with-compensated-summation?lq=1 — dr.blochwave, Nov 02 '15 at 23:05

ilian · Answer 1 · 2015-11-02T20:09:01.007

13

There are a couple of issues at play.

One is that the implementation of Total has changed, and it is now using several threads in parallel.

Furthermore, there are platform-specific differences in how Timing works. On Windows, it will only measure the CPU time used by the main kernel thread, excluding any subthreads. On Linux, and I believe OS X, it will include all threads in the kernel process.

As mentioned in the documentation,

On certain computer systems with multiple CPUs, the Wolfram Language kernel may sometimes spawn additional threads on different CPUs. On some operating systems, Timing may ignore these additional threads. On other operating systems, it may give the total time spent in all threads, which may exceed the result from AbsoluteTiming.

edited Nov 02 '15 at 20:09

answered Nov 02 '15 at 16:51

ilian

25,474
4
117
186

4

How can one find out that a function like Total is using several threads? Its only attribute is Protected. Can I implant some Developer`ThreadID[] into the evaluation somehow? – Karsten7 Nov 02 '15 at 17:13
2

@Karsten7. No top level interface for this, it's a pthread-level API. Your OS should however have tools to monitor threads as they are being launched, their resource usage etc. – ilian Nov 02 '15 at 18:15
3

Would it not be possible to fix the Windows implementation so that it takes all threads into account (like the Task Manager does)? I appreciate that the documentation technically allows for this behavior, but from a usefulness point of view, it's very unhelpful (in my opinion). Not considering separate processes is, however, absolutely fine (again, IMO). – Oleksandr R. Nov 02 '15 at 22:02
1

Has the implementation of compensated summation changed at all in 10.3? http://mathematica.stackexchange.com/questions/79174/compilation-of-total-with-compensated-summation/79194#79194 – dr.blochwave Nov 02 '15 at 23:02
2

@OleksandrR. Yes, developers are aware of this in principle. It would be nice to have consistency between platforms and taking all threads in account seems more useful to me, though presumably there could be arguments for the other side too. – ilian Nov 02 '15 at 23:03
@blochwave Not to my knowledge. – ilian Nov 02 '15 at 23:22

score 7 · Answer 2 · answered Nov 02 '15 at 23:23

This is just a comment with code. Here is a very simple multithreaded compiled function (not a sum calculation) to prove that the issue has nothing to do with Total specifically, but is a more general (and unfortunate) issue with Timing:

fc = Compile[{{x, _Real, 0}, {n, _Integer, 0}}, 
  Block[{y = x}, Do[y = Tan[y], {n}]; y],
  Parallelization -> True, RuntimeAttributes -> Listable
 ];

fc[Range[$ProcessorCount], 5*^7] // Timing // AbsoluteTiming
(* -> {1.281250, {0., (* numerical results omitted *) }} *)

(Adding CompilationTarget -> "C" will make no difference to the result. Although, note that the code has to return something, otherwise many C compilers will eliminate it, resulting in the timing of 0 being absolutely correct.)

Incorrect Timing of Total

2 Answers2

Linked