Parallel function evaluation for minimal value

Question

I am asking myself, if the evaluation of a function f looking for a minimal value over a discrete set of values can be parallelize (brute scan for a potential minimal value over a rectangular region with specified resolution). Is it possible or just not suitable for parallel computing do to the need to always compare to a reference which needs to happen in the main kernel? As far as I understood it in

Why won't Parallelize speed up my code?

the forced evaluation in the main kernel with SetSharedVariable can cause a significant lost in speed, which I think is the case in my horribly parallelized evaluation (see below). Any suggestions? I am pretty sure, I am just not seeing the obvious perspective. I dont want to use NMinimize, I only want to scan rapidly (if possible also in parallel) a rectangular region with specified resolution and pick up the minimal value. Sorry, if this is a duplicate, I was not able to find an answer. Thanks.

Minimal example:

Function:

f = Sin[x - z + Pi/4] + (y - 2)^2 + 13;

Sequential evaluation with do:

Clear[fmin]
fmin
n = 10^1*2;
fmin = f /. {x -> 0, y -> 0, z -> 0};
fmin // N
start = DateString[]
Do[
 ftemp = f /. {x -> xp, y -> yp, z -> zp};
 If[ftemp < fmin, fmin = ftemp];
 , {xp, 0, Pi, Pi/n}
 , {yp, -2, 4, 6/n}
 , {zp, -Pi, Pi, 2*Pi/n}
 ]
end = DateString[]
DateDifference[start, end, {"Minute", "Second"}]
fmin // N

Horribly parallelized evaluation

Clear[fmin]
fmin
n = 10^1*2;
fmin = f /. {x -> 0, y -> 0, z -> 0};
fmin // N
SetSharedVariable[fmin];
start = DateString[]
ParallelDo[
 ftemp = f /. {x -> xp, y -> yp, z -> zp};
 If[ftemp < fmin, fmin = ftemp];
 , {xp, 0, Pi, Pi/n}
 , {yp, -2, 4, 6/n}
 , {zp, -Pi, Pi, 2*Pi/n}
 ]
end = DateString[]
DateDifference[start, end, {"Minute", "Second"}]
fmin // N

I guess z -> yp is a typo and should be z -> zp? Are there any reasons why you are using, e.g., f /. {x -> 0, y -> 0, z -> 0} and don't have f defined using SetDeleyed(f[x_, y_, z_] := Sin[x - z + Pi/4] + (y - 2)^2 + 13)? — Karsten7, Oct 30 '15 at 16:47
Yes, that is a typo, I will correct it. No, no real reason, I just like to use sometimes expressions instead of functions. Hmmm, ... would a function evaluate faster? — Mauricio Fernández, Oct 31 '15 at 14:10

score 3 · Accepted Answer · answered Oct 29 '15 at 15:24

3

Don't compare to a single (shared) main-kernel variable (fmin) on each kernel. Instead, allow each kernel to find the smallest of the points it has checked. Let each kernel have its own private fmin. Then you'll have $KernelCount candidates for the minimum. Finally select the smallest of these.

ParallelCombine is made for precisely this type of approach. It may be a good idea to use Method -> "CoarsestGrained".

answered Oct 29 '15 at 15:24

Szabolcs

234,956
30
623
1,263

Thank you very much, I was not aware of ParallelCombine, or I overread it. – Mauricio Fernández Oct 29 '15 at 15:27
Do you mean using ParallelCombine[Do[ftemp = f /. {x -> xp, y -> yp, z -> yp}; If[ftemp < fmin, fmin = ftemp];, {xp, 0, Pi, Pi/n}, {yp, -2, 4, 6/n}, {zp, -Pi, Pi, 2*Pi/n}], Method -> "CoarsestGrained"]? Just want your confirmation before adding a speed comparison. – Karsten7 Oct 29 '15 at 16:30
1

@Karsten7. No, not quite. I just implemented what I meant and it is terribly slow. Then I remembered that I noticed before that ParallelCombine is much, much slower than it is supposed to be. A naive reimplementation in terms of ParallelSubmit is much faster. I searched my mailbox and found when I reported this to WRI more than a year ago. They never responded. I'm too tired now but I'll try to write this up tomorrow. – Szabolcs Oct 29 '15 at 20:18

Karsten7 · Answer 2 · 2015-10-30T17:08:31.940

3

The most straightforward parallelization of you code without a slowdown due to using SetSharedVariable is to use:

f[x_, y_, z_] := Sin[x - z + Pi/4] + (y - 2)^2 + 13

n = 10^1*2.;

LaunchKernels[];

AbsoluteTiming[
 ParallelEvaluate[fmin = f[0., 0., 0.];];
 ParallelDo[
  If[# < fmin, fmin = #] &@f[xp, yp, zp];, 
  {xp, 0., Pi, Pi/n}, {yp, -2., 4., 6./n}, {zp, -Pi, Pi, 2*Pi/n}];
 fmin = Min[ParallelEvaluate[fmin]]
 ]

$\ ${0.0699944, 12.01}

For comparison:

SetSharedVariable[fmin]

ParallelDo[ftemp = f /. {x -> xp, y -> yp, z -> yp};
  If[ftemp < fmin, fmin = ftemp];, {xp, 0, Pi, Pi/n}, {yp, -2, 4, 6/n}, 
  {zp, -Pi, Pi, 2*Pi/n}] // AbsoluteTiming

$\ ${34.0979, Null}

Your fmin is just the Min of all the ftemp you are computing. Therefore, you could replace your Do loop with something like

Min[Table[
  f /. {x -> xp, y -> yp, z -> yp}, {xp, 0, Pi, Pi/n}, {yp, -2, 4, 6/n}, 
  {zp, -Pi, Pi, 2*Pi/n}]]

And for parallelization change Table to ParallelTable:

Min[ParallelTable[
  f /. {x -> xp, y -> yp, z -> yp}, {xp, 0, Pi, Pi/n}, {yp, -2, 4, 6/n}, 
  {zp, -Pi, Pi, 2*Pi/n}]]

Speed comparison:

AbsoluteTiming[
 fmin = Min[
   Table[N@f /. {x -> xp, y -> yp, z -> yp}, {xp, 0, Pi, Pi/n}, {yp, -2, 4, 6/n}, 
    {zp, -Pi, Pi, 2*Pi/n}]]
 ]

$\ ${0.261962, 12.0522}

AbsoluteTiming[
 fmin = Min[
   ParallelTable[
    N@f /. {x -> xp, y -> yp, z -> yp}, {xp, 0, Pi, Pi/n}, {yp, -2, 4, 6/n}, 
    {zp, -Pi, Pi, 2*Pi/n}]]
 ]

$\ ${0.120665, 12.0522}

AbsoluteTiming[
 fmin = Min[
   ParallelTable[
    N@f /. {x -> xp, y -> yp, z -> yp}, {xp, 0, Pi, Pi/n}, {yp, -2, 4, 6/n}, 
    {zp, -Pi, Pi, 2*Pi/n}, Method -> "CoarsestGrained"]]
 ]

$\ ${0.0999586, 12.0522}

edited Oct 30 '15 at 17:08

answered Oct 29 '15 at 15:34

Karsten7

27,448
5
73
134

Hmmm, but what about when I go up with the resolution? The table will explode in resources, which I am not interested in. I only need a scalar value, I mean the minimal value. Or is this not important since the table is not assigned to any symbol and therefore deleted after the completition of Min[ ] ? – Mauricio Fernández Oct 29 '15 at 15:37
@MauricioLobos Please see my latest edit for a version that will use less memory. Using ParallelTable consumes more memory, even more than using Table. It depends on the amount of your RAM, if you will hit a memory or a time limit first, as computation time also explodes. – Karsten7 Oct 29 '15 at 16:21
@MauricioLobos Using ParallelTable makes it possible to visualize the intermediate data (for the given example) and to find the intervals in which increasing the resolution makes most sense. – Karsten7 Oct 29 '15 at 17:05
Thank you. May be this is also something NMinimize does if you use the Method->"RandomSearch". – Mauricio Fernández Oct 29 '15 at 17:20
@Mauricio and Karsten: there appears to be a bug in 10.3 which causes significant slowdowns with paralellelization. It's been giving me a really hard time before I remembered this thread. Witness Sorry, I don't have time to get to the bottom of this today, but I'll get back to it later – Szabolcs Oct 30 '15 at 11:48
@Szabolcs Wow, those slowdowns are impressive. Thanks for the information. Since I still have 10.0.2 I think I will stay with it until 10.4 or 11. – Mauricio Fernández Oct 30 '15 at 15:35
@MauricioLobos I'm using Mathematica v10.3 on Windows 10 and don't witness such slowdowns. – Karsten7 Oct 30 '15 at 17:15
@Karsten7. You mean compared to 10.2? – Szabolcs Oct 30 '15 at 17:45
@Szabolcs I don't have 10.2 installed anymore, but my timings are in the same range as your 10.2 timings. I just did a test run using v9.0.1 and it's only 12% faster. – Karsten7 Oct 30 '15 at 18:01

Parallel function evaluation for minimal value

2 Answers2

Linked