How to collect result continuously (interruptible calculation) when running parallel calculations?

Question

This is the most common pattern to compute a table of results:

Table[function[p], {p, parameters}]

(regardless of how it's implemented, it could be a Map)

The problem with this is that if the calculation is interrupted before it's finished, the partial results will be lost.

We can do this in a safely interruptible way like so:

Do[AppendTo[results, {p, function[p]}], {p, parameters}]

If this calculation is interrupted before it's finished, the intermediate results are still preserved. We can easily restart the calculation later, for those parameter values only for which function[] hasn't been run yet.

Question: What is the best way to achieve this when running calculations in parallel?

Assume that function[] is expensive to calculate and that the calculation time may be different for different parameter values. The parallel jobs must be submitted in a way to make best use of the CPU. The result collection must not be shared between the parallel kernels as it may be a very large variable (i.e. I don't want as many copies of it in memory as there are kernels)

---

Motivation: I need this because I want to be able to make my calculations time constrained. I want to run the function for as many values as possible during the night. In the morning I want to stop it and see what I got, and decide whether to continue or not.

---

Notes:

I'm sure people will mention that AppendTo is inefficient and is best avoided in a loop. I think this is not an issue here (considering that the calculations run on the subkernels and function[] is expensive). It was just the simplest way to illustrate the problem. There could be other ways to collect results, e.g. using a linked list, and flattening it out later. Sow/Reap is not applicable here because they don't make it possible to interrupt the calculation.

About the long running time: The most expensive part of the calculations I'm running are in C++ and called through LibraryLink, but they still take a very long time to finish.

Answer 1

Regarding using Sow instead of AppendTo, you may find this trick useful:

Last[Last[Reap[CheckAbort[Do[Pause[0.1]; Sow[x], {x, 30}], ignored]]]]

(Try running this and aborting it partway through. It runs for 3 seconds due to the Pause[0.1] commands.)

Do is used instead of Table, and the results are returned with Sow. The CheckAbort catches when you abort your computation partway through and does the useful tidying up (in this case, returning something, anything, to the enclosing Reap).

You can combine this with a version of Sow that always run on the master kernel:

SetSharedFunction[ParallelSow];
ParallelSow[expr_] := Sow[expr]

(Tangentially related blog post I did: http://blog.wolfram.com/2011/04/20/mathematica-qa-sow-reap-and-parallel-programming/)

Then you could use this parallelized version:

In[3]:= Last[
 Last[Reap[
   CheckAbort[ParallelDo[Pause[0.1]; ParallelSow[x], {x, 30}], 
    ignored]]]]

Out[3]= {6, 1, 7, 2, 8, 3, 9, 4, 10, 5, 16, 11, 17, 12, 18, 13, 19, \
14, 20, 15, 21, 26, 22, 27, 23, 28, 24, 29, 25, 30}

However, as you can see, the results come in in an unpredictable order so something slightly cleverer is in order. Here is one way (probably not the best but the first thing I thought of):

In[5]:= Catch[
 Last[Last[
   Reap[CheckAbort[
     Throw[ParallelTable[Pause[0.1]; ParallelSow[x], {x, 30}]], 
     ignored]]]]]

Out[5]= {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, \
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30}

The Throw is used to jump outside the Reap if the ParallelTable finishes. (Getting messy!)

To be safe this should be wrapped up in a function and tags (a.k.a. the optional second argument) should be used on the Throw, Catch, Sow, Reap.

Answer 2

Here's another possible way using dynamic, first the serial version:

timeLeft[start_, frac_] := With[
    {past = AbsoluteTime[] - start},
    If[frac == 0 || past < 1, "-", Floor[past/frac-past]]
];

SafeMap[func_, list_] := DynamicModule[
    {len, size, abortedQ, lastresult, starttime, n},

    len = Length[list]; size = 0; abortedQ = False; 
    lastresult; starttime = AbsoluteTime[]; n = 0;

    Monitor[
        Table[
            If[TrueQ[abortedQ], $Aborted,
                lastresult = func[list[[n]]];
                size += ByteCount[lastresult];
                lastresult
            ], 
            {n, Range[len]}
        ],
        Refresh[Panel@Column[{
            ProgressIndicator[n/len, ImageSize -> 350],
                Row[{Button["Abort", abortedQ = True],
                    Grid[{{"Element", "Memory (kB)", "Time left (s)"},
                        {StringForm["``/``", n, len],
                         ToString @ NumberForm[size/10.^3, {3, 1}],
                         ToString @ timeLeft[starttime, n/len]}
                        }, 
                    Spacings -> {1, 1}, ItemSize -> {10, 1}, Dividers -> Center
                ]}, Spacer[5]]
            }], UpdateInterval -> 0.5, TrackedSymbols -> {}
        ]
    ]
]

Try running the following example:

results = SafeMap[(Pause[1];Plot[x^2-#^2==0,{x,0,10}])&, Table[i, {i,10}]]

The panel displays the current element being evaluated as well as the total memory used thus far, and the estimated time remaining. If you click "abort" you get the partially generated list. For parallelization, only a minor change is needed:

Clear[SafeMap];
SafeMap[func_, list_, ker_:$KernelCount] := DynamicModule[
    {len, bag, size, lastresults, starttime, n, results, t},

    len = Length[list]; size = 0; starttime = AbsoluteTime[]; 
    results = {}; SetSharedVariable[results, size];

    Monitor[
        t = Table[ParallelSubmit[{i}, 
        With[{r = func[list[[i]]]}, size += ByteCount[r]; AppendTo[results, {i, r}]]], {i, Range[len]}];
        CheckAbort[WaitAll[t], AbortKernels[]];
        SortBy[results, First]
        ,
        Dynamic@Refresh[Panel @ Column[{
            ProgressIndicator[Length[results]/len, ImageSize -> 350],
                Row[{Button["Abort",  AbortKernels[]],
                    Grid[{{"Element", "Memory (kB)", "Time left (s)"},
                        {StringForm["``/``", Length[results], len],
                         ToString @ NumberForm[size/10.^3, {3, 1}],
                         ToString @ timeLeft[starttime, Length[results]/len]}
                        }, Spacings -> {1, 1}, ItemSize -> {10, 1}, Dividers -> Center
                ]}, Spacer[5]]
            }], UpdateInterval -> 0.1, TrackedSymbols -> {}
        ]
    ]
]

This allows you to abort by CMD+. or by pressing the button. For example

SafeMap[(Pause[1]; #^2)&, Table[i, {i, 30}], 4]

Answer 3

In this answer, I have implemeted Abort-able Table based on Do loops and Reap-Sow, using a technique similar to what is described in the answer of @Andrew (look at the bottom of the post for the second, more compact implementation). It seems that all you have to do is take that code and replace Do with ParallelDo.

Answer 4

Have you tried ParallelDo? Here's an example implementation:

First, we need a function to simulate a lengthy calculation. f randomly generates a number $0<p<1$; if the number is $>0.5$ it calls Abort[], otherwise it returns the number afterwards.

f := If[# > .5, Abort[], #] &@RandomReal[]

Generate some dummy data (not sure whether SetSharedVariable is necessary, at least it doesn't hurt),

data = ConstantArray[0, 10]
SetSharedVariable[results];
results = {};

Launch the calculation and return results,

ParallelDo[AppendTo[results, f], {i, 10}]
results

The program aborts the calculation (almost) every time, and then prints the values calculated.

Note that this method may become very ineffective for longer result lists, as they're stored as fixed-size arrays internally. In that case.