I have a package that redefines a system symbol, and I am struggling to find a clean solution that will ensure that those redefinitions will still work inside a parallelized environment.
The core structure of the package is of the following form:
BeginPackage["Utilities`"];
f::usage = "...";
Begin["`Private`"];
f[x_] := x;
End[];
EndPackage[];
DistributeDefinitions["Utilities`"];
(Shamelessly stolen from here, so hopefully I'm not the only one that does this.) Doing the DistributeDefinitions call in this way has several advantages:
- it does not attempt to launch new subkernels when they are not necessary
- it keeps the package definitions in some mysterious place where they are always available for any new subkernels that get launched subsequently, even if e.g. one of them crashes and gets re-launched
- it does no harm if the package is called from inside a parallelized environment (i.e. it doesn't return errors of the form "you tried to parallelize from inside a subkernel, but this is not Inception and you are not Leonardo di Caprio")
Now, I want to expand this package to redefine the workings of some system symbol. For the sake of argument, suppose that I want to modify RandomComplex in the following form:
Unprotect[RandomComplex];
RandomComplex[{range1_List, moreRanges___}, number_] := Transpose[
RandomComplex[#, number] & /@ {range1, moreRanges}]
Protect[RandomComplex];
Unfortunately, even if I do this within the BeginPackage[] and EndPackage[] calls, the redefinition does not get sent to any subkernels that get launched afterwards. Thus, suppose I write this into the code above, load the package, launch some new kernels, and then run
ParallelEvaluate[RandomComplex[{{0, 1 + I}, {0, 1 + I}}, 3]]
I will then get a bunch of error messages, because each subkernel is using the unpatched version.
How can I make sure that this redefinition gets distributed downstream, ideally in a way that preserves the advantages I mentioned above, and with the least amount of pain possible.
(In particular, I can obviously run ParallelEvaluate on the redefinition, but then it is not stable against a subkernel crash-plus-relaunch, and it will kick and scream if I Get or Needs the package from inside a parallelized environment.)
Getand register it to be evaluated on newly launched kernels. http://mathematica.stackexchange.com/q/131856/12 To be fair, distribution may not be that bad, but it's a complicated thing and I don't like complicated things because they tend to cause surprises.ParallelNeeds, as well as registering init code is simple and predictable. – Szabolcs Dec 08 '16 at 22:12Getcall inside some funky init code? Or should I just ask for aParallelNeeds? The problem with the latter is that it does not allow me to specify the source of the package, so I have no idea how to use it in that situation. – Emilio Pisanty Dec 08 '16 at 22:20ParallelNeedsand you don't need to refer to a file. Or useGet(which can refer to a file), and put it in some init code. Simply load the parallel tools (evaluateParallelize) then setParallel`Developer`$InitCode = Hold[Get["your/file.m"]]before launching any kernels. If you do not need to care about kernels which will be launched in the future in ways you cannot control, then just doParallelEvaluate[Get["your/file.m"]]. – Szabolcs Dec 08 '16 at 22:29