Package found with `Needs`, but not with `ParallelNeeds`

Question

I want to use a self-written package in a parallel computation. However while Needs["mypackage`"] works without problems, ParallelNeeds["mypackage`"] can't open my package.

The immediate reason is related to the fact that I added to $Path, and the package is in one of the directories I've added. ParallelTable[$Path,{1}] reveals that this addition is not propagated to the subkernels, despite the fact that it is changed in the user kernel init file ($UserBaseDirectory/Kernel/init.m), and thus long before any subkernels are launched in the Notebook. Explicitly using DistributeDefinitions[$Path] didn't help either.

So my question is: Why isn't the $Path propagated to the kernels, and more importantly, what can I do about it?

Answer 1

You can just add you package path to the path on each of the kernels

ParallelEvaluate[AppendTo[$Path, ToFileName[{$HomeDirectory, "MyPackages"}]]]

now running

ParallelTable[$Path, {1}]

will show that the change has been propagated.

---

Besides using an init.m file, there is another way to run code on each kernel after start-up (full credit goes to this excellent answer by Oleksandr R.)

Needs["Parallel`Developer`"]; (* Load required package *)
$InitCode = Hold[AppendTo[$Path, ToFileName[{$HomeDirectory, "MyPackages"}]]];

Answer 2

I've now found the solution.

The reason that the path is not available is that it is set in an init file, and subkernels do not execute any init files. However, as it turns out, the reason why they do not execute any init files is that they are explicitly passed the option -noinit when started (you can easily see that by looking at the process listing on your OS; e.g. on Linux, ps x reveals the command line if the window is wide enough).

How to change the options on start does not seem to be documented, however I've found out that in the package SubKernels` there exists a variable SubKernels`Protected`stdargs which contains a string with the (non-mathlink) arguments passed to subkernels. By default it has the value " -subkernel -noinit". Therefore executing the following before launching the subkernels causes them to execute the standard initialization file:

Needs[SubKernels`]
SubKernels`Protected`stdargs = " -subkernel"

Obviously this can also be used to pass other options to the subkernels (an obvious candidate would be -initfile to have separate initialization for main and sub kernels, however I haven't tested that yet).

Answer 3

I use the following solution for this problem:

• I have a package called AddPath which can add pre-defined directories to the $Path. This package is placed in $UserBaseDirectory/Autoload, so it gets loaded on parallel kernels as well.

• I put another very basic package in Autoload which simply contains

  Needs["AddPath`"]
  AddPath[{"Pack1","Pack2",...}]
  

  Then I always have these packages available in the $Path both on the main kernel and subkernels.

Caveat: In order for a package to load from Autoload, it must follow a certain structure.

I also use a ParallelAddPath function which registers packages to be added to the $Path on newly started parallel kernels. In this way it is similar to ParallelNeeds. Implementation details:

• Register "initialization code" to run on any newly started parallel kernel

Finally, I set up argument-completion for AddPath to make it easier to type. Implementation notes:

• Argument completions for user-defined functions

I realize that this is overkill, and you could just create a file Autoload/MyInit/init.m which contains AppendTo[$Path, ...]. But I have been using this AddPath setup for quite a while and it works well for me, so I thought I would share it. I also think it is relevant for a question about changing $Path. Note: I only started using the Autoload part today though, before that I used the standard init.m to load AddPath.

---

I originally came up with AddPath for version management (I had multiple versions of packages I was developing) and dependency management. I looks roughly like below.

BeginPackage["AddPath`"]

AddPath::usage = "AddPath[\"package\"] will add \"package\" to the $Path."
RemovePath::usage = "RemovePath[\"package\"] will remove \"package\" to the $Path."
ParallelAddPath::usage = "ParallelAddPath[\"package\"] will register \"package\" to be added to the $Path on parallel kernels.";

Begin["`Private`"]

Get["AddPath`Config`"];

AppendTo[packageList, _ -> Null];
AppendTo[dependencies, _ -> {}];

AddPath::path = "The package `` is already in the $Path.";
AddPath::pack = "`` is not a known package.";

SetAttributes[AddPath, Listable]
AddPath[pack_String] := 
    Module[{deps, path},
        Quiet@AddPath@Replace[pack, dependencies];
        path = Replace[pack, packageList];
        If[path === Null,
            Message[AddPath::pack, pack];
            Return[]
        ];
        If[MemberQ[$Path, path],
   Message[AddPath::path, pack];
   Return[]
  ];
  AppendTo[$Path, path];
    ]

RemovePath::path = "The package `` is not in the $Path.";
RemovePath::pack = AddPath::pack;

(* RemovePath will not check for equivalent path names.
   ~/dir and /Users/someone/dir are considered different *)

SetAttributes[RemovePath, Listable]
RemovePath[pack_String] :=
    Module[{path},
        path = Replace[pack, packageList];
        If[path === Null,
            Message[RemovePath::pack, pack];
            Return[]
        ];
        If[Not@MemberQ[$Path, path],
   Message[RemovePath::path, pack];
   Return[]
  ];
  $Path = DeleteCases[$Path, path];
    ]


ParallelAddPath[arg : (_String|{___String})] :=
    (Parallelize;
     ParallelNeeds["AddPath`"];
     Parallel`Protected`AddInitCode[
         Parallel`Client`HoldCompound[AddPath[arg]]
     ];)


(* Set up argument completion. *)
addCompletion = FE`Evaluate[FEPrivate`AddSpecialArgCompletion[#]] &;    
addCompletion[# -> {Most[packageList][[All, 1]]}]& /@ {"AddPath", "RemovePath", "ParallelAddPath"};

End[] (* `Private` *)

EndPackage[]

I manually edit packageList and dependencies in Config.m whenever I need to. These define the package locations (usually Git repos) and their dependencies.

packageList = {
  "Spelunking" -> "~/Repos/Spelunking",
  "SciDraw" -> "~/Documents/Mathematica/SciDraw-0.0.7/packages",
   ...
  "MATLink10" -> "~/Documents/Mathematica/MATLink10",  (* example of multiple versions of the same package *)
  "MATLink11" -> "~/Documents/Mathematica/MATLink11",
  "MATLink1Dev" -> "~/Repos/MATLink1",
   ...
  "LTemplate" -> "~/Repos/LTemplate",
  "IGraphM" -> "~/Repos/IGraphM",
  ...
  _ -> Null
};

dependencies = {
  ...
  "IGraphM" -> "LTemplate",
  ...
  _ -> {}
};