5

I have a fairly large project that logically splits into several subunits, and for reasons of testing, documentation, and discoverability, I would like to make each subunit its own (sub-)package, like so:

PillsyPackage`
  |
  +- `SubpackageA`
  |
  +- `SubpackageB`

However, all of these sub-packages need to share some private functions (they all work on the same data structures which should be opaque, so I want the accessors to be private).

Ideally, I'd like to give the sub-packages access to PillsyPackagePrivate`. What's the best way to do this in terms of syntax and file layout?

Pillsy
  • 18,498
  • 2
  • 46
  • 92
  • 2
    "need to share some private functions" - in the Standard Add-on Packages, this was solved by setting up a Common.m package containing the private functions that will be shared among other packages that need those functions. – J. M.'s missing motivation Oct 01 '18 at 18:05
  • 5
    Use new-style packages. Description here. – Leonid Shifrin Oct 01 '18 at 18:40
  • 1
    The drawback with applying a Common.m solution is that JITL (Just-in-time-loading via DeclarePackage) doesn't support a "names file". The drawback with a new-style package is that it is not scalable in terms of supporting JITL. If these two aspects are important (and they aren't always) but I argue they increase in importance with the size of a system so they might be for a "fairly large project". I am putting out shortly an extension of BeginPackage-EndPackage (which I will link to) that addresses this very issue. – Ronald Monson Oct 01 '18 at 20:07
  • @RonaldMonson DeclarePackage is an old style of doing this, which is no longer used, at least not internally in the company. The current mechanism is to use the paclet symbol autoloading and paclet project loading, which plays well with new-style packages. – Leonid Shifrin Oct 01 '18 at 20:31
  • 2
    @LeonidShifrin Yes I suspected as much as checking a few weeks ago its vestiges seem to be only in Mma's authortools. But I wonder if something important has been lost here - at least I am yet to see how the Stub attribute transfers as previously discussed ( I'm not just talking about public functions but private ones the OP mentions e.g. what if you want these privately shared functions to incorporate JITL? can the current mechanism handle this?). My sense is that this new mechanism is tilted towards public functions. Naturally, I don't fully understand paclet symbol autoloading so ... – Ronald Monson Oct 01 '18 at 21:07
  • ... there may well be this functionality but equally, as none of this is officially documented or a roadmap advertised for its polishing/scaling/release - here's hoping! - I guess we are constrained with what we currently have. I guess the same applies to BeginPackage/EndPackage whose ongoing presence seems unlikely to be deprecated anytime soon but I agree, to have a flowchart for what idioms might be reasonably adopted by developers in given situations would be great. – Ronald Monson Oct 01 '18 at 21:10
  • I do this in my own packages by creating an autoloader for all the symbols that sets up the right context path. It works incredibly well and is easy to extend. Can describe more if there’s interest. – b3m2a1 Oct 02 '18 at 08:37
  • @b3m2a1 That is exactly the sort of thing I'm interested in learning more about. Could you blow it up into an answer? You'll get at least one vote out of it. :) – Pillsy Oct 03 '18 at 14:45

1 Answers1

3

Here's how I do this. I have a standard load script I build from template for all of my packages. You can see one for my main development package here.

What it does then is look for a directory called Packages in the top-level package directory (although there are config options to change this, if I remember correctly).

Then if the directory structure looks like:

MyPackage
 Packages
  + pkg1.m
  + pkg2.m
  A
   + subpkg1.m
  B
   + subpkg2.m

I'll get out three contexts, "MyPackage`", "MyPackage`A", "MyPackage`B`". The way I then make all the symbols in each see eachother when they load is that I use lazy-loading. So first I scan through all the found package files and register all the symbol names in the appropriate context and setup an OwnValues pattern so that when they're called for the first time they will load the package they're declared in using a special PackageGet function. This is done by the PackageDeclare function in the load script.

That PackageGet function then uses Internal`WithLocalSettings and System`Private`NewContextPath and System`Private`RestoreContextPath to set up a $ContextPath that looks like:

{
 "MyPackage`",
 "MyPackage`A`",
 "MyPackage`B`",
 "MyPackage`PackageScope`Private`" (* I put a bunch of utility functions here *)
 }

This way each symbol sees each other when they load, but no private declarations or anything like that.

I haven't fully described what I do, as I also have ways to handle circular dependencies and things, but this is a basic overview that perhaps one might implement for themselves.

I have found it to be a useful and powerful idiom reminiscent of the way python handles package loading.

b3m2a1
  • 46,870
  • 3
  • 92
  • 239