10

I'm new to the mathematica platform and I am developing a simulation framework as my first exercise and have been tearing my hair figuring how to make packages interact with each other. Ill explain a minimum package experiment I developed to understand my problems and please I would like a theoretical explanation of why this happens, if it is a bug or if I am just dumb with mathematica.

The thing is that if I have packages that use each others exported symbols or functions inside their functions, after the first package is loaded he loads the others declared as needed as well, but then only that package that was loaded first can reference all other exported stuff, and the others for some reason can only look at the exported data of later packages/contexts in the context path. To make every package able to see every other exports I need to load them again individually. This makes no sense to me, I thought the whole point of having needs in any package constructor is to avoid loading myself every package of my packages set. Is this a bug? Btw, the exported symbols from different packages are found right away by mathematica is just the symbols inside functions that cant reference each other.

So basically I defined three packages, ja1,ja2,ja3. All have the same structure like this:

Package ja1

BeginPackage["SimulationsSystem`ja1`", { "SimulationsSystem`ja2`", "SimulationsSystem`ja3`"}]

shadow::usage="shadow"
var1::usage = "var1"
pro1::usage = "pro1"
(* Exported symbols added here with SymbolName::usage *)  

Begin["`Private`"] (* Begin Private Context *) 

shadow=1;
var1=1;
pro1[]:=var1+var2+var3

End[] (* End Private Context *)

EndPackage[]

Package ja2

BeginPackage["SimulationsSystem`ja2`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja3`"}]

shadow::usage="shadow"
var2::usage = "var2"
pro2::usage = "pro2"
(* Exported symbols added here with SymbolName::usage *)  

Begin["`Private`"] (* Begin Private Context *) 

shadow=2;
var2=1;
pro2[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]

Package ja3

BeginPackage["SimulationsSystem`ja3`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja2`"}]

shadow::usage="shadow"
var3::usage = "var3"
pro3::usage = "pro3"
(* Exported symbols added here with SymbolName::usage *)  

Begin["`Private`"] (* Begin Private Context *) 

shadow=3;
var3=1;
pro3[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]

Commands run: so after defining the packages I tried to run each pro function, called each var and called shadow to see the nearest Package/context working. And here is the result:

In[2]:= << SimulationsSystem`ja2`
$ContextPath

Out[5]= {"SimulationsSystem`ja2`", "SimulationsSystem`ja1`", \
"SimulationsSystem`ja3`", "SimulationsSystem`SimulationsSystem`", \
"SimulationsSystem`Models`Basic`", \
"SimulationsSystem`SimulFunctions`", \
"SimulationsSystem`TurnPackages`", \
"SimulationsSystem`EventFunctions`", "SimulationsSystem`Classes`", \
"PacletManager`", "WebServices`", "System`", "Global`"}

In[6]:= pro1[]

Out[6]= 2 + SimulationsSystem`ja1`Private`var2

In[7]:= pro2[]

Out[7]= 3

In[8]:= pro3[]

Out[8]= 1 + SimulationsSystem`ja3`Private`var1 + \
SimulationsSystem`ja3`Private`var2

In[9]:= var1

Out[9]= 1

In[10]:= var2

Out[10]= 1

In[11]:= var3

Out[11]= 1

In[12]:= shadow

Out[12]= 2

In[13]:= << SimulationsSystem`ja3`
$ContextPath

Out[14]= {"SimulationsSystem`ja3`", "SimulationsSystem`ja2`", \
"SimulationsSystem`ja1`", "SimulationsSystem`SimulationsSystem`", \
"SimulationsSystem`Models`Basic`", \
"SimulationsSystem`SimulFunctions`", \
"SimulationsSystem`TurnPackages`", \
"SimulationsSystem`EventFunctions`", "SimulationsSystem`Classes`", \
"PacletManager`", "WebServices`", "System`", "Global`"}

In[15]:= pro1[]

Out[15]= 2 + SimulationsSystem`ja1`Private`var2

In[16]:= pro2[]

Out[16]= 3

In[17]:= pro3[]

Out[17]= 3

In[18]:= var1

Out[18]= 1

In[19]:= var2

Out[19]= 1

In[20]:= var3

Out[20]= 1

In[21]:= shadow

Out[21]= 3

In[22]:= << SimulationsSystem`ja1`
$ContextPath

Out[23]= {"SimulationsSystem`ja1`", "SimulationsSystem`ja3`", \
"SimulationsSystem`ja2`", "SimulationsSystem`SimulationsSystem`", \
"SimulationsSystem`Models`Basic`", \
"SimulationsSystem`SimulFunctions`", \
"SimulationsSystem`TurnPackages`", \
"SimulationsSystem`EventFunctions`", "SimulationsSystem`Classes`", \
"PacletManager`", "WebServices`", "System`", "Global`"}

In[24]:= pro1[]

Out[24]= 3

In[25]:= pro2[]

Out[25]= 3

In[26]:= pro3[]

Out[26]= 3

In[27]:= var1

Out[27]= 1

In[28]:= var2

Out[28]= 1

In[29]:= var3

Out[29]= 1

In[30]:= shadow

Out[30]= 1

You can see in the output that only after packages are loaded by themselves they are able to recognize every other packages' exported symbols and functions.

Martin Perez
  • 133
  • 9

2 Answers2

14

General

First of all, let me say that the mutual dependency of this type is usually a sign of sub-optimal design - if your two functions need to access each other, then it may mean that they should actually belong to the same package. However, in some rare cases such design can indeed simplify things.

What you observed can be understood by looking at the mechanics of package loading and symbols creation. I dealt with this problem before, and will re-post here almost verbatim my answer from the MathGroup thread, and extend it a little:

Mutual import of packages in the standard package format

The problem

Suppose you have two packages, the main one, and the helper one. The main one needs some of the functionality of the helper one, and you want to keep the context of the helper package on the context path after the main one loads. But, here is the problem: the helper one also needs some of the functionality of the main one. Such situation may be preferable for better designs in some rare cases.

So, here is a naive attempt:

(* Main package *)

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

and a helper (second) package

(* Helper package *)

BeginPackage["MyHelperPackage`"]

HFun1::usage="This is a helper function";

Begin["`Private`"];

Needs["MyMainPackage`"];

HFun1[x_,y_]:=Fun1[x]+Fun2[y]

End[]
EndPackage[]

Everything looks fine, so the function HFun1 ought to use functions Fun1 and Fun2 from the main package (this was what I was thinking anyway)). Now we try to use this:

Needs["MyMainPackage`"]

HFun1[1, 2] 

(* MyHelperPackage`Private`Fun1[1] + MyHelperPackage`Private`Fun2[2] *)

Fun3[1, 2]

(* 3 (MyHelperPackage`Private`Fun1[1] + MyHelperPackage`Private`Fun2[2]) *)

Alas, what we hoped for did not happen. The point is that the package MyHelperPackage` is read before those definitions in the main package (because listing in in a list of dependent packages in BeginPackage["MyMainPackage`",...] causes Needs to be called on MyHelperPackage` before even the public part of the main package is read, and therefore, while the context "MyMainPackage`" is on the contextpath of MyHelperPackage`, when the latter is read, the function names are not yet there in that context and could not be found. Therefore, MyHelperPackage` made up its own private names for Fun1 and Fun2, which is obviously what we don't want.

Solution #1: hidden import of one of the packages

If we don't need the context of the helper package to remain on the contextpath after the main package gets loaded, then there is no problem at all - import the helper package privately. In that case, the public portion of the main package with all public functions is read first, and this those names are already there by the time the helper package is loaded. Of course, the helper package must still import the main one (hidden import say). In code, this will now look like:

BeginPackage["MyMainPackage`"]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Needs["MyHelperPackage`"];

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

Now we can check:

Needs["MyMainPackage`"]

HFun1[1, 2] 

(* HFun1[1, 2] *)

Fun3[1, 2]

(* 3 (1 + Sqrt[2]) *)

We see that the problem has been solved in part, but that the helper context is then still unavailable: the HFun1[1, 2] evaluates to itself.

Solution 2: using a trick with idle BeginPackage - EndPackage

If we do need to leave the helper package on the context path (as in our example), this can be accomplished by the following trick: add lines

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]
EndPackage[]

to the end of your main package, which in its final form will look like:

BeginPackage["MyMainPackage`"]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Needs["MyHelperPackage`"];

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]
EndPackage[]

In this case, starting with a fresh kernel, everything works as planned:

Needs["MyMainPackage`"]

HFun1[1, 2]

(* 1 + Sqrt[2] *)

Fun3[1, 2]

(* 3 (1 + Sqrt[2]) *)

Solution #3: Use fully-qualified symbol names, in one of the packages

This solution is perhaps not as good as the other ones.The bad thing about it is that fully-qualified symbols couple together the two packages (the one from where they come and the one where they are used), more tightly than it may be desirable. This may lead to various problems, such as regressions due to refactorings, where a function is moved to a different package, or entire package's context is changed. However, in practice this one is often used.

So, the main package would look now just like in our initial naive attempt:

BeginPackage["MyMainPackage`",{"MyHelperPackage`"}]

Fun1::usage="Fun1[x_] squares its argument";
Fun2::usage=  "Fun2[x_] computes a square root of its argument";
Fun3::usage  = "Fun3[x_,y_] is a more complex function";

Begin["`Private`"]

Fun1[x_]:=x^2;
Fun2[x_]:=Sqrt[x]
Fun3[x_,y_]:=(x+y)*HFun1[x,y];

End[]

EndPackage[]

and I decided to use fully qualified names for symbols from the main package used in a helper one (which may make more sense than otherwise, since main package's functions may represent more stable public API):

(* Helper package *)

BeginPackage["MyHelperPackage`"]

HFun1::usage="This is a helper function";

Begin["`Private`"];

HFun1[x_,y_]:=MyMainPackage`Fun1[x]+MyMainPackage`Fun2[y]

End[]
EndPackage[]

Depending on the situation, this version may be a version of choice (e.g. you know for sure that the symbol long names are not going to change etc.), but still I'd avoid this in favor of one of the other mentioned solutions.

Summary

I have presented three different ways to solve the symbol resolution issue for mutual package imports. I think that one should be able to use one of the suggested solutions successfully in the majority of cases.

Now, one may argue that mutual imports may reflect bad design, but IMO they can, very occasionally, be quite handy and lead to actually better designs.

Leonid Shifrin
  • 114,335
  • 15
  • 329
  • 420
  • Great, I get it now, thanks for the explanation Leonid. Now I wonder if what I am doing is a bad practice, mutual import just seemed right to me for what I am doing, but it might be my OOP background impairing my capacity to work better with Mathematica. – Martin Perez Sep 06 '12 at 23:19
  • I am developing a simulation framework to be able to grasp any simulation and it has many natural modules (Simulation main system, object kinds definitions "like classes", events functions, system functions, turn packages "collection of events per object kind to be used in simulation, and the simulation models themselves"), and each of them proposes a global data structure that each other can access and manage per turn, using contexts to change the associated turn with the data structure. It just seems functionally natural that way in desing to me. What do you think or recommend to avoid that? – Martin Perez Sep 06 '12 at 23:25
  • 3
    Any difference between appending the BeginPackage vs appending a Needs? – Rojo Sep 07 '12 at 00:01
  • 1
    Yeah the needs do not keep the package context in the context path. – Martin Perez Sep 07 '12 at 03:24
  • 1
    @MartinPerez I don't think it is your OOP background, but sometimes it is unclear how to separate them. This can lead to circular dependencies (mutual dependence has a circle of size 2). Per Leonid's example, I would consider removing the dependency HFun1 on Fun1 and Fun2 by having them passed as parameters. Or, have settable "constants" like $PreRead. Since functions can be passed like any other variable, I'd consider something like this to break your dependencies. – rcollyer Sep 07 '12 at 03:41
  • @Rojo Martin listed the main difference, which is, private vs public package import. Appending Needs leads to private import, meaning that the context of the imported package won't be kept on the context path after this one had been loaded. There is a more subtle (related) difference. When you indicate extra contexts via the second argument of BeginPackage, they are cached internally. Imagine that you load a given package itself privately, then neither its context nor contexts of those packages it imports would remain on the $ContextPath. However, if you later load this package via ... – Leonid Shifrin Sep 07 '12 at 09:21
  • 1
    @Rojo Needs, not only its context will be added again to the $ContextPath, but also the contexts of all those packages it was importing via the second argument of BeginPackage. At some point, it took me a while to decipher this behavior, since it looks like you call Needs on one context and get several contexts added to the $ContextPath. Of course, this seems natural in retrospect, but this is still something worth knowing. – Leonid Shifrin Sep 07 '12 at 09:24
  • 1
    @MartinPerez My experience is that in Mathematica, mutual imports indicate insufficiently thought out design most of the time. However, there may be rare cases where they might be justified. The usual arguments apply: minimize coupling, maximize coherence. – Leonid Shifrin Sep 07 '12 at 09:26
  • @rcollyer Could you please explain a little more what you mean with settable "constants" like PreRead. I understand Function names are just symbols themselves but cant grasp the relation between that and the PreRead example you are mentioning. – Martin Perez Sep 07 '12 at 17:45
  • @MartinPerez $PreRead is settable to any function you want. So, using Leonid's example, you could have MyHelperPackage` expose two symbols: $HelperFun1 and $HelperFun2 which HFun1 would then use instead of Fun1 and Fun2 directly. This removes the dependency, but the helper functions would have to be set to something reasonable as a default in case you forget to set them. $PreRead is used in that way. – rcollyer Sep 07 '12 at 18:18
  • @rcollyer Could you please post some code example showing your proposed solution to end the dependency? What I do not see is how to assign to PreRead multiple public symbol and function definitions that are gonna call themselves mutually and will be used in multiple packages. Please consider the case of three or more mutually dependent packages symbols/functions, like my ja1,ja2,ja3 example. – Martin Perez Sep 07 '12 at 18:30
  • @MartinPerez you're confusing my example with the illustration of what already exists in Mathematica. $PreRead is a single symbol, so it is incapable of being assigned multiple values. Instead, I advocate that MyHelperPackage` expose two symbols: $HelperFun1 and $HelperFun2, and HFun1 be rewritten to rely on them, not Fun1 and Fun2. For instance, if HFun1[x_, y_]:= Fun1[x] + Fun2[y], initially, it becomes HFun1[x_, y_]:= $HelperFun1[x] + $HelperFun2[y] after the change. This breaks the explicit dependency on the other package. ... – rcollyer Sep 07 '12 at 18:53
  • 1
    @MartinPerez (cont'd.) But, that is only cutting one dependency which breaks apart the cycle. This becomes much more complex with each addition, but in essence you want to find a minimum spanning tree. In your example, ignoring the similarities in the pro functions, I'd look for a way to make two of the packages completely independent, i.e. make them reusable on their own. If that is not feasible, combine some or all of them which is what my methods, above, do. – rcollyer Sep 07 '12 at 19:01
  • @rcollyer So what you are proposing is basically to merge any definitions that may be cause of mutual dependence in the same package, right? In case of mutual dependence with more than 2 packages moving definitions won't do unless you merge all packages into one. Is that what you advocate? Remember that Leonid's HFun example was just simple enough to show how a dependency could be handled without moving the functions definitions to other packages, exactly to avoid doing what I understand you are saying. – Martin Perez Sep 07 '12 at 19:05
3

When implementing Leonid possible solution in the second case (needing contexts to stay in context path) found an important inconvenience (at least for me) when working with the workbench. It is that any symbol in the middle of code belonging to a package that was not included as needed in BeginPackage[] would not be colored GRAY (default color for public/imported definitions) and would not give you the USAGE info. So I think I found out an even better solution that also handles mutual import over many simultaneos packages uniformly and keeps the workbench export utility working. But this is not the perfect solution yet, still have to move above the export block the needs block after exporting symbols

Solution:

The new structure for any package would be

1.-BeginPackage["mainpackage"]

2.-public definitions with usage (its important that all definitions stay before the needs)

3.-needs["helperpackage"] for all needed packages

4.-EndPackage[]

5.-BeginPackage["mainpackage",{"helper1","helper2",...}] (no code between this beginpackage and the begin private statement)

6.-Begin["private"]

7.-The rest as usual

Like this employing my previous example with ja1,ja2,ja3:

Package JA1

BeginPackage["SimulationsSystem`ja1`"]
shadow::usage="shadow"
var1::usage = "var1"
pro1::usage = "pro1"
(* Exported symbols added here with SymbolName::usage *)
Needs["SimulationsSystem`ja2`"]
Needs["SimulationsSystem`ja3`"]  
EndPackage[]

BeginPackage["SimulationsSystem`ja1`", { "SimulationsSystem`ja2`", "SimulationsSystem`ja3`"}]
Begin["`Private`"] (* Begin Private Context *) 
shadow=1;
var1=1;
pro1[]:=var1+var2+var3

End[] (* End Private Context *)

EndPackage[]

Package JA2

BeginPackage["SimulationsSystem`ja2`"]

shadow::usage="shadow"
var2::usage = "var2"
pro2::usage = "pro2"
(* Exported symbols added here with SymbolName::usage *)  
Needs["SimulationsSystem`ja1`"]
Needs["SimulationsSystem`ja3`"]  
EndPackage[]

BeginPackage["SimulationsSystem`ja2`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja3`"}]
Begin["`Private`"] (* Begin Private Context *) 
Needs["SimulationsSystem`ja3`"]
shadow=2;
var2=1;
pro2[]:=var1+var2+var3
End[] (* End Private Context *)

EndPackage[]

Package JA3

BeginPackage["SimulationsSystem`ja3`"]

shadow::usage="shadow"
var3::usage = "var3"
pro3::usage = "pro3"
(* Exported symbols added here with SymbolName::usage *)  
Needs["SimulationsSystem`ja1`"]
Needs["SimulationsSystem`ja2`"]

pro5::usage = "pro5  "  
EndPackage[]

BeginPackage["SimulationsSystem`ja3`", { "SimulationsSystem`ja1`", "SimulationsSystem`ja2`"}]
Begin["`Private`"] (* Begin Private Context *) 
shadow=3;
var3=1;
pro3[]:=var1+var2+var3
pro5[]:=a
End[] (* End Private Context *)

EndPackage[]

Output:

In[1]:= << SimulationsSystem`ja1`
$ContextPath

During evaluation of In[1]:= shadow::shdw: Symbol shadow appears in multiple contexts {SimulationsSystem`ja2`,SimulationsSystem`ja1`}; definitions in context SimulationsSystem`ja2` may shadow or be shadowed by other definitions. >>

During evaluation of In[1]:= shadow::shdw: Symbol shadow appears in multiple contexts {SimulationsSystem`ja3`,SimulationsSystem`ja2`,SimulationsSystem`ja1`}; definitions in context SimulationsSystem`ja3` may shadow or be shadowed by other definitions. >>

Out[1]= {"SimulationsSystem`ja1`", "SimulationsSystem`ja2`", \
"SimulationsSystem`ja3`", "PacletManager`", "WebServices`", \
"System`", "Global`"}

In[2]:= pro1[]

Out[2]= 3

In[3]:= pro2[]

Out[3]= 3

In[4]:= pro3[]

Out[4]= 3

In[5]:= var1

Out[5]= 1

In[6]:= var2

Out[6]= 1

In[7]:= var3

Out[7]= 1

In[8]:= shadow

Out[8]= 1

It Works!

Martin Perez
  • 133
  • 9
  • 1
    Really? Why do we need these tricks? I think the whole Package setup is not nice and completely oldfashioned and there should be something better. I wished these fundamental problems in software design would be addressed in some version of Mathematica. However, it has not changed much since 1987 or so. – Rolf Mertig Sep 07 '12 at 09:37
  • @RolfMertig I disagree with you this once, Rolf. I think the package system is very well designed and thought out, in that the user gets enormous control over all encapsulation mechanisms, most of which are exposed to us. The problem under discussion I view a consequence of this. Should we have a more rigid packaging system, it might have addressed this one, but it would restrict what one can possibly do. I do agree that such a system might be needed, but it can as well be implemented on top of the existing one (which, indeed, has not been done yet). – Leonid Shifrin Sep 07 '12 at 09:43
  • @LeonidShifrin Sure, one can control everything, and I and you can (though I basically end up using full context names a lot), and one can manipulate $Path and alike. But try to explain this to someone new. And that "Workbench" does not help at all with subpackages. It is really really not enough for non-specialists. And even I get annoyed all the time when things get slightly more involved. There are subtle issues with SetDelayed and contexts, also circular Needs can be tricky. And by now I gave up with DeclarePackage. It is just too cumbersome to get it to do what one wants. – Rolf Mertig Sep 07 '12 at 10:36
  • @RolfMertig Yes, I agree, we are just given the raw system not really optimized for professional large-scale software development, and this is just one aspect of it. I hope this will change in the future. The good news is that everything we need can be implemented in the top-level Mathematica. – Leonid Shifrin Sep 07 '12 at 11:28
  • @LeonidShifrin So how would we design a Package["context", {"subcontext1", ...}, body] construct? I.e., I want a package to be a mathematica construct, with a head, and side effects (like modification of $Path), when executed. Might not be too hard. – Rolf Mertig Sep 07 '12 at 13:25
  • 1
    @RolfMertig I would use Block and modify locally the relevant variables ($ContextPath, $Packages and $Context), plus modify globally some of them too ($ContextPath, $Packages). Can't offer a complete solution right now, will think about it, but it does not seem to be too hard. The key point is that the parsing (creation of symbols in the correct context) is completely controlled by the current values of $Context and $ContextPath, which both are exposed to the user. – Leonid Shifrin Sep 07 '12 at 17:42