How symbol lookup actually works

Question

Bug introduced in V6 and fixed in V11.3

The behavior indeed changed but now the documentation is clear about it.

---

This code is inconsistent with the description from Power Programming with Mathematica:

x = 5;
temp`x = 6;
Begin["temp`"]
{x, Global`x, temp`x}

The result in my Mathematica session is {5, 5, 6}, but it's {6, 5, 6} in Section 8.1.1 of Power Programming with Mathematica (page 231 of the hardcopy or page 249 of the PDF).

Answer 1

This behaviour has changed since that book was published. I am writing this additional answer to make it clear how Mathematica 9 searches contexts for symbols and that even the current version 9 documentation is incorrect in describing this.

How symbol lookup actually works

When you enter a symbol name such as x, Mathematica will check if a symbol with this name already exists. It will first search the contexts from $ContextPath for x, one by one. If it doesn't find it there, it'll search the context from $Context for it. If it still doesn't find it, then it will create a new symbol named x in $Context.

Thus $ContextPath controls where to look for symbols, while $Context controls where to create new symbols.

Your observations are explained by these rules, noting that Begin will change $Context only but not $ContextPath. Note that BeginPackage will change both $Context and $ContextPath.

Warning: the documentation contains an error in Mathematica versions older than 11.3.

The $ContextPath documentation states that

$ContextPath is a global variable that gives a list of contexts, after $Context, to search in trying to find a symbol that has been entered.

In fact $ContextPath is searched before $Context in the current version.

In old versions this was not the case, as the Wagner book describes. I don't know when the change happened.

The Contexts tutorial does correctly state the order of search in the current version:

Since $Context is searched after $ContextPath, you can think of it as having "." appended to the file search path.

Answer 2

Not an answer. Just screen shot. I booted up version 2.2 to verify the book result. And it does verify:

The same result we get with version 5.2:

---

Something changed since the book was written (screenshot from version 9 follows):

Answer 3

It appears since version 3 shadowed variables are given priority, as demonstrated below. In the temp` context x is taken as temp`x unless Global`x exists.

Remove[temp`x, Global`x]
temp`x = 6;
Begin["temp`"];
{x, Global`x}
End[];

{6, x}

Remove[temp`x, Global`x]
temp`x = 6;
x = 5;
Begin["temp`"];
{x, Global`x}
End[];

{5, 5}

Answer 4

This is because of $ContextPath. If the symbol outside the Begin/End is created in a context that is listed in $ContextPath, Mathematica will recognize it as belonging to that context even when called from a different context (even if Begin/End is put in a package file and called via Get):

$ContextPath

{"DocumentationSearch`", "ResourceLocator`", "PacletManager`",
 "QuantityUnits`", "WebServices`", "System`", "Global`"}

WebServices`x;
Begin["temp`"];
Print@Context@x;
End[];

 WebServices`

Now if I remove "WebServices`" from $ContextPath:

$ContextPath = DeleteCases[$ContextPath, "WebServices`"];
WebServices`x;
Begin["temp`"];
Print@Context@x;
End[];

temp`

Now what has changed in related code in successive versions, I have no idea.

Answer 5

It seems that when the book appeared, the behaviour of how a symbol is resolved was different. We have two important things:

• the current $Context which is usually Global` unless you change it with e.g. Begin as you did
• the $ContextPath which is a list of contexts that are searched when you type in a symbol like x without explicit context

Now there seems to be confusion how a symbol is found. The documentation to $ContextPath reads

$ContextPath is a global variable that gives a list of contexts, after $Context, to search in trying to find a symbol that has been entered.

This suggests that it is first checked whether a symbol is in the current $Context and after that all entries in $ContextPath are checked.

This is how it seemed to work when the book was written and it is completely understandable that you are confused why it behaves differently. In this tutorial it is written the other way around

Since $Context is searched after $ContextPath, you can think of it as having "." appended to the file search path.

Therefore, let me explain your code as I understand what happens.

x=5  
temp`x=6;
{x,temp`x}

You create two symbols in different contexts, the first one in Global` and the last one in temp` . As I understand it, the x in the last line is the global x because the context Global` is in $ContextPath rather than because your current $Context is Global`

Begin["temp`"]
{x,Global`x}
{temp`x,Global`x}
End[]

With the Begin you set $Context to temp` , but that doesn't matter, because when you refer to x, it is found first in the Global` context which is an element of your $ContextPath. The rest should be clear, because when you specify the context of a symbol explicitly, there is no room for guessing.

Finally, please note that you can refer to the symbol x in the current context (meaning in $Context) by using `x and therefore

Begin["temp`"]
`x
End[]

will give 6.

Appendix

I haven't seen that almost the same question was already asked and has an answer which goes along what I wrote. Another important aspect was mentioned there that I haven't said: $Context controls where a new symbol is created (if there couldn't be found an existing one). Please read the following:

Result about Context is inconsistent with the description of “Power Programming with Mathematica”

Answer 6

(this is wiki answer, as just noticed it is duplicate, but will keep it here for easy reference)

The result is different from Version 2.2 and version 10.1, this is just to show the difference. Something changed between 1993 and today:

Answer 7

I think the following also fits the present discussion.

Who shadows whom?

When you run

(*cleanup*)
$Context = "Global`";
Quiet@Remove["temp`*", "Global`*"];
(*--*)
Global`x = "global";
BeginPackage["temp`"];
temp`x = "temp";

The last line generates the message

...::shdw: Symbol x appears in multiple contexts {temp`,Global`}; definitions in context temp` may shadow or be shadowed by other definitions. >>

(*cleanup*)
$Context = "Global`";
Quiet@Remove["temp`*", "hemp`*", "Global`*"];
(*--*)
BeginPackage["temp`"];
temp`x = "temp";
Global`x = "global";

generates

...::shdw: Symbol x appears in multiple contexts {Global`,temp`}; definitions in context Global` may shadow or be shadowed by other definitions. >>

It looks like even the developer of this particular message was unsure about what exactly happens here ("may shadow or be shadowed").

The following would more correct in the first case:

...::shdw: Symbol x appears in multiple contexts {temp`,Global`} on the context path. Definitions in the first context, temp`, will shadow any other definitions. >>

The message should be the same in the second case, the order of the contexts should be exactly as they are in Append[$ContextPath, $Context].

Why is this message more correct?

A symbol x can be defined in multiple contexts that are not on the context path just fine, the following generates no error message:

Global`x = "global";
foo`x = "foo";
bar`x = "bar";

By the way: You get no error message if you later do AppendTo[$ContextPath, "foo`"], even though the shadowing 'ambiguity' problem now exists.

The "::shdw" message should really be saying that within the set of contexts Append[$ContextPath, $Context] ("the context path") there exist two or more symbols with SymbolName x.

The original message is not clear about which definition will be given precedence, although the system knows this unambiguously: When an unqualified symbol x appears, it will be searched in Append[$ContextPath, $Context] in-order (it will be created in $Context if it cannot be found).

BeginPackage prepends the mentioned context to the context path, such that its definitions will take precedence.