Scoping problem?

Question

Why

(f[1, 0] /. {x : 0 | 1 :> 1 - x}) -> 0

returns (expectedly) f[0, 1] -> 0, while

Module[{x}, (f[1, 0] /. {x : 0 | 1 :> 1 - x}) -> 0]

returns f[1 - x$2749, 1 - x$2749] -> 0? I can not imagine a consistent explanation for this behavior. There since Mma 7 at least.

Answer 1

Ultimately, the culprit is not simply the lexical scoping of Module, as comments have mentioned, but the scoping rules for Rule and RuleDelayed.

tl;dr: x_ appears to belong to the scope of the outermost ->, not the innermost :>, and so x is not lexically scoped by :>—even though it would be lexically scoped and protected from replacement in Module in other cases, e.g. if we got rid of the top-level rule.

The following is a long discussion about the issue.

What's not the problem: how Module behaves with rules

I want to contribute a few more examples, which I think illustrate how both the rule scoping and module scoping works:

 In[1]:=  x = 100; (x_ :> x + 1) -> 0
Out[1]:=  (x_ :> x + 1) -> 0
In[2]:=  x = 100; (5 /. x_ :> x + 1) -> 0
Out[2]:=  6 -> 0
In[3]:=  Module[{x = 100}, x_ :> x + 1]
Out[3]:=  x_ :> x + 1
In[4]:=  Module[{x = 100}, (x_ :> x + 1) -> 0]
Out[4]:=  (x_ :> x$90724 + 1) -> 0
In[5]:=  Module[{x = 100}, (5 /. x_ :> x + 1) -> 0]
Out[5]:=  101 -> 0
In[6]:=  Module[{x = 100}, Hold[x]]
Out[6]:=  Hold[x$98978]

Note that all of the above behavior remains if we replace -> with :>.

These are a lot of examples all at once, but we'll use them.

Importantly, changing Module to Block produces the "expected behavior" in Out[1] and Out[2]. Module uses lexical scoping, while Block uses dynamic scoping.

To see how this works, consider Ins 4, 5, and 6. They show that even held expressions get their variables renamed. :> has the attribute HoldRest, meaning that its latter part is held. In In[5], x$NNNNN is released from its hold via the use of /., and evaluated. (This also shows us that variables are initialized only after renaming takes place; giving a symbol a definition is a dynamic operation, not a lexical one.)

But, even though Module renames held symbols, it avoids renaming lexically bound symbols.

This means that for Module the free symbols (ones that are not local to another construct inside it) are renamed before evaluation; for an example of a symbol that is local to a construct inside module, and therefore not renamed, consider Module[{x = 5}, Module[{x = 3}, x]], which returns 3.

This is supposed to be the case for Rules too. From the documentation for Rule:

Symbols that occur as pattern names in lhs are treated as local to the rule. This is true when the symbols appear on the right-hand side of /; conditions in lhs, and when the symbols appear anywhere in rhs, even inside other scoping constructs.

Further, consider the statement from the documentation for RuleDelayed:

Module and With do not affect local variables of RuleDelayed

with the example

In[7] :=  Module[{x = 1}, a /. x_ :> x + 1]
Out[7]:=  1 + a

which is equivalent to our In[3].

So, what's happening here?

Let's describe what happens in the case of In[3]. :> looks at the symbols on its rhs, sees which appear as pattern names on its lhs, then binds them to be local to the rule, protecting them from Module's lexical scoping.

In contrast, in the case of In[4], I believe that the x_ on the lhs of x_ :> x + 1 belongs to the scope of the outermost Rule, not the innermost RuleDelayed, and is therefore not a symbol on the rhs of :> which is named by a pattern on its lhs, and so is not bound by :>. Further, since the outermost Rule does not consider symbols on its lhs which do not name patterns, such as x, to be local, it releases x to be bound and renamed by Module.

(Note again that the distinction between Rule and RuleDelayed is only relevant insofar as RuleDelayed has the attribute HoldRest and holds x + 1; replacing Rule with RuleDelayed produces the same behavior.)

This is unusual to me. I personally think the x_ there should belong to the scope of the inner :>, not the outer ->.

To bolster the notion that this is in fact what is going on, consider the following, which shows the latter-mentioned fact that symbols on the lhs of Rule don't get lexically bound by named patterns on the lhs:

In[8] :=  Module[{x = 100}, Hold[{x_, x} -> x]
Out[8]:=  Hold[{x_, x$100384} -> x]

This shows us that the x named by a pattern on the lhs is only in the scope of Rule when it's on its rhs. Otherwise, it's not local to Rule, and Module grabs it.

This is fine. The issue is, as you've pointed out, when there's an inner rule which seems like it ought to lexically scope the x, and doesn't! So, take another look at In[4] now; note that x acts like any other symbol appearing on the lhs of Rule, instead of being lexically bound by being on the rhs of :>.

Your example corroborates the first notion: if x_ belonged to :>, then x would be lexically scoped in the :> expression, and the replacement would proceed as expected.

Is this behavior good for users?

I can maybe understand why one might want the above behavior: then all patterns in the lhs of any rule are indeed patterns for that rule, which you could use in replacements. But this is still true even if you change the lexical scoping rules to make them more intuitive, since evaluation essentially implements the expected scoping behavior via attributes like HoldRest. In contrast, as we've seen, Holds are transparent to lexical scoping.

The main thing wrong with it for me, really, is the unexpected failure of lexical scoping with nested rules, which is, as far as I can tell, not necessary for dynamic evaluation to behave as expected even in cases where one wants to use a whole rule as a pattern for a replacement—but I could be wrong on that.

Why should a user expect In[2] to produce different output than In[5]? The lexical scoping is "not compatible" with the dynamic scoping. Admittedly, in many cases this is exactly what we want. We want to transform held expressions in place before evaluating, as Module and With do.

The problem is that the evaluation rules create the appearance of lexical scoping while using a different mechanism, which makes the apparent scoping in In[2] (x = 100; (5 /. x_ :> x + 1) -> 0) a red herring. It works for a reason one wouldn't expect it to, given how Module does respect lexical scoping by :>: the x here only appears well-scoped because it is held.

In either case, I think granting the lexical scope to the outermost rule instead of the innermost one (assuming that I'm correct in that being what's happening) is counterintuitive for the user, and relies on evaluation timing instead of the apparent scoping suggested by both the syntax highlighter and the behavior in Module in non-nested cases. I'd be curious why the decision was made to grant the outermost rule lexical priority!