Name rewriting for Function fails in closure

Question

At its heart, Mathematica is a dynamically scoped language. While the choice of dynamic scope for Mathematica is very much a defensible one, lexical scope is too useful to do without, and Mathematica tries to fake it. Unfortunately, the abstraction is leaky.

One of the leaks comes is visible in constructs like With and Function, where symbols naming formal parameters sometimes must be changed to avoid the possibility of clashes. An example of this is:

In[1]:= With[{fun = Function[x, Take[x, 2]]},
         gun[x_] := fun[x];

         gun[{1, 2, 3}]]
Out[1]= {1, 2}

This is exactly what we wanted to have happen, and we can see how Mathematica pulled it off:

In[2]:= DownValues[gun];
Out[2]= {HoldPattern[gun[x$_]] :> Function[x, Take[x, 2]][x$]}

The formal parameter x for gun was changed to x$, so it does not clash with the formal parameter of the anonymous function bound to fun by the With statement.

Now, say we do this sort of thing often enough that we abstract out the construction of that anonymous function:

In[3]:= makeFun[len_] :=
         Function[x,
          Take[x, len]];

We can see signs of potential danger already:

In[3]:= makeFun[2] 
Out[3]= Function[x$, Take[x$, 2]]

The x is already being renamed, and the renaming could clash with other renamings. We can test this:

In[4]:= With[{fun = makeFun[10]},
         hun[x_] := fun[x];

         hun[1]]
Function::flpar: Parameter specification 1 in Function[1,Take[1,10]] should be a symbol or a list of symbols. >>
Function::flpar: Parameter specification 1 in Function[1,Take[1,10]] should be a symbol or a list of symbols. >>
Out[4]= Function[1, Take[1, 10]][1]

Ouch. Checking DownValues indicates that the predicted name clash did indeed come to pass:

In[5]:= DownValues[hun]
Out[5]= {HoldPattern[hun[x$_]] :> Function[x$, Take[x$, 10]][x$]}

Is there any way (perhaps with the right suite of init.m tweaks) to keep this from happening? It really compromises the ability to treat functions as first class objects in a modular way.

FWIW, this is something I saw for the first time in Mathematica 10.2. I have vague memories of doing this sort of thing is the past without particular difficulty, but I could well be mistaken on that count.

Answer 1

One possible workaround is to "turn off" the name rewriting that Function does is makeFun, and provide a name for the formal parameter you know is unique. There are several ways of doing this, but this one is mine:

In[6]:= Module[{x, function},
          Attributes[function] = HoldAll;

          makeFun[len_] :=
           function[x,
            Take[x, len]] /. function -> Function];

This works:

In[7]:= With[{fun = makeFun[2]},
         pun[x_] := fun[x];

         pun[{1, 2, 3}]]
Out[7]= {1, 2}

In[8]:= DownValues[pun]
Out[8]= {HoldPattern[pun[x$_]] :> Function[x$217974, Take[x$217974, 2]][x$]}

The unique symbol created by Module[{x}, ...] is used as a formal parameter name, protecting us from accidental clashes (though if we really wanted to, we could probably engineer a name clash on purpose).

Answer 2

Here are a couple of other options:

Use system option "StrictLexicalScoping"

If you use

SetSystemOptions["StrictLexicalScoping" -> True]

Then your code runs fine (I changed the input to hun to avoid other errors):

With[{fun = makeFun[10]}, hun[x_] := fun[x]; hun[Range[15]]]

(* {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} *)

Reasonably general top-level solution

A few years ago I wrote a tiny framework to deal with such problems (that was before Daniel did his work and made the "StrictLexicalScoping" option available). So, we can do

SetSystemOptions["StrictLexicalScoping" -> False];
ClearAll[hun];

to get back to the original state, and then load the code:

Import["https://gist.githubusercontent.com/lshifr/1683497/raw/AutoRenamings"]

Use runWithRenamings locally

And now:

runWithRenamings[
  With[{fun = makeFun[10]}, hun[x_] := fun[x];
  hun[Range[15]]]
]

(* {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} *)

which is, wrap your code where definitions are made, into runWithRenamings. For the case at hand it produces this definition:

?hun

Global`hun

hun[x$2879$_]:=Function[x$,Take[x$,10]][x$2879$]

In general, runWithRenamings attempts to ensure that the renamed symbols produced via the renaming machanism will always be unique.

Dynamic environment based on runWithRenamings

You can make a dynamic environment that would catch the code using e.g. SetDelayed, which is a common enough use case:

ClearAll[withAutoRenamings];
SetAttributes[withAutoRenamings, HoldAll];
withAutoRenamings[code_] :=
  Internal`InheritedBlock[{SetDelayed},
    Module[{inSetDelayed},
      Unprotect[SetDelayed];
      s_SetDelayed /; ! TrueQ[inSetDelayed] := 
         Block[{inSetDelayed = True}, runWithRenamings[s]]; 
      Protect[SetDelayed];
      code]]

This will work also in cases like the following:

def[] := With[{fun = makeFun[10]}, hun[x_] := fun[x];hun[Range[15]]]

so that

withAutoRenamings @ def[]

(* {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} *)

while runWithRenamings @ def[] would still produce errors.

Remarks

Unless you have strong reasons not to use "StrictLexicalScoping" option (such as, e.g. if this setting somehow breaks some legacy code, where things that "StrictLexicalScoping" -> True fixes, were used as features), you will be better off using it.

The top-level code I gave mostly as an illustration, and also there you may have a more direct control over things, but it is surely incomplete, and it also generally requires that you wrap the runWithRenamings environment around the problematic piece of code.