29

I answered “Equating matrices (or higher order tensors) element-wise” with:

{A, B} = RandomInteger[3, {2, 4, 3, 2}];

Block[{Equal},
  SetAttributes[Equal, Listable];
  A == B
]
{{{False, False}, {False, True},  {False, False}}, 
 {{False, True},  {False, False}, {True, False}},
 {{False, True},  {False, False}, {False, False}}, 
 {{False, False}, {False, False}, {True, False}}}

But it fails if A and B are packed arrays:

{A, B} = Developer`ToPackedArray /@ {A, B};

Block[{Equal},
  SetAttributes[Equal, Listable];
  A == B
]
False

One may observe that setting SetAttributes[Equal, Listable] outside of Block also fails to effect the Listable behavior. This also applies to Unequal and SameQ. This is of interest to me, but my prime concern is that despite the Block, Equal is being recognized and receiving special treatment. I expected symbols localized with Block to behave generically.


  • How does this fit into the main evaluation loop?

  • How can I get get true "blocking" of a symbol such that it behaves generically within a scoping construct?

  • What other symbols besides Equal, Unequal, and SameQ are handled differently?

Mr.Wizard
  • 271,378
  • 34
  • 587
  • 1,371
  • Hehehehehehehehe +1 – Rojo Mar 20 '12 at 17:45
  • @Rojo I couldn't crack this nut myself. Leonid, where are you? – Mr.Wizard Mar 20 '12 at 18:03
  • Can not add much this time. It looks like Equal (and perhaps some other operators) are internally overloaded on packed array arguments for speed, in a way incompatible with the main evaluation sequence. I suspect that this was hard-wired to the system, so that such redefinitions are not even expressed as internal rules (amenable to Block). But these are all just guesses. – Leonid Shifrin Mar 20 '12 at 18:11
  • @Leonid thank you. That's my operating assumption as well, though I find it troubling. – Mr.Wizard Mar 20 '12 at 18:16
  • 3
    I think corner cases like this are to be expected for a high-level interpreted language which contains efficient data structures for special cases. You have to bypass the main evaluator as early as possible to be efficient, and at some point you have to make a choice which evaluation model to use. I am actually surprised that these corner cases are so few. A more consistent way to gain efficiency would probably be through making it possible to compile a larger subset of the language. – Leonid Shifrin Mar 20 '12 at 18:22
  • 1
    @Leonid I suppose then that the only thing "troubling" is my lack of knowledge of Mathematica. :-) – Mr.Wizard Mar 20 '12 at 22:52
  • 5
    Oh no, this is troubling, because this sort of explicitly violates the semantics of the language (albeit in one of its darker corners). I was just saying that it is hard to avoid such violations in a language with two distinct efficiency scales. Compiled languages solve this by having interpreted REPL but also compilation to native code. Some interpreted languages solve it by having JIT compilers in them. The solution currently used by Mathematica probably suits Mathematica as a system but is hardly adequate for it as a general-purpose programming language, from a purist's viewpoint. – Leonid Shifrin Mar 20 '12 at 23:26
  • That is, provided that my guesses about hard-wiring this stuff are right, and this is not just a plain bug. – Leonid Shifrin Mar 20 '12 at 23:28
  • 1
    Using Block[{Equal},SetAttributes[Equal, Listable];Equal[a_,b_]:=Hold[Equal[a,b]];A==B]//ReleaseHold gives {{{False, False}, {False, True}, {False, True}}, {{False, False}, {False, False}, {True, False}}, {{False, True}, {False, False}, {False, False}}, {{False, True}, {False, False}, {False, True}}} – celtschk Apr 10 '12 at 10:24
  • 2
    @celtschk using Trace on your expression v. Mr. Wizard's shows that by redefining Equal, as you do, it is actually threaded over the lists, yet Mr. Wizard's is not. Using SetSystemOptions["PackedArrayOptions" -> "UnpackMessage" -> True] reveals that in both cases some unpacking does occur, but in Mr. W's case we get "Unpacking array with dimensions {1}" v. "Unpacking array with dimensions {4,3,2} in call to Equal" in your case. By redefining Equal it automatically unpacks the full array. – rcollyer Apr 11 '12 at 17:21
  • 1
    Almost every built-in function that has no visible definitions (e.g. downvalues, try Thread, Outer, Flatten, whatever) associated with it seems to have this behaviour. I tested using With[{f = Equal}, Block[{f}, SetAttributes[f, Listable]; With[{r = f[a, b]}, Hold[r]]]]. This property of builtins can be removed by issuing any user-definition (as @celtschk showed above). Another common "unusual" behaviour of builtins is that their definitions take effect after any user-made definitions. – Szabolcs Apr 12 '12 at 15:37
  • Since this seems to be a common feature of all builtins, maybe it's like this to be able to make optimizations for packed arrays. Without this, packed arrays would always be unpacked when passed to a Listable function. Which builtin is Listable that is also optimized for packed arrays I am not sure, I did not find it (I didn't look much either). But if we find one, that could be a support for this hypothesis. – Szabolcs Apr 12 '12 at 15:39
  • I just noticed another case where Block doesn't work as expected with built-in operations: Block[{Plus},Hold@Evaluate[1+1]] gives Hold[2] instead of Hold[1+1] (writing Plus[1,1] instead of 1+1 doesn't change that result). That one cannot even be circumvented with an user-definition: Block[{Plus},Plus[a_,b_]:=plus[a,b];Hold@Evaluate[1+1]] still gives Hold[2], not Hold[plus[1,1]]. – celtschk May 11 '12 at 13:36
  • @celtschk yet Block[{Plus = z}, ToString[1 + 1]] returns "z[1, 1]" -- I guess a lesson in this is that plain Block[{x}, ...] doesn't really clear everything it should, but giving something to replace it with ({x = ...}) helps. – Mr.Wizard May 11 '12 at 21:10
  • 1
    @Mr.Wizard: Block[{Plus},Plus=z;ToString[1+1]] also gives "z[1, 1]". Therefore the difference seems to be between OwnValues and DownValues. – celtschk May 12 '12 at 11:31
  • I am revising some tags around [tag:functions] and it seems like [tag:system-function] is only used here throughout the entire site. Do you have any special plans with it? – István Zachar Jun 22 '12 at 17:01
  • @István at least in the context of this question I have no justification for the "system-functions" tag over the existing "functions" tag. Conceivably it may have a future meaning, such as using functionality of the system on which Mathematica is installed, therefore it should not IMO be a synonym. – Mr.Wizard Jun 22 '12 at 18:22
  • That is not a conclusive answer :) Be reasonable: no one else uses this tag, so I would suggest removing it. You can reintroduce it any time when you feel like there is a serious background for it. Agreed or rejected? – István Zachar Jun 23 '12 at 00:35
  • A different epoch now and probably already noted but for the record it probably was just a bug: A==B returns (False) for both non-packed and packed arrays. – Ronald Monson Oct 06 '20 at 17:32

2 Answers2

2

you are good! Based on your comments, I crafted this:

Block[{Equal, H = Developer`ToPackedArray},
 SetAttributes[Equal, Listable];
 Equal[x_H, y_H] := 
  Equal[Developer`FromPackedArray[x], Developer`FromPackedArray[x]];
 a == b]

which works in both cases.

Update

Mr.Wizard said " I expected symbols localized with Block to behave generically." Although the above works, as mentioned in the comments, there is an observable difference between Equal and other user defined symbols in the rewrite/eval loop in regards to Listable and automatic unpacking of packed arrays. I played with different definitions and couldn't find why this happens.

Clear[f, g, h, z1, z2];
ClearAttributes[{z1, z2}, {Listable}];
f[a_, b_] := Block[{Equal, H = Developer`ToPackedArray},
   SetAttributes[Equal, Listable];
   Equal[x_H, y_H] := 
    Equal[Developer`FromPackedArray[x], 
     Developer`FromPackedArray[x]];
   Print[ a == b  // FullForm];
   a == b];
g[a_, b_] := Block[{Equal, H = Developer`ToPackedArray},
   SetAttributes[Equal, Listable];
   Print[ a == b  // FullForm];
   a == b];
h[a_, b_] := Block[{Equal = z1, H = Developer`ToPackedArray},
    SetAttributes[z1, Listable];
    Print[ a == b  // FullForm];
    a == b] /. z1 -> Equal;
i[a_, b_] := Block[{Equal = z2, H = Developer`ToPackedArray},
    SetAttributes[Equal, Listable];
    Print[ a == b  // FullForm];
    a == b] /. z2 -> Equal;

Then I did

{a , b } = {{1, 2}, {1, 3}};
{c , d} = Developer`ToPackedArray /@ {a, b};

which produces,

In[211]:= f[a, b]

List[Equal[1,1],Equal[2,3]]

Out[211]= {True, False}

In[212]:= f[a, c]

List[Equal[1,1],Equal[2,2]]

Out[212]= {True, True}

In[213]:= g[a, b]

List[Equal[1,1],Equal[2,3]]

Out[213]= {True, False}

In[214]:= g[a, c]

Equal[List[1,2],List[1,2]]

Out[214]= True

In[215]:= h[a, b]

List[z1[1,1],z1[2,3]]

Out[215]= {True, False}

In[216]:= h[a, c]

List[z1[1,1],z1[2,2]]

Out[216]= {True, True}

In[217]:= i[a, b]

z2[List[1,2],List[1,3]]

Out[217]= False

In[218]:= i[a, c]

z2[List[1,2],List[1,2]]

Out[218]= True

So one can see that there is a difference between g[a,c] and h[a,c]: in g Equal does not unpack, whereas in h the user-defined z1 does. I think all the other behaviours can be explained from the evaluation (rewrite) steps as explained in the Mathematica documentation.

Anyway, just wanted to comment finally that although Mr Wizard's is a fair claim. There are a number of areas where other than pure symbolic/rewrite manipulation is occurring, and that simply Block is not probably considering. For example -hope not to trivial for you-, Block[{Equal}, ToExpression["?Equal"]] still prints the Equal documentation, instead of a reference to an undefined symbol. So, like in this case, maybe Equal (and other built-ins) have special behaviour which Block is not touching.

Sorry, I will leave it as answer, but now probably I should say it is not...

Update 2

Actually, just checked that Block leaves ::usage untouched! So, if you do

f::usage = "Symbol f";

then

Information[f]

Symbol f

And if you do Block still you get the same

Block[{f}, Information[f]]

Symbol f

so Block and Information are not working together, even for user-defined symbols!

Last update

As noticed by some, symbols like Plus have special behaviour too. So this

Block[{Plus}, Print[Trace[Plus[1, 2]]]]; (* 1 *)
Block[{Global`Plus}, Print[Trace[Plus[1, 2]]]]; (* 2 *)
Block[{Global`Plus}, Print[Trace[Global`Plus[1, 2]]]]; (* 3 *)

produces

{1+2,3}

{1+2,3}

{}

which demonstrates that Plus retains the built-in behaviour in 1 and 2, being really overridden only with a syntax like 3, which is not convenient. By the way, 2 generates a warning.

Bottom, line, to answer Mr.Wizard's request -I believe this answer was suggested in the comments- it seems the only generic way to override a built-in is to provide your own user defined symbol instead. If one wants to redefine, say Plus, IMHO it is not a burden as whatever definition one wants to introduce can be done with the user-defined symbol and still one has the convenience of the syntactic sugar. To wit

Block[{Plus = plus}, plus[0, _] := 0; Print[Trace[0 + 2]]];

which produces

{{Plus,plus},plus[0,2],0}

So, I will leave it like this.

carlosayam
  • 2,080
  • 16
  • 21
  • isn't this just another example of the observation from the comments - that any user definition will cause the array to be unpacked? For example using Equal[x_dummy]=Null will have the same effect – Simon Woods Dec 19 '12 at 20:31
  • I'm sorry, but I agree with Simon. I don't think this addresses the the heart of my question. – Mr.Wizard Dec 19 '12 at 21:44
  • I thought the point of your question was if Equal had special behaviour in Block, that's what I understood. I edited my answer to clarify this. Besides, I tried the code without the x_H case and it didn't work, so I disagree with the general unpacking behaviour mentioned. – carlosayam Dec 19 '12 at 22:12
  • Apologies, you guys were right @Mr.Wizard, there are observable differences between built-in symbols in Block as opposed to user defined. I wonder if there are more than the ones mentioned, or even if Block cannot fully clear a symbol in all areas of the Mathematica kernel - regardless if it is built-in or user-defined. – carlosayam Dec 20 '12 at 06:38
  • @Mr.Wizard, see Update 2! I bet there is a negative answer to your question "How can I get true blocking of a symbol such that it behaves generically within a scoping construct?" - there is not such thing, even for user-defined symbols. – carlosayam Dec 20 '12 at 07:42
  • +1 for your efforts on this. I think messages are a special case as these are meta-definitions rather than definitions or attributes upon the symbol itself. Block changes the definition (and Attributes but only in a limited fashion as I discovered) of f but not the symbol itself. That is, inside the block f is still f but theoretically "generic" -- nevertheless Messages still exist that reference this name. – Mr.Wizard Dec 20 '12 at 08:01
  • @Mr.Wizard see what will be my last update about Plus – carlosayam Dec 20 '12 at 11:12
  • @Mr.Wizard, what do you mean with Attributes in a limited fashion? – Rojo Feb 11 '13 at 03:28
  • @Rojo I only meant the limitation illustrated in the question. – Mr.Wizard Feb 11 '13 at 10:00
2

Maybe it helps. It's another ways to copy and change internal functions attributes using Function. Here we create a function myEqual just as Equal but listable, so we don't have to use Block:

{A, B} = Developer`ToPackedArray /@ {A, B};
myEqual=Function[Null,Equal[#1,#2],Listable];
A ~myEqual~ B

{{{True,False},{False,False},{False,False}},{{False,False},{False,False},{False,False}},{{False,False},{False,False},{False,False}},{{False,False},{False,False},{False,False}}}

Note that Null inside Function is just an undocumented property. If you get more comfortable you can just use: Function[{a,b},Equal[a,b],Listable];

You can make my equal local, just as you did in your example:

{A, B} = Developer`ToPackedArray /@ {A, B};
Module[{myEqual},
    myEqual=Function[Null,Equal[#1,#2],Listable];
    A ~myEqual~ B
]

Update

As @OleksanrR. commented. A clever way to define myEqual is:

Function[Null, Equal[##], Listable]

So we don't have to restricted to 2 arguments.

Murta
  • 26,275
  • 6
  • 76
  • 166