47

Leonid Shifrin once wrote (excerpted):

(Unevaluated) is one of a very few "magic symbols", along with Sequence and Evaluate - these are deeply wired into the system and can not be easily replicated or blocked, unlike Hold - in that sense, Unevaluated is more fundamental.

HoldPattern is a normal (usual) head with HoldAll attribute for the purposes of evaluation, but its magic shows in the pattern-matching: it is invisible to the pattern-matcher, and is very important ingredient of the language since it allows pattern-matcher to be consistent with the evaluation process.

David B. Wagner writes:

The symbols Evaluate, Unevaluated, and Sequence are magic cookies. This is a whimsical term used by computer scientists to refer to any type of value that has special significance to the software system of which it is a part. The behavior of these symbols is not the result of any values or attributes they possess; rather, it is "wired into" the kernel. This implies these behaviors simply cannot be altered, nor can they be duplicated.

These are fundamental to the language but they are not widely recognized as such. I have never attempted to make an exhaustive list of these "magic Symbols" but I think it would be useful to the community to have one.

What are all the "magic" Symbols in the Mathematica language?

What makes each of these Symbols special?

  • I believe these should be considered distinct from atomic objects, which while also being "deeply wired into the system" and "not easily replicated" are a form of data types with special handling.

A related question with a somewhat different scope:

user64494
  • 26,149
  • 4
  • 27
  • 56
Mr.Wizard
  • 271,378
  • 34
  • 587
  • 1,371
  • ToExpression will definitely be one. – Wjx Aug 02 '16 at 10:27
  • @Wjx I would argue that it is not, at least by my understanding of the way others are using this therm. ToExpression does not seem to involve any special evaluation. That it performs I complex task does not make it "magic" since conceivably we could reimplement it ourselves. – Mr.Wizard Aug 02 '16 at 10:34
  • May I ask how? It seems that it's hard to convert an string to a complex expression? – Wjx Aug 02 '16 at 11:04
  • @Wjx A lot of hard work! I mean that with enough effort one could write out all the parsing rules and create an expression assembler. Leonid has actually done work in this area I believe. In no way do I mean to suggest that the function is trivial; far from it! However I am saying that it does not require anything that does not already exist in the language, as far as I know. By contrast we cannot write Unevaluated robustly ourselves. – Mr.Wizard Aug 02 '16 at 11:20
  • 3
    Oh, gotcha! You mean it's writable, but in contrast, Unevaluate or so is simply CANNOT be written directly. – Wjx Aug 02 '16 at 11:25
  • Three are things that appear as magic cookies to the notebook user because the front-end transforms their input strings into something strange before it is seen by an evaluator (or maybe it is never seen by an evaluator). How do these fit into your question? – m_goldberg Aug 02 '16 at 17:07
  • 3
    Would you consider things like Catch/Throw or Check/CheckAbort/CheckAll to fall into the "magic" category? They pretty much need special kernel hooks to function correctly, but it feels like a different level of magic than Unevaluated etc. – WReach Aug 02 '16 at 17:29
  • @WReach Thanks for the comment. I think those things should be mensioned here in their own category, like the pattern object "meta-expressions." I'll add that to my answer later, unless you would like to post separately about it. This question can be adjusted as necessary. I think all of this information is worth collecting. – Mr.Wizard Aug 02 '16 at 23:46
  • 3
    @WReach - I think Catch and Throw could be written - they're non trivial and would involve Catch rewriting the entire expression given as its first argument - but LISP macros are written do that kind of thing a lot and I don't think there's anything special about them here. I believe the Check family can be written as well. (For the curious, Let Over Lambda is probably the best book on LISP macros ... ever!) – davidbak Aug 03 '16 at 00:14
  • 1
    @davidbak It is exactly this kind of discussion/information that I hoped this question would inspire. Thank you. (I know this site isn't supposed to be for discussions per se, but discussion for the purpose of developing questions or answers is always welcome.) It sounds a little strange but it might be good to start an answer to list things that are not "magic" Symbols though they at first appear so, with explanations. – Mr.Wizard Aug 03 '16 at 05:21
  • How about operators like << and >>? – Brett Champion Aug 03 '16 at 19:20
  • Sooo... what's your definition of "magic" here, @Mr.Wizard? It is entirely not clear to me, and I couldn't grasp the essence - other than the ostensive definition you provide by picking certain functions that you find "magical". Could you perhaps provide a more solid organizing principle? Is it about being written entirely in C (irreproducible in Mathematica)? Or about functions controlling evaluation? – István Zachar Aug 03 '16 at 19:23
  • @István I am not quite sure myself! That's one of the things I am trying to figure out. Leonid Shifrin, David B. Wagner, and Robby Villegas all use the term; I did not create it. If the only "magic Symbols" are "Evaluate, Unevaluated, and Sequence" then I want to know why certain other symbols are not. (I think Szabolcs makes a convincing case that Function is also a "magic Symbol" for example.) I also want to identify related classes like pattern "meta-expressions" which have certain "magic" attributes yet are distinct. – Mr.Wizard Aug 04 '16 at 02:38
  • 2
    Would you consider Update as magic? – Greg Hurst Mar 26 '21 at 14:58
  • @ChipHurst I cannot think of how that would be implemented so I think it is. – Mr.Wizard May 05 '21 at 15:52
  • 1
    @ChipHurst Everything in the 'Basic examples' section of the Update documentation is not magic, it is black sorcery of the darkest kind. – Emilio Pisanty May 05 '21 at 16:07

3 Answers3

28

uh oh, another "work in progress" answer from me that may never be finished.

Robby Villegas, Working with Unevaluated Expressions:

Unevaluated is not meant to be a function or stable data type. It is to be used as a wrapper on an argument in stage 1, before argument evaluation. It is a signal to the evaluator to suppress the usual evaluation of that argument.

It is transparent to the function receiving the argument. You can think of it as a shuttle giving the argument safe transport to the function's code, keeping the evaluator away. Unevaluated vanishes before the argument is fed to the function, since its purpose is fulfilled.

As Szabolcs wrote:

I'll also note that Function is a very special symbol that has very special evaluation rules: the evaluator will hold y in an expression of the form f[x][y] only when f is Function (and in no other situation). In particular, it is not possible to bestow a custom symbol with this property.

  • Pattern elements

Arguably a distinct class, these Symbols are "magic" only in pattern matching and not main evaluation itself. Robby Villegas describes these as having "meta-expression status."

Via a method from jkuczm, May 8 '15

    {Alternatives, Blank, BlankNullSequence, BlankSequence, Condition, Except,
    HoldPattern, IgnoringInactive, Literal, Longest, Optional, OptionsPattern,
    OrderlessPatternSequence, Pattern, PatternSequence, PatternTest, Repeated,
    RepeatedNull, Shortest, Verbatim}

Some nonstandard evaluation appears to take place here.

undetermined

These might have limited "magic" behavior in one form or another.

It is not immediately apparent to me how to fully emulate the partial evaluation used on the left hand side of Set; perhaps special evaluation methods are used. Still this probably doesn't belong here.

These also probably don't belong here but they certainly are unusual.

  • Attributes, like Orderless, Temporary, Listable, NHoldAll, etc., cannot be re-implemented in top-level code robustly. They would seem to fit the description of "... any type of value that has special significance to the software system of which it is a part."
Mr.Wizard
  • 271,378
  • 34
  • 587
  • 1,371
  • 4
    There is also Nothing. It is like Sequence[], but it only gets removed from lists. – Szabolcs Aug 02 '16 at 09:51
  • @Szabolcs Unfortunately it is not present in the version I use therefore I cannot examine its behavior, or definition if such exists. Have you looked into its mechanism at all? – Mr.Wizard Aug 02 '16 at 09:52
  • 2
    @Szabolcs What exactly is magical there? Common use case can be coded e.g.: not /: {a___, not, b___} := {a, b} – Kuba Aug 02 '16 at 09:53
  • @Kuba Okay, I just reviewed that section. I don't see anything that really fits the description of "magic" quoted in the question, e.g. While should be easy to emulate. Trace does things "we" cannot but I don't think it really fits here, though I forgot about it or I might have grouped it with Stack. – Mr.Wizard Aug 02 '16 at 10:07
  • Maybe it isn't special then. – Szabolcs Aug 02 '16 at 10:12
  • $ConditionHold probably falls into the undetermined category, right next to RuleCondition. – WReach Aug 02 '16 at 15:57
  • MatrixForm doesn't seem that special, but I couldn't work out how to replicate its behaviour. – mikado Aug 02 '16 at 19:43
  • @mikado MatrixForm basically is a formatting rule that creates a certain Box form (GridBox I believe, without checking). For the purpose of this question I am am not considering special Front End box forms and behaviors "magic" -- I should probably clarify that. – Mr.Wizard Aug 02 '16 at 23:48
  • Why is OrderlessPatternSequence in blue above? It's a built-in symbol. Someone needs to update the Mathematica keywords on this site. Who to contact? – QuantumDot Aug 04 '16 at 09:47
  • @Quantum, admittedly it hasn't been updated in a while. – J. M.'s missing motivation Aug 08 '16 at 14:14
  • Regarding the Function discussion, it is possible for f[x][y] to hold y. For example ResourceFunction["AbortOnMessage"][Table[First[i], {i, 5}]] holds the Table expression. It looks like it achieves this through a clever use of StackInhibit. – Greg Hurst Mar 26 '21 at 14:55
  • @ChipHurst Leonid showed how to do this with Stack trickery in the linked Q&A. Do you think that Function should not be included here? I don't think this is quite the same. – Mr.Wizard May 05 '21 at 15:59
  • I'd keep it here, the Stack magic seems kludgy. – Greg Hurst May 05 '21 at 20:36
13

[This response is intended as a "big list" contribution rather than a comprehensive treatment.]

Program Control Symbols

If we define "magical symbol" to mean a symbol whose implementation cannot reasonably be achieved in user code, then the various program control symbols would almost certainly qualify.

Language Features:

While it is possible in principle to implement the language features in user code through clever code transformation, the effort amounts to replacing the built-in evaluator completely. The reason is that Mathematica supports evaluation of code generated at run-time. Static analysis of the code is not sufficient to identify all evaluatable code -- the program must actually be run. Thus, the code transformer would have to take over the full evaluation process in order to accomplish its task. The implementation effort would be larger than writing the built-in evaluator in the first place -- and the program control symbols would be "magical" in the rewritten evaluator.

Even Common Lisp, with far more well-defined evaluation semantics than Mathematica, was forced to provide throw, catch, go and unwind-protect as special forms for the same reasons.

Some function interact with the evaluator:

These are somewhat less magical than the language features, but they do require access to internal state of the evaluation implementation.

Community
  • 1
WReach
  • 68,832
  • 4
  • 164
  • 269
  • I believe your description of Common Lisp's implementation of throw and catch is incorrect. They are "special forms" because catch has an implicit progn. "Special form", in Common Lisp, does mean "special evaluation" but in this case it is no more than not evaluating all arguments before the call. (In other words, it is the HoldAll attribute.) It could have been provided as a macro. (And perhaps is, in some implementations.) The reason these things are harder in Mathematica than Common Lisp (or Scheme) is that the Mathematica evaluator has a lot more kinds of "special form". – davidbak Aug 04 '16 at 02:56
  • @davidbak I think if it were possible to implement catch as a macro, it would have been. That was a driving goal in the design of Common Lisp: to reduce the set of special forms to the absolute minimum possible. – WReach Aug 04 '16 at 14:11
2

The behavior of GeoGraphics`$GeoBackgroundType (I think introduced in V12.2) is quite unusual and might not be readily reproducible top-level.

This is an internal flag to toggle between vector & raster geo data. It's use is explained around the 8:00 mark here.

It's use:

GeoGraphics[]


GeoGraphics`$GeoBackgroundType = "Legacy";

GeoGraphics[]

The strange behavior is with Unset and Clear:

GeoGraphics`$GeoBackgroundType=.

GeoGraphics`$GeoBackgroundType

"Hybrid"

Clear[GeoGraphics`$GeoBackgroundType]

During evaluation of In[9]:= 
    Clear::spsym: Special symbol GeoGraphics`$GeoBackgroundType cannot be cleared.


GeoGraphics`$GeoBackgroundType

"Hybrid"
Greg Hurst
  • 35,921
  • 1
  • 90
  • 136