16

This question is inspired by that one. Reading it, I noticed that while (I think) I know what happens when entering an expression directly to the kernel (i.e. after typing math in the terminal), the notebook adds extra complications which I don't fully understand.

To start with, I describe what I think happens when interacting directly with the Kernel (so you can correct me if I'm wrong):

  • After the expression is input, it first is a string, which is passed to $PreRead (if set). That function is expected to return a string, which gets assigned to InString[$Line].
  • Then this string is parsed (I guess with MakeExpression), and the resulting expression assigned to In[$Line]
  • Then $Pre is applied to this expression, if defined.
  • Finally, the expression obtained this way is evaluated

(After the expression is evaluated, other things happen to generate the output, but that's outside the scope of this question.)

Now, for Notebooks, things are obviously more complicated:

  • To start with, as far as I understand, the notebook cells contain boxes, not strings. However I noticed that nevertheless InString is set. What is not clear to me is whether this string is part of the processing, or if the boxes-to-string conversion is only done in order to set InString.
  • Next, there's the Notation package, which also is involved in the pre-evaluation process. However being a package, I guess it has to hook into some lower-level interface to do its work.
  • There's also the fact that notebooks can define their own private contexts. Those contexts also have to somehow be communicated to the kernel. Since expressions from different notebooks can be sent to the kernel in arbitrary order, those contexts need to be associated with the expression, and setting them must thus be part of expression pre-evaluation.
  • There might be other things which I'm not aware of.

Therefore my question is:

What is the exact sequence of events which happens when submitting a notebook cell, as far as customizable behaviour is concerned?

Community
  • 1
celtschk
  • 19,133
  • 1
  • 51
  • 106

2 Answers2

17

Preamble

I will interpret your question in the narrow context of customizibility. Most of the information can be found in the documentation, but is alas scattered over several places. Below I will attempt to assemble a single sequence, mostly using this source. Where my description deviates from it, I will put the LS sign indicating that what follows ismy interpretation. Also, I have to make a disclaimer that some parts of this I haven't tested thoroughly enough to be fully sure that I am 100 percent correct, so I invite anyone to edit this if some mistakes are found.

Evaluation sequence

With that out of the way, here is what I believe is a reasonably accurate description of what happens when you press SHIFT+ENTER to evaluate a cell in the FrontEnd session:

  • Evaluate CellProlog, if defined for a given cell (LS: CellProlog is an option for the cell which defines a piece of code to evaluate, not a function of the input).

  • Evaluate CellEvaluationFunction, if defined for a given cell

    • In a StandardForm or TraditionalForm cell, CellEvaluationFunction is applied to the BoxData expression representing the input to be evaluated.
    • In an InputForm cell, it is applied to the string corresponding to the input to be evaluated.

    LS: In all cases, this is a function which is applied to the input (string or boxes). At this point, one can divert the entire evaluation loop as one wants, including evaluations not involving Mathematica kernel at all (e.g. code in other languages, processed externally. See this excellent answer by WReach for the exposition of the possibilities this offers).

    All the steps below assume the default CellEvaluationFunction, which is Identity

  • Read in input (LS: in a string or boxes form, presumably depends on the cell type).

  • Apply $PreRead function, if defined, to the input string (LS: also for a box expression if an input is read as boxes. Note that if CellEvaluationFunction has been defined and still calls the kernel to evaluate the result, then $PreRead is applied to the result of CellEvaluationFunction execution).

  • Print syntax warnings if necessary.

  • Apply $SyntaxHandler function if there is a syntax error.

  • LS: Call MakeExpression to create an expression from boxes

  • LS: As a part of expression creation, call $NewSymbol on every new symbol to be created, then create that symbol. The choice of context where the symbol is created is based on the current (at the time of symbol's creation) values of $Context and $ContextPath. Obviously, cell and notebook contexts, when enabled, must be communicated to the kernel as a part of the cell evaluation process. That this is indeed the case, you can confirm by evaluating $Context variable in such a cell. How and at which stage this is done I don't know, perhaps someone will add this info.

  • Assign InString[n].

  • Apply $Pre function, if defined, to the input expression.

  • Assign In[n].

  • Evaluate expression which is a process with many steps as explained at here.

  • Apply $Post function, if defined.

  • Assign Out[n], stripping off any formatting wrappers.

  • Apply $PrePrint function, if defined.

  • Assign MessageList[n] and clear $MessageList.

  • Print expression, if it is not Null.

  • Increment $Line.

  • Clear any pending aborts.

  • Evaluate CellEpilog, if defined for a given cell.

Notation` package

The Notation` package hooks into MakeExpression. What this means is that the notation changes one can do with it are limited to what is possible to do on the level of fully parsed boxes, rather than strings. In other words, your input must represent a syntactically valid string of Mathematica code, for the purposes of parsing to boxes.

This means that there will be a number of things you can not do, such as change the precedence of operators, or introduce syntax which won't normally parse to boxes. Generally, this means that you can not assign custom string syntax, rather you can somewhat augment Mathematica's one, within what can be done on the box level.

Another limitation of Notation` is that, since it is based on MakeExpression, and the latter is not called by Get (which is called to read in packages), Notation` will not work for packages without some extra work-arounds. One such work-around I suggested here. While it is really a rather ugly hack, it illustrates the way Notation` works, to some extent. More information about the workings of Notation` can be obtained, of course, by reading the source code of Notation.m.

Further information

Some additional sources of information include

  • Reference pages for the functions or options such as CellProlog,CellEpilog, CellEvaluationFunction, etc.
  • A good brief (but dated) description is the 1992 WRI technical report "Mathematica internals" by David Withoff (available online as .pdf).
  • Some interesting investigations on the order of some of the operations here were carried out by Alexey Popkov in this question.

Remarks

I am almost certain that I missed or misrepresented something, simply because I don't have a lot of personal experience with all this steps, or exhaustive tests performed to test this description. If you found a mistake, feel free to edit.

Community
  • 1
Leonid Shifrin
  • 114,335
  • 15
  • 329
  • 420
  • 4
    The signature "Preamble" immediately reveals the author without the need to scroll down :) – István Zachar Oct 01 '12 at 17:23
  • Thank you for this elaborate answer, I've learned a lot from it (although there are still some parts of my question which are unanswered, like when and how the context is set). BTW, you can also completely change the behaviour using $PreRead; for example, using the raw kernel (I can't currently use the front end), the following will pass all further commands to the operating system: $PreRead = Function[str,"Import[\"!"~~str~~"\", \"Text\"]"]. Note that you shouldn't do that on a session you want to use further. – celtschk Oct 01 '12 at 18:43
  • @celtschk I added a note on contexts, which is not fully complete but may still make this a little more clear. I also added a sub-section on Notation` package, at least on my understanding of it. Re: $PreRead - yes, you are right, but it works for many cells already on the box level, not string level, while CellEvaluationFunction works on a string level. Good if this was helpful, and thanks for the accept. I planned this answer as a CW so that the community would enhance and correct it. – Leonid Shifrin Oct 01 '12 at 18:59
  • @IstvánZachar lol :-). You seem to be right, I am very predictable. – Leonid Shifrin Oct 01 '12 at 19:01
  • Thank you for your additions. I don't really understand your comment on $PreRead vs. CellEvaluationFunction; it seems to me that it says quite the opposite to what you write in the answer. As I understand the answer, CellEvaluationFunction works either on boxes or on strings, while $PreRead always works on strings, while your commens seems to say that CellEvaluationFunction always works on strings, while $PreRead works on box level for some cells. BTW, the most important difference probably is that CellEvaluationFunction is local to the cell, while $PreRead is global. – celtschk Oct 01 '12 at 19:19
  • @celtschk Yes, you are right for the CellEvaluationFunction - sorry for the confusion, it indeed works either on strings or on boxes. With $PreRead, however, the situation is rather strange (something I never understood). It ought to work on strings. However, typing in $PreRead = (Print[#]; #) & and then evaluating say a=1 reveals that it actually works on boxes, at least in this case. If you use the Program - style cell with Evaluatable ->True option, OTOH, then both $PreRead and CellEvaluationFunction work on strings. There seems to be little difference between them in ... – Leonid Shifrin Oct 01 '12 at 19:30
  • @celtschk ... this respect, although there must be differences, e.g. in the time they fire, or perhaps some other differences. I wouldn't mind if someone more knowledgable in this matter than me would add some more info here. – Leonid Shifrin Oct 01 '12 at 19:32
  • Thanks for the clarification. However, I'm not sure that "Print" by itself can distinguish boxes and strings — I think you can encode boxes inside strings. Did you try Print[InputForm[#]]? (I don't currently have access to a front end, so I can't test). – celtschk Oct 01 '12 at 21:08
  • OK, I've now checked, and it indeed receives boxes; however InString receives a string with encoded boxes. – celtschk Oct 02 '12 at 15:53
  • @celtschk Interesting ... the fy\unny thing is that I was just checking this myself and wanted to write a comment, at this very moment. Alas have no time for digging deeper into it now. – Leonid Shifrin Oct 02 '12 at 15:55
1

As far as the Notation package is concerned, I suspect, that it acts on the MakeExpression (and MakeBoxes for output) stages (not something lower-level).

An interesting Add-on to your question is: At which stages does MMA define/handel which of it's internal replacements? (trivial example: Spaces are treated as Times[]) Where can we look up, erase or change the definition of that? I suspect it is the MakeExpression stage, too.

NoEscape
  • 842
  • 4
  • 14