Language`* documentation project

Question

Language` context is around for a long time. It is also heavily used (APIFunctions / FormFunctions dependencies deployment rely on it).

It is undocumented and while I understand WRI won't support it just because we know how to use it, it will be very handy to have clear usage descriptions of:

? Language`*

Feel free to add/edit exsiting answers, they are based on our experience rather than internal knowledge so they may not be accurate.

Index:

• `*Definition*

  • `DefinitionList
  • `ExtendedDefinition
  • `ExtendedFullDefinition

• `*Mutation*

  • `SetMutationHandler
  • `HasMutationHandlerQ
  • `HandleMutation
  • `MutationFallthrough

• `*ExpressionStore*

  • `NewExpressionStore
  • `ExpressionStore

TODO:

{ `ArrayObject, `DestructureLValue, `FromArrayObject
, `FromVariableLengthInteger,  `FullGet, `GetLValueSymbol
, `InactivateExclusionsDefault, `LinkCombine, `NewArrayObject
, `PropertyItemMap, `TimeVariableLengthInteger
, `ToVariableLengthInteger, `UpdateInactivateExclusions
}

Answer 1

Language`*Mutation*

First seen here, implemented by Taliesin Beynon

What do they do?

The only functions I really know about in this bunch are the ?Language`*Mutation* functions.

This is an interface that allows you to build things that work like CloudExpression. For example, I can make a CloudExpression and use AppendTo on it:

ce = CreateCloudExpression[<||>]
(* ==> CloudExpression[...]*)
AppendTo[ce, 1 -> 2]
(* ==> CloudExpression[...] *)

This is really implemented at the UpValues level, but for many things the expression itself might be too deep to use via UpValues, e.g.:

ce[[1]] = 4
(* ==> 4*)

For this they use Language`SetMutationHandler to handle what would otherwise have been a failed assignment:

obj["uuid"][[1]] = 2
(* > Set::setps: obj[uuid] in the part assignment is not a symbol. *)
(* ==> 2*)

And there are a bunch of symbols these days that use this:

ToExpression[
  Names["*`*"]
, StandardForm
, Function[Null, If[Language`HasMutationHandlerQ[#], #, Nothing], HoldFirst]
]
({Audio`AudioGraph, AudioStream, CloudExpression, EntityStore, 
 InitializationValue, PersistentObject, PersistentValue})

Language`SetMutationHandler

This is the heart of the interface. We call it like:

Language`SetMutationHandler[type, handler]

which registers a mutation handler for type. CloudExpression for instance has the handler CloudExpression`PackageScope`CloudExpressionMutate.

Here's a concrete example of this in action:

oop`mut~SetAttributes~HoldAllComplete
oop`mut[Set[f_[oop`ob[s_], p___], v_]] := Set[f[s, p], v]
sym = {1, 2};
oop`ob[sym][[1]] = 2
sym
(Set::setps: oop`ob[sym] in the part assignment is not a symbol.)
(* ==> 2 )
( ==> {1, 2} *)
LanguageSetMutationHandler[oopob, oop`mut]
oop`ob[sym][[1]] = 2
sym
(* ==> 2 )
( ==> {2, 2} *)

Language`HasMutationHandlerQ

Pretty clear from the previous section

Language`HandleMutation

This is how the MutationHandler system does its dirty work. For example:

Language`HandleMutation[oop`ob[sym][[1]] = 2]
(* Set::noval: Symbol sym in part assignment does not have an immediate value. *)
(* ==> HoldComplete[1] *)

One thing to note, if the Handler doesn't do anything, the system will keep trying to mutate it until it bottoms out:

Language`SetMutationHandler[oop`ob, oop`mut2]
Language`HandleMutation[oop`ob[sym][[1]] = 2]
(* oop`ob::mutreclim: Too many nested mutations occurred. *)
(* oopob::modfl: The modification oopob[sym][[1]]=2 could not be performed. *)
(* ==> $Failed *)

Language`MutationFallthrough

Language`MutationFallthrough seems to tell the system to abort the mutation process and raise the most recent failed result, for instance this total failure:

oop`mut3~SetAttributes~HoldAllComplete
Language`SetMutationHandler[oop`ob, oop`mut3]
Language`HandleMutation[oop`ob[sym][[1]] = 2]
(* oopob::modfl: The modification oopob[sym][[1]]=2 could not be performed. *)
(* ==> $Failed *)

is transmogrified into this basic failure we'd expect without the mutation system when using Language`MutationFallthrough:

oop`mut3[___] := Language`MutationFallthrough
Language`HandleMutation[oop`ob[sym][[1]] = 2]
(* Set::setps: oop`ob[sym] in the part assignment is not a symbol. *)
(* ==> HoldComplete[2] *)

Answer 2

Language`*ExpressionStore

This is a hash-map system (without vectorized calls) that stores items without incrementing the ref count allowing them to be garbage collected naturally

Highly useful for various types of caching. First mentioned by Jason B here.

NewExpressionStore

Language`NewExpressionStore["name"]

Creates a new ExpressionStore

ExpressionStore

A cache that can be accessed via a slightly funky API:

put

Attaches a value to an expression and a key:

$store["put"[expr, key, value]]

get

Accesses a value attached to an expression by key

$store["get"[expr, key]]

remove

Removes the values attached to an expression:

$store["remove"[expr]]

listTable

Lists all cached values and their expressions:

$store["listTable"[]]

Notes

Hashing by explicit identity

This means that two copies of an expression, since they are different objects at the C++ level at which Mathematica is implemented, get hashed and thus stored differently:

Module[
 {
  store = Language`NewExpressionStore["random_store"],
  obj1, 
  obj2,
  sym
  },
 obj1 = sym[];
 obj2 = sym[];
 store["put"[obj1, "key", "value"]];
 {store["get"[obj1, "key"]], store["get"[obj2, "key"]]}
 ]

{"value", Null}

Ref counts preserved

The reference counter for an expression is not incremented by addition to a store. Thus if it is garbage collected it will be removed from the store

Module[
 {
  store = Language`NewExpressionStore["random_store"],
  obj,
  sym,
  res
  },
 obj = sym[];
 store["put"[obj, "key", "value"]];
 res = store["get"[obj, "key"]];
 obj =.; (* activate garbage collection *)
 {res, 
  store["get"[obj, "key"]]}
 ]

{"value", Null}

Edit

As provided by Jason B. here we also have "getKeys" and "containsQ". For those searching from the web, the lack of ref-count incrementing here makes this akin to a weak hash map.

Answer 3

Language`*Definition*

[...] Language`ExtendedDefinition and Language`ExtendedFullDefinition are
analogous to Definition and FullDefinition but capture the definition of a
symbol in such a way as it can be reproduced in another kernel. [...]

From groups.google mathematica thread

Language`ExtendedDefinition

Usage

• ExtendedDefinition[ sym ] generates _Language`DefinitionList all values associated with sym

• The result can be used to load/overwrite kernel state with those values: Language`ExtendedFullDefinition[] = Language`DefinitionList[...]

Options

See ExtendedFullDefinition...

Basic example

bar[x_] := x + 2;
foo[x_] := {x + 1, bar[x]};

temp = Language`ExtendedDefinition[foo]

(* Language`DefinitionList[
     HoldForm[foo] -> {
       OwnValues -> {}, SubValues -> {}, UpValues -> {}
     , DownValues -> {HoldPattern[foo[x_]] :> {x + 1, bar[x]}}
     , NValues -> {}, FormatValues -> {}, DefaultValues -> {}
     , Messages -> {}, Attributes -> {}
     }
   ]
 *)

 ClearAll[foo, bar];

 foo[1]
 (* foo[1] *)

 Language`ExtendedFullDefinition[] = temp;
 foo[1]
 (* {2, bar[1]} *)

Possible issues

• DefinitionList values for sym will not merge but overwrite existing state so any previous information about sym will be lost.

  ClearAll[foo, bar];
  
  foo[x_, y_] := x + y;
  
  foo[1]     (* ==> foo[1] *)
  foo[1, 2]  (* ==> 3 *)
  
  Language`ExtendedFullDefinition[] = temp;
  
  foo[1]     (* ==> {2, bar[1]} *)
  foo[1, 2]  (* ==> foo[1,2] *)
  

---

`ExtendedFullDefinition

Usage

Same as ExtendedDefinition but recursively gathers all relevant definitions

Options

ExtendedDefinition has them too but they are more relevant here.

• "ExcludedContexts" which context to ignore when gathering definitions

  Very useful, by default it includes all(?) built in paclets and System` which automatically fixes a problem of accidentally saving information about $MachineId and friends: Not FullDefinition for Save. MachineID not Protected?

  Language`ExtendedFullDefinition[$MachineID] 
    (*Language`DefinitionList[]*)
  
  Language`ExtendedFullDefinition[$MachineID, "ExcludedContexts" -> {}]
    (*Language`DefinitionList[ stuff...]*)
  

• "IssueReadProtectedMessages"

  By default False but when toggled it can help catch possible problems:

  SetAttributes[foo, ReadProtected];
  Language`ExtendedFullDefinition[foo, "IssueReadProtectedMessages" -> True]
  
  (* Language`ExtendedFullDefinition::rpsym :  Symbol foo is ReadProtected.*)
  (* Language`DefinitionList[] *)
  

Basic example

bar[x_] := x + 2;
foo[x_] := {x + 1, bar[x]}

temp = Language`ExtendedFullDefinition[foo]

(* Language`DefinitionList[
     HoldForm[foo] -> {
       OwnValues -> {}, SubValues -> {}, UpValues -> {}
     , DownValues -> {HoldPattern[foo[x_]] :> {x + 1, bar[x]}}
     , NValues -> {}, FormatValues -> {}, DefaultValues -> {}
     , Messages -> {}, Attributes -> {}
     }
   , HoldForm[bar] -> {
       OwnValues -> {}, SubValues -> {}, UpValues -> {}
     , DownValues -> {HoldPattern[bar[x_]] :> x + 2}
     , NValues -> {}, FormatValues -> {}, DefaultValues -> {}
     , Messages -> {}, Attributes -> {}
     }
   ]
*)

---

`DefinitionList

As far as I know it is just a symbolic wrapper returned and used by Extended*Definition functions.

---

Further reading:

• https://mathematica.stackexchange.com/search?q=ExtendedFullDefinition