6

Take a list of Associations. Using the Dataset function, this can be converted to a structured dataset.

associationList={
<|"a"->1,"b"->"x","c"->9|>,
<|"a"->2,"b"->"y","c"->8|>,
<|"a"->3,"b"->"z","c"->7|>}
dataset=Dataset[associationList]

enter image description here

Consider these three functions which can perform simple operations on each association.

function1=Function[{input},input[[{"c","b"}]]];
function2=Function[{input},Keys[input]];
function3=Function[{input},input[[Keys[input]]]];

When the first function is applied, it Takes elements based on hard-coded Keys.

dataset[All,function1]
Map[function1,associationList]

enter image description here

When the second function is applied, it programmatically gives all Keys.

dataset[All,function2]
Map[function2,associationList]

enter image description here

However, when the final function is applied, something peculiar occurs. For the list of Associations, it evaluates the Keys and uses these to Take elements, as I expected. For the Dataset, it gives an error.

dataset[All,function3]
Map[function3,associationList]

enter image description here

I initially assumed a Dataset was just a wrapper for a list of (nested) Associations. And that dataset[All, func] would loop over all internal Associations and apply func, similar to Map. This idea was too naive.

Can someone explain why the current behaviour makes sense?

Note that I am not looking for a work-around to apply function3 to my Dataset. I know I can obtain the desired answer in multiple ways. I am trying to understand the connection between Associations and Dataset.

LBogaardt
  • 1,595
  • 1
  • 11
  • 21
  • I have learned that Needs["Dataset"], Needs["TypeSystem"] and GetType[dataset] gives Struct[{"a", "b", "c"}, {Atom[Integer], Atom[String], Atom[Integer]}]. – LBogaardt Nov 06 '19 at 15:36
  • 2
    I think you are on the write track. Dataset isn't just a wrapper for Associations but rather includes information on the types of data contained within. It is interesting to note at the WTC this year, we were told that this extra information causes more problems than it solves and is being done away with. In fact, lots of major changes are coming to Dataset. FYI, you can see the structure in your example above by using InputForm[dataset]. – kickert Nov 06 '19 at 15:54

1 Answers1

7

This is a bug in the type deduction system of Dataset. In principle, your example should work as you expect. The problem is that when given a query, Dataset attempts to predict the type of the result before executing the query. Now in principle, this should happen silently in the background without you seeing anything. In your case, something like the following happens:

  • The query is All, Function[{input}, …] (note how function3 has been evaluated)
  • The All simply causes the inner function to be mapped, therefore the outer list stays a list (a TypeSystem`Vector[…, 3] to be precise)
  • Now it tries to figure out what Function[{input}, ...] returns when applied to a TypeSystem`Struct[{"a", "b", "c"}, …].
    • Dataset "knows" how to handle pure functions (i.e. Function[…]), so it tries to deduce the resulting data type.
    • It does so by first inserting the type (i.e. TypeSystem`Struct[…]) into the body of the function.
    • At this point, it tries to deduce the resulting type of TypeSystem`Struct[…][[Keys[TypeSystem`Struct[…]]]]
    • The deduction of expressions of the form …[[…]] with symbolic indices is unfortunately broken, resulting in the error message you are seeing. (See here for more information)

The are numerous work-arounds for this (see for example the post linked above for some), you can for example hide the pure function behind a symbol with down-values (the type system does not attempt to deduce the type in this case):

function3b[arg_] := function3@arg

dataset[All, function3b]
Map[function3, associationList]

enter image description here

Lukas Lang
  • 33,963
  • 1
  • 51
  • 97