Dataset: get number of levels and elements

Question

The summary box for Dataset objects shows the number of levels and elements at the bottom.

How can we get these values programmatically?

For example, "4 levels, 350 elements" in the Planets dataset:

Answer 1

If we are not afraid of using undocumented functions, we can get the counts the same way that the summary box gets them:

info @ Dataset[data_, type_, _] :=
  {TypeSystem`TypeDepth[type],  TypeSystem`AtomCount[type, data]}

info @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)

Or, as per @Szabolcs' suggestion:

info2 @ ds_Dataset :=
  {TypeSystem`TypeDepth[#],  TypeSystem`AtomCount[##]}& @@
    Through[{Dataset`GetType, Dataset`GetData} @ ds]

info2 @ ExampleData[{"Dataset", "Planets"}]
(* {4, 350} *)

Answer 2

This is not quite the same thing, but strongly related and very useful to know:

It appears that the undocumented option AllowedHeads, which appears in several functions in v10, can help here.

ds = ExampleData[{"Dataset", "Planets"}];

To get the depth,

ArrayDepth[
 Normal[ds],
 AllowedHeads -> {Association, List}
]

This also works:

ArrayDepth[
 Normal[ds],
 AllowedHeads -> All
]

To get the element count from this generalized array,

Times @@ Dimensions[
  Normal[ds],
  AllowedHeads -> All
]

These counts refer only to the rectangular part of the generalized array made of lists and associations. They won't give the counts 4 and 350. For that we'd need to descend deeper.

---

Out of curiosity, this is the complete list of System` functions having this option:

ArrayDepth, ConjugateTranspose, Depth, Dimensions, Transpose

Transpose and ConjugateTranspose are useful for taking the transpose of mixed List-Association structures.

Depth would theoretically be good for finding the number of levels in the dataset, but I don't know how to use this option with it. It seems to take only the values True and False.

Answer 3

There's at least the brute force approach

getFromDs[what_String][ds_Dataset] := 
 FirstCase[ToBoxes@ds, 
  RowBox[{___, i_, Shortest@___, what, ___}] :> 
   With[{res = ToExpression@i}, res /; IntegerQ[res]], $Failed, 
  Infinity]

getLevels = getFromDs["levels"]; getElements = getFromDs["elements"];

Through@{getLevels, getElements}@ExampleData[{"Dataset", "Planets"}]

(* {4, 350} *)

This is an attempt at a disgusting function for the depth: the maximum number of lists or associations you can go through from the top down before bouncing in something else

We setup some tests

test[dep] ^= Inactivate@{
    dep[{1}] -> 1,
    dep[4] -> 0,
    dep[h[4, g[8]]] -> 0,
    dep[<|"a" -> 3|>] -> 1,
    dep[<|2|>] -> 0,
    dep[<|"a" -> 2|>] -> 1,
    dep[{<|"a" -> 2|>, d}] -> 2,
    dep[<|{2}|>] -> 0,
    dep[<|"a" -> {2}, "b" -> 8|>] -> 2,
    dep[<|"a" :> {<|"c" -> h[2]|>}|>] -> 3,
    dep[<|"x" -> {1}, "x" -> 3|>] -> 1,
    dep[<|"x" :> <|"x" -> {1}, "x" -> 3|>|>] -> 2
    };

runTest[sym_] := 
 Inactive[TestReport][
   test[sym] /. (x_ -> y_) :> Inactive[VerificationTest][x, y]] // 
  Activate

The code

dep[ds_Dataset] := dep[Normal@ds];
dep[stuff_] := idep[0, stuff];

SetAttributes[idep, HoldAllComplete];
idep[i_, l_List | 
    l_Association?(Function[i, AssociationQ@Unevaluated@i, 
       HoldAllComplete])] := 
  1 + Max[Function[x, idep[i, x], HoldAllComplete] /@ 
      Unevaluated@l /. {a_Association :> Values@a}];
idep[i_, a_Association /; 
    MatchQ[Unevaluated@a, 
     HoldPattern[
      Association[(_Rule | _RuleDelayed) ... | {(_Rule | \
_RuleDelayed) ...}]]]] :=

  Module[{v = Block[{Association = HoldComplete}, a]},
   With[{rhs = v[[All, 2]]}, 
    1 + Max[Function[x, idep[i, x], HoldAllComplete] /@ 
        Unevaluated@rhs // ReleaseHold]]];
idep[i_, _] := i;

When we run the tests we see that the last one failed runTest[dep]["TestsFailed"]. That was expected. I don't know exactly how it makes sense to design this. The fact that associations don't become associations until they are evaluated, plus the fact that one has the possibility of holding any of its arguments through using :> instead of -> makes it confusing. Should one count as valid associations that aren't yet associations but are lexically well written? This does that, but this assumes that all valid unevaluated associations are Associations with a list or sequence of rules inside, and doesn't check if those rules may have repeated keys which might end up deleting an element, and you can't really check without evaluating them and leaking.

Answer 4

Here is another way of calculating the number of levels and elements. It first extracts the Values of each Association and then uses the resulting List to calculate the numbers. It would be nice to know if there is a simpler way of calculating the number of levels from the List.

levelsAndElements[data_Dataset] := Module[{lists, elements, levels},
  lists = Normal@data //. assoc_Association :> Values[assoc];
  elements = Length@Flatten[lists];
  levels = Depth[DeleteCases[lists, Except[_List], Infinity]] - 1;
  {levels, elements}]

levelsAndElements[ExampleData[{"Dataset", "Planets"}]]

{4, 350}