Reshaping associations, generalization of GroupBy

Question

Let's say we have a set a\of associations:

dataset = {
  <|"type" -> "a", "subtype" -> "I", "value" -> 1|>,
  <|"type" -> "a", "subtype" -> "II", "value" -> 2|>,
  <|"type" -> "b", "subtype" -> "I", "value" -> 1|>,
  <|"type" -> "b", "subtype" -> "II", "value" -> 2|>
  }

where every entry is unique in terms of {#type, #subtype},

I'd like to build a nested association for more handy querying, e.g. I would like to have:

nested["a", "II", "value"]

2

I can start with

GroupBy[dataset, {#type &, #subtype &}]

<|
   "a" -> <|
     "I" -> {<|"type" -> "a", "subtype" -> "I", "value" -> 1|>},
     "II" -> {<|"type" -> "a", "subtype" -> "II", "value" -> 2|>}|>, 
   "b" -> <|
     "I" -> {<|"type" -> "b", "subtype" -> "I", "value" -> 3|>}, 
     "II" -> {<|"type" -> "b", "subtype" -> "II", "value" -> 4|>}
|>|>

But nested["a", "I"] points to a list with one association, what is expected but I would like to drop that list.

It seems that the third argument of GroupBy isn't generalized to handle nested grouping...

So basically I would like to have ... "I" -> <|"type" -> "a", ....

What is a generic way to go?

I can do:

• nested GroupBy:

  GroupBy[dataset, #type &, GroupBy[#, #subtype &, First] &]
  

• Map later:

  GroupBy[dataset, {#type &, #subtype &}] // Map[First, #, {-3}] &
  

But the first is not handy in general while the second is ugly (and not general either).

---

Acceptable outputs are:

<|
       "a" -> <|
         "I" -> <|"type" -> "a", "subtype" -> "I", "value" -> 1|>,
...
|>

or

<|
       "a" -> <|
         "I" -> <|"value" -> 1|>,

...
|>

or

<|
       "a" -> <|
         "I" -> 1 ,

...|>

but the first is the most desired one because we may have more that one ("value") key left.

Answer 1

One approach is to employ a helper function that unwraps singleton lists:

{delist[v_]} ^:= v

With this, the GroupBy expression is fairly succinct:

dataset // GroupBy[{#type&, #subtype& -> delist}]

(*
  <| "a" -> <| "I" -> <|"type" -> "a", "subtype" -> "I", "value" -> 1|>
             , "II" -> <|"type" -> "a", "subtype" -> "II", "value" -> 2|>
             |>
   , "b" -> <| "I" -> <|"type" -> "b", "subtype" -> "I", "value" -> 1|>
             , "II" -> <|"type" -> "b", "subtype" -> "II", "value" -> 2|>
             |>
   |>
*)

This generalizes to deeper nesting:

dataset // GroupBy[{#type&, #subtype&, #type&, #subtype& -> delist}]

(*
  <| "a" ->
      <| "I" -> <|"a" -> <|"I" -> <|"type" -> "a", "subtype" -> "I", "value" -> 1|>|>|>
       , "II" -> <|"a" -> <|"II" -> <|"type" -> "a", "subtype" -> "II", "value" -> 2|>|>|>
       |>
   , "b" ->
       <| "I" -> <|"b" -> <|"I" -> <|"type" -> "b", "subtype" -> "I", "value" -> 1|>|>|>
        , "II" -> <|"b" -> <|"II" -> <|"type" -> "b", "subtype" -> "II", "value" -> 2|>|>|>
        |>
   |>
*)

Answer 2

Instead of a nested association solution, would Query and Select be acceptable.

Query[Select[#type == "a" && #subtype == "I" &], "value"]@dataset

(* {1} *)

This form is more descriptive on what is happening and does not require reshaping of the list of associations.

If your data is such that there is only ever one item intersecting a particular "type" and "subtype" then tack on First.

First@Query[Select[#type == "a" && #subtype == "I" &], "value"]@
  dataset

(* 1 *)

Hope this helps.

---

Extension

You can extend this to a more general case in which you parametrise the filter by both key and value.

filterBy[filter_] := Function[Evaluate[And @@ ReleaseHold[Hold[Slot][First@#] == Last@# & /@ filter]]]

then with

target = {{"type", "a"}, {"subtype", "I"}};

Query[SelectFirst[filterBy[target]], "value"]@dataset

(* 1 *)

Answer 3

I have posted code doing a very similar thing here - the functions pushUp and pushUpNested. That code was more general, since there I provided a declarative interface to group by values or their parts. To do what you need, I'll redefine slightly (assuming you run that code):

ClearAll[pushUpNested];
pushUpNested[{}, elemF_: Identity] := elemF;
pushUpNested[specs : {_List ..}, elemF_: Identity ] := 
   Composition[
     Map[pushUpNested[Rest[specs], elemF]], 
     pushUp@First[specs]
   ];

Now we create a transform:

transform = pushUpNested[{{"type"}, {"subtype"}}, First]


(* 
   Map[Map[First]@*GroupBy[#1[[Sequence[Key["subtype"]]]] &]]@*
   GroupBy[#1[[Sequence[Key["type"]]]] &]
*)

which we can now apply to get the nested structure:

nested = transform@dataset

(*

   <|
     "a" -> <|
       "I" -> <|"type" -> "a", "subtype" -> "I", "value" -> 1|>, 
       "II" -> <|"type" -> "a", "subtype" -> "II", "value" -> 2|>
     |>, 
     "b" -> <|
       "I" -> <|"type" -> "b", "subtype" -> "I", "value" -> 1|>, 
       "II" -> <|"type" -> "b", "subtype" -> "II", "value" -> 2|>
     |>
   |>

*)

The advantage of using pushUpNested is that it makes it very easy and declarative to construct such transforms, and the transform is available for inspection as a stand-alone fully-prepared function.

Answer 4

I believed this question to be a duplicate of Create Nested List from tabular data and proposed, with minor variation, the same answer:

fn[x_List] := GroupBy[x, First -> Rest, fn]
fn[{a_}] := a

nested = fn[dataset]

<|"a" -> <|"I" -> <|"value" -> 1|>, "II" -> <|"value" -> 2|>|>, 
 "b" -> <|"I" -> <|"value" -> 1|>, "II" -> <|"value" -> 2|>|>|>

nested["a", "II", "value"]

2

WReach suggested a different reading of this question however. To that end I propose:

ClearAll[fn]

fn[p_, r___][x_List] := GroupBy[x, Lookup[p] -> KeyDrop[p], fn[r]]
fn[][{x_}] := x

nested = dataset // fn["type", "subtype"]

<|"a" -> <|"I" -> <|"value" -> 1|>, "II" -> <|"value" -> 2|>|>, 
  "b" -> <|"I" -> <|"value" -> 1|>, "II" -> <|"value" -> 2|>|>|>

nested["a", "II", "value"]

2

Answer 5

Here is the approach I would take to transform your dataset to a nested Association:

Clear[ makeNested ];
makeNested[ assoc_, keylist_] := GroupBy[
   assoc,
   keylist
 ] // Apply[
    Association,
    #,
    { Length @ keylist }
  ] &

Now makeNested[ dataset, { #type &, #subtype &} ] and

makeNested[ dataset, { #type &, #subtype &, #type &, #subtype & } ] will work as wanted.

For example:

(nested = makeNested[ dataset, { #type &, #subtype & } ]) // Dataset

nested["a","I","value"]

1

Update: A more streamlined solution

Inspired by WReach's solution I tried to streamline this into a compact function that also get's rid of the keys that have already been used in the nesting. So here it is:

Clear[ formatLeaves ];
Options[ formatLeaves ] = {
   "DropKeys" -> None
};
formatLeaves/: List[ formatLeaves[ data_ ] ] := Module[
   {
      keys = OptionValue[ formatLeaves, "DropKeys" ]
   },
   KeyDrop[ data, keys ]
]

Clear[ makeNested ];
makeNested[ data_ , keylist_?(VectorQ[ #, StringQ ]&) ] := Module[
   {
      listKeys = Map[Key] @ keylist,
      lastKey
   },
   lastKey = Last @ listKeys;
   SetOptions[ formatLeaves, "DropKeys" -> listKeys ];
   GroupBy[
      data,
      ReplaceAll[
         listKeys,
         Rule[
            lastKey,
            lastKey -> formatLeaves
         ]
      ]
   ]
]

Now:

(nested = makeNested[ dataset, { "type", "subtype" } ]) // Dataset

nested[ "a","I","value" ]

1

Naturally this saves space in memory and on disk:

Map[ByteCount] @ { nested, dataset }

{2064, 2280}

Answer 6

If you need to retain the original keys, I'd be inclined to follow Edmund's answer; alternatively, if you are happy enough to throw away the keys (also given that "every entry is unique") one flexible approach follows a nice solution of your own.

 RecurAssocMerge[a : {__Association}] := Merge[a, RecurAssocMerge];
 RecurAssocMerge[a_] := Last[a];
 RecurAssocMerge[ini_Association, path_List, value_] := 
 RecurAssocMerge[{ini, Fold[<|#2 -> #|> &, value, Reverse@path]}];
 RecurAssocMerge[ini_Association, fullPath_List] := RecurAssocMerge[ini, Most@fullPath, Last@fullPath]; (* added from previous post to handle single list specification *)

Now build up incrementally after extracting values

Query[All, Values]@dataset // Fold[RecurAssocMerge, <||>, #] &

(* <|"a" -> <|"I" -> 1, "II" -> 2|>, "b" -> <|"I" -> 1, "II" -> 2|>|> *)