Alternative to Databin

Question

Project

I'm continuing my work building a reporting system for my students' Khan Academy data. See post about OAuth here: Oauth for Khan Academy API

I've built a data structure that I intend to update with new data I'll fetch from the Khan Academy API

Working data structure with sample student data (I typically render as a Dataset to view):

<|"studentKeys" -> 
<|"ag5zfmtoYW4tY12345" -> <|"2017-08-04T17:11:38.000Z" -> <|"nickname" -> "Student, First", 
       "marks" -> <|"practiced" -> 0, "mastery1" -> 0, 
         "mastery2" -> 4, "mastery3" -> 152|>, "timeSpent" -> 456, 
       "percentageComplete" -> 77, "missionName" -> "algebra"|>|>, 
   "ag5zfmtoYW4tY12346" -> <|"2017-08-04T17:11:41.000Z" -> <|"nickname" -> "Student1, First1", 
       "marks" -> <|"practiced" -> 0, "mastery1" -> 0, 
         "mastery2" -> 0, "mastery3" -> 88|>, "timeSpent" -> 0, 
       "percentageComplete" -> 44, "missionName" -> "algebra"|>|>, 
   "ag5zfmtoYW4tY12347" -> <|"2017-08-04T17:11:42.000Z" -> <|"nickname" -> 
        "Student2,First2", 
       "marks" -> <|"practiced" -> 4, "mastery1" -> 2, 
         "mastery2" -> 0, "mastery3" -> 162|>, "timeSpent" -> 335, 
       "percentageComplete" -> 82, "missionName" -> "algebra"|>|>, 
   "ag5zfmtoYW4tY12348"-> <|"2017-08-04T17:11:44.000Z" -> <|"nickname" -> "Student3, First3", 
       "marks" -> <|"practiced" -> 0, "mastery1" -> 3, 
         "mastery2" -> 1, "mastery3" -> 142|>, "timeSpent" -> 1168, 
       "percentageComplete" -> 72, "missionName" -> "algebra"|>|>|>|>

Problem

I want to keep the data in the Wolfram Cloud and be able to update it. Is it possible to use a CloudObject? I'd really like to avoid having to deploy a relational database somewhere outside the Wolfram Cloud. Thanks.

Stuff I've tried

Databins worked, but they've got a 25kb limit so they won't scale.

Here's what I've tried:

demoDb = <|"studentKeys" -> \
<|"ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9\
pZC5raGFuYWNhZGVteS5vcmcvYzRkZjBiODc4NjYzNDM0ZmJmODYyYmY4ZDcwYjE1MDIM"\
 -> <|"2017-08-04T20:14:23.000Z" -> <|"nickname" -> "first, student", 
         "marks" -> <|"practiced" -> 0, "mastery1" -> 0, 
           "mastery2" -> 4, "mastery3" -> 152|>, "timeSpent" -> 456, 
         "percentageComplete" -> 77, "missionName" -> "algebra"|>|>, 
     "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6\
Ly9pZC5raGFuYWNhZGVteS5vcmcvYjM2MTViMjVjMTA4NDA1OWI0ZTY2Zjk2MzMwZWJhZD\
kM" -> <|"2017-08-04T20:14:24.000Z" -> <|"nickname" -> 
          "second, student", 
         "marks" -> <|"practiced" -> 0, "mastery1" -> 0, 
           "mastery2" -> 0, "mastery3" -> 88|>, "timeSpent" -> 0, 
         "percentageComplete" -> 44, "missionName" -> "algebra"|>|>, 
     "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6\
Ly9pZC5raGFuYWNhZGVteS5vcmcvYzdmY2U5NjY5OWFkNGQzODhiYTk0YmQwN2EwNDY4ZW\
MM" -> <|"2017-08-04T20:14:25.000Z" -> <|"nickname" -> 
          "third, student", 
         "marks" -> <|"practiced" -> 4, "mastery1" -> 2, 
           "mastery2" -> 0, "mastery3" -> 162|>, "timeSpent" -> 335, 
         "percentageComplete" -> 82, "missionName" -> "algebra"|>|>, 
     "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6\
Ly9pZC5raGFuYWNhZGVteS5vcmcvY2NjYjFhN2M3NTdjNGEzOGIwMDA3YTkyOWZiOWM2Ym\
YM" -> <|"2017-08-04T20:14:27.000Z" -> <|"nickname" -> 
          "fourth ,student", 
         "marks" -> <|"practiced" -> 0, "mastery1" -> 3, 
           "mastery2" -> 1, "mastery3" -> 142|>, "timeSpent" -> 1168, 
         "percentageComplete" -> 72, 
         "missionName" -> "algebra"|>|>|>|>;

(* AppendTo data in demoDb.  Works fine here *)

AppendTo[demoDb["studentKeys"][
   "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly\
9pZC5raGFuYWNhZGVteS5vcmcvYzRkZjBiODc4NjYzNDM0ZmJmODYyYmY4ZDcwYjE1MDIM\
"],
  <|"2017-08-04T20:18:23.000Z" -> <|"nickname" -> "first, student", 
     "marks" -> <|"practiced" -> 0, "mastery1" -> 0, "mastery2" -> 4, 
       "mastery3" -> 152|>, "timeSpent" -> 456, 
     "percentageComplete" -> 77, "missionName" -> "algebra"|>|>];


ce = CreateCloudExpression[demoDb];

(* AppendTo date in ce.  Throws error *)

AppendTo[ce["studentKeys"][
   "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly\
9pZC5raGFuYWNhZGVteS5vcmcvYzRkZjBiODc4NjYzNDM0ZmJmODYyYmY4ZDcwYjE1MDIM\
"],
  <|"2017-08-04T22:20:23.000Z" -> <|"nickname" -> "first, student", 
     "marks" -> <|"practiced" -> 0, "mastery1" -> 0, "mastery2" -> 4, 
       "mastery3" -> 152|>, "timeSpent" -> 456, 
     "percentageComplete" -> 77, "missionName" -> "algebra"|>|>];

Here is the error it throws:

Association::setps: <|ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9pZC5raGFuYWNhZGVteS5vcmcvYzRkZjBiODc4NjYzNDM0ZmJmODYyYmY4ZDcwYjE1MDIM-><|2017-08-04T20:14:23.000Z-><|nickname->first, student,marks-><|practiced->0,mastery1->0,mastery2->4,mastery3->152|>,timeSpent->456,percentageComplete->77,missionName->algebra|>,2017-08-04T20:18:23.000Z-><|nickname->first, student,marks-><|practiced->0,mastery1->0,mastery2->4,mastery3->152|>,timeSpent->456,percentageComplete->77,missionName->algebra|>|>,<<1>>,<<1>>,ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9pZC5raGFuYWNhZGVteS5vcmcvY2NjYjFhN2M3NTdjNGEzOGIwMDA3YTkyOWZiOWM2YmYM-><|2017-08-04T20:14:27.000Z-><|nickname->fourth ,student,<<3>>,missionName->algebra|>|>|> in the part assignment is not a symbol.

My suggestion is a CloudExpression. It's easily mutable by part, so you can easily add and update student data. — b3m2a1, Aug 05 '17 at 01:01
I messed with those @b3m2a1 but I struggled to get the AppendTo function to work correctly. Since it is mutable, I'm assuming I can completely overwrite old data with new data. Is that right? — Matt Green, Aug 05 '17 at 02:20
Definitely possible, but if you're gonna fully update it a CloudObject makes more sense. What problem did you have with AppendTo? — b3m2a1, Aug 05 '17 at 02:21
I tried to AppendTo on my CloudExpression that had the above format and it threw an error. Can't remember which one. The same AppendTo process worked find on my local data. I can post code in an hour or so. @b3m2a1 — Matt Green, Aug 05 '17 at 02:24
@b3m2a1 I've added what happened with CloudExpression and AppendTo... not sure what the error means... — Matt Green, Aug 05 '17 at 03:37
that's clearly a mutation corner case they haven't handled. Try using this assignment form instead: AppendTo[demoDb["studentKeys", key], data] — b3m2a1, Aug 05 '17 at 03:38

score 2 · Answer 1 · answered Aug 05 '17 at 04:33

Thank you again @b3m2a1 for the help.

The following code runs perfectly. The key line was:

Map[AssociateTo[ce["studentKeys", key], #] &, Normal@data];

Here's the whole thing:

ClearAll[demoDb]
demoDb = <|"studentKeys" -> \
<|"ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6Ly9\
pZC5raGFuYWNhZGVteS5vcmcvYzRkZjBiODc4NjYzNDM0ZmJmODYyYmY4ZDcwYjE1MDIM"\
 -> <|"2017-08-04T20:14:23.000Z" -> <|"nickname" -> "first, student", 
         "marks" -> <|"practiced" -> 0, "mastery1" -> 0, 
           "mastery2" -> 4, "mastery3" -> 152|>, "timeSpent" -> 456, 
         "percentageComplete" -> 77, "missionName" -> "algebra"|>|>, 
     "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6\
Ly9pZC5raGFuYWNhZGVteS5vcmcvYjM2MTViMjVjMTA4NDA1OWI0ZTY2Zjk2MzMwZWJhZD\
kM" -> <|"2017-08-04T20:14:24.000Z" -> <|"nickname" -> 
          "second, student", 
         "marks" -> <|"practiced" -> 0, "mastery1" -> 0, 
           "mastery2" -> 0, "mastery3" -> 88|>, "timeSpent" -> 0, 
         "percentageComplete" -> 44, "missionName" -> "algebra"|>|>, 
     "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6\
Ly9pZC5raGFuYWNhZGVteS5vcmcvYzdmY2U5NjY5OWFkNGQzODhiYTk0YmQwN2EwNDY4ZW\
MM" -> <|"2017-08-04T20:14:25.000Z" -> <|"nickname" -> 
          "third, student", 
         "marks" -> <|"practiced" -> 4, "mastery1" -> 2, 
           "mastery2" -> 0, "mastery3" -> 162|>, "timeSpent" -> 335, 
         "percentageComplete" -> 82, "missionName" -> "algebra"|>|>, 
     "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6\
Ly9pZC5raGFuYWNhZGVteS5vcmcvY2NjYjFhN2M3NTdjNGEzOGIwMDA3YTkyOWZiOWM2Ym\
YM" -> <|"2017-08-04T20:14:27.000Z" -> <|"nickname" -> 
          "fourth ,student", 
         "marks" -> <|"practiced" -> 0, "mastery1" -> 3, 
           "mastery2" -> 1, "mastery3" -> 142|>, "timeSpent" -> 1168, 
         "percentageComplete" -> 72, 
         "missionName" -> "algebra"|>|>|>|>;


ce = CreateCloudExpression[demoDb]


data = <|"2017-08-04T22:24:24.000Z" -> <|"nickname" -> 
      "first, student", 
     "marks" -> <|"practiced" -> 0, "mastery1" -> 0, "mastery2" -> 4, 
       "mastery3" -> 152|>, "timeSpent" -> 456, 
     "percentageComplete" -> 77, "missionName" -> "algebra"|>|>;

Map[AssociateTo[
    ce["studentKeys", 
     "ag5zfmtoYW4tYWNhZGVteXJUCxIIVXNlckRhdGEiRnVzZXJfaWRfa2V5X2h0dHA6\
Ly9pZC5raGFuYWNhZGVteS5vcmcvYzRkZjBiODc4NjYzNDM0ZmJmODYyYmY4ZDcwYjE1MD\
IM"], #] &, Normal@data];

Alternative to Databin

Project

Problem

Stuff I've tried

1 Answers1

Linked

Related