13

Part, AppendTo, PrependTo, AddTo, etc. allow in-place modification of a list, but only Part requires that the list be referenced through a simple symbol, e.g. the following all does what you'ld expect:

foo = {1, 2, 3, 4};
AppendTo[foo, 5]

{1, 2, 3, 4, 5}

foo[[2]] = 100;
foo

{1, 100, 3, 4, 5}

But if the list reference is something more complex, then Part fails:

bar[1] = {4, 3, 2, 1};
AppendTo[bar[1], 5]

{4, 3, 2, 1, 5}

bar[1][[2]] = 100;

Set::setps: bar[1] in the part assignment is not a symbol. >>

I know there have been work-arounds posted, but does anyone know the rationale for why Part precludes this behavior?

Community
  • 1
Eric Parker
  • 175
  • 4

1 Answers1

15

The reason is that AppendTo and PrependTo actually replace the whole list with a new one, rather than only changing a single element. In that sense, they are not really "in-place". They are in-place in the sense that the result gets assigned back to the same variable.

You can see using Trace, that AppendTo and PrependTo in fact expand:

bar[1]={4,3,2,1};
Trace[AppendTo[bar[1],5]]

(* 
   {AppendTo[bar[1],5],{bar[1],{4,3,2,1}},bar[1]=Append[{4,3,2,1},5],
    {Append[{4,3,2,1},5],{4,3,2,1,5}},bar[1]={4,3,2,1,5},{4,3,2,1,5}} 
*)

and are mostly syntactic sugar on top of Set and Append / Prepend.

With Part, the situation is different, when we try to assign to parts of an expression in-place. In such a case, Part is a lower-level command (compared to most others), and only supports such operations when the expression being modified in-place is stored in a Symbol (rather than an indexed variable, as in your example, or a more general expression). This limitation must have to do with expressions being based on arrays internally, and also with efficiency (because it makes massive part assignments quite efficient). I have discussed this issue in a little more detail here.

To put it slightly differently, AppendTo and PrependTo have the same O(n) complexity as Append and Prepend, so them being in-place is a syntactic convenience, but doesn't require any new non-trivial (or lower-level) behavior from the language - and thus, in particular, can be implemented in the top-level. Part[s, pos] = newelem is an operation that we expect to have O(1) complexity, however. The top-level can only provide O(n) by doing something along the lines of bar[1] = ReplacePart[bar[1], 2 -> 100], but this is unacceptable performance-wise.

So, the language needs to do some lower-level work to make the complexity right, but then it is natural that, for a quite general language based on immutable expressions, there will be limitations like that (only Symbols allowed). If you allow other expressions (like indexed variables) to be L-values for part assignments, you'll inevitably have to allow top-level evaluation during those assignments. That, and the absence of true pass-by-reference semantics, would make both implementation much more complex, and performance in general much harder to predict. My guess is that these are some of the reasons why those limitations exist.

Community
  • 1
Leonid Shifrin
  • 114,335
  • 15
  • 329
  • 420
  • Wow! I read through the very detailed discussion you referenced, and Part makes a lot more sense now. Mma must use some sort of copy-on-modify and maybe that has something to do with the limitations on Part. Anyway, interesting to think about. – Eric Parker Jan 27 '15 at 19:50
  • @EricParker Yes, it does use copy-on-modify. – Leonid Shifrin Jan 27 '15 at 19:51
  • 1
    @EricParker In fact, here is a reference to a more detailed discussion of this. Thanks for the accept, by the way - but generally, it is a good practice to wait for some time, to encourage more answers to appear. – Leonid Shifrin Jan 27 '15 at 20:01
  • Actually it's a good practice to accept Leonid's answers immediately. That way the rest of us don't get tempted into thinking we can do better. Delusions of adequacy and all that... – Daniel Lichtblau Jan 27 '15 at 22:07
  • (I'm not really trying to discourage other responses. Often there are multiple really good answers that are complementary rather than redundant.) – Daniel Lichtblau Jan 27 '15 at 22:07
  • @EricParker: The $O(n)$ performance aspect of Append is somewhat important to know about, since often beginners who are used to other languages will use it to iteratively grow lists element-by-element, and are often surprised by the horrible $O(n^2)$ performance of the list-building process in Mathematica. For such list-building applications, Reap and Sow provide a better-performing $O(n)$ alternative comparable to what you'd expect from other languages. – DumpsterDoofus Jan 27 '15 at 23:24
  • @DanielLichtblau Well, coming from you, Daniel, this is the best thing I could hear. But, this would raise the bar of expectations for what to see in my answers way to high :). – Leonid Shifrin Jan 27 '15 at 23:34
  • @DumpsterDoofus I was already aware of the $n^2$ behavior of AppendTo. I generally accumulate into a list by using an undefined symbol (e.g. item): list = {list, item[...]}; then at the end flatten and replace item with List: list = Flatten[list] /. item -> List. The flatten operation is very fast. – Eric Parker Jan 28 '15 at 20:49
  • @EricParker What you describe is a special case of what is usually called linked lists in Mathematica. Using them is indeed another good method of accumulation of intermediate results, which was the only one before the introduction of Reap and Sow into the language. I have posted a rather detailed post devoted to them, here. – Leonid Shifrin Jan 28 '15 at 20:55