13

AppendTo is not efficient so I'm not quite sure of the best way to implement its analogue for joining (e.g. in python this would be extend function). So how can this be improved:

ClearAll[JoinTo];
SetAttributes[JoinTo,HoldFirst];
JoinTo[a_, b_List] := (a = Join[a,b];)
M.R.
  • 31,425
  • 8
  • 90
  • 281
  • 1
    need to account for cases where e.g. a is an integer or has some other head – Mike Honeychurch Jan 02 '16 at 23:36
  • 7
    This will also make a copy of whatever it is replacing, hence will be inefficient if used in a loop. You might consider nesting List, as in a={a,b} or maybe use of Reap[...Sow[a]...], depending on what is the application at hand. – Daniel Lichtblau Jan 02 '16 at 23:56
  • Shouldn't using an association be efficient (at least if beforehand the size of Join[a,b] were known so that values had just to be updated instead of inserted)? AssociateTo at least claims to offer efficient in place modification. – Sascha Jan 03 '16 at 00:01
  • @Sascha Maybe...Depends on how many AssociateTo operations are done, and there may be an ordering issue (I think new things get added at the back and not the front). I'm not sure what algorithmic complexity to expect for adding new items; might be O(log n) for n=length of the association (which of course is much better than O(n)). Best I can suggest is that this approach be tried and timed. – Daniel Lichtblau Jan 03 '16 at 16:32
  • @Daniel Lichtblau, I probably should have written my comment more clearly to convey that it contains two separate ideas: (1) updating values of associations instead of adding key-value pairs might be efficient (one could create an association with "empty" key-value pairs and update values as necessary; call it pre allocation if you will). (2) Just try out the build-in function AssociateTo to add key-value pairs and compare with list operations. – Sascha Jan 03 '16 at 16:42
  • Consider using a linked list and then flattening. You may either do this explicitly at the top level, or else use Internal\Bagand friends. The latter are employed bySowandReap`. By the way, it would be helpful if you could state clearly in what respects you would like your idea to be improved upon. – Oleksandr R. Feb 02 '16 at 23:14
  • @OleksandrR. Does Mathematica have linked lists? – M.R. Feb 02 '16 at 23:18
  • @M.R. Yes, of course. Well, it has lists, and you can link them, so that's sufficient. See for example (25474) – Oleksandr R. Feb 02 '16 at 23:46
  • Oh right, thought they might have added as a first class symbol – M.R. Feb 03 '16 at 00:25

3 Answers3

4

We can also use ApplyTo (new in 12.2) together with Splice (new in 12.1):

a = Range[5];
a //= Append[Splice @ Range[6, 10]];
a

{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

It is fast:

a = Range[10^6];
a //= Append[Splice @ Range[10^6]]; // AbsoluteTiming // First

0.06516

and can be packed into a function:

Attributes[joinTo] = HoldFirst;
joinTo[a_, b_] := a //= Append[Splice @ b]

a = Range[10^6];
joinTo[a, Range[10^6]]; // AbsoluteTiming // First

0.062041

a // Length

2000000

eldo
  • 67,911
  • 5
  • 60
  • 168
3

Mapping AppendTo over the list of items to append escapes the copy inefficiency.

a = Range[5];
AppendTo[a, #] & /@ Range[6, 10];

a
(* {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} *)

Update with function

ClearAll[joinTo]
Attributes[joinTo] = {HoldFirst};
joinTo[s_Symbol, items_List] := Last@(AppendTo[s, #] & /@ items)

joinTo[a,Range[11,15]]
(* {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15} *)

Hope this helps.

Update 2: Benchmarks

With joinTo in this post:

a = Range[10000];
joinTo[a, Range[10000]] // AbsoluteTiming // First
(* 0.837118 *)

With JoinTo (the OP's version):

ClearAll[JoinTo];
SetAttributes[JoinTo, HoldFirst];
JoinTo[a_, b_List] := (a = Join[a, b];)

b = Range[10000];
JoinTo[b, Range[10000]] // AbsoluteTiming // First
(* 0.000032 *)

To check if the codes are doing the same thing:

a == b
(* True *)
C. E.
  • 70,533
  • 6
  • 140
  • 264
Edmund
  • 42,267
  • 3
  • 51
  • 143
  • 3
    I don't usually add benchmarks to people's answers unasked, but in this case I felt that it was important. When you use Join you only create one copy, regardless of the length of the list. When you use AppendTo the way that you do, you create n copies when appending a list of length n. – C. E. Jan 03 '16 at 07:02
  • @Pickett Oh, I thought AppendTo was not making a copy but adding to the original list. Good to know. Thanks. – Edmund Jan 03 '16 at 13:00
  • @Pickett You have shown it is much slower. How would you demonstrate that it is using more memory? – Edmund Jan 03 '16 at 13:03
  • MaxMemoryUsed indicates that it is not using more memory, however the efficiency that M.R. and Daniel (in his comment) are talking about is not memory efficiency but how fast the computation runs in a loop. What we see here is exactly why it is common to hear that one should avoid AppendTo in favor of, for example, Sow/Reap, or linked lists. – C. E. Jan 03 '16 at 14:11
  • @Pickett I see. Thanks. – Edmund Jan 03 '16 at 15:30
2

Allow to join more than two lists (double underscore behind b)

ClearAll[JoinTo, a, b];
SetAttributes[JoinTo, HoldFirst];
JoinTo[a_, b__List] := (a = Join[a, b];)

a = {1, 2};

JoinTo[a, Range@3, {99}]

a

{1, 2, 1, 2, 3, 99}

eldo
  • 67,911
  • 5
  • 60
  • 168