Efficient List of Lists

Question

I'm running some analysis that requires the creation, row by row, of a large list of lists. The big list is called "Nodes" and it will ultimately contain hundreds of thousands of rows. The first few rows will look like this ...

Row 1: {{1,3,5}, {2,4,6}, {7}, {6,7,8,9}}

Row 2: {{7,3,8}, {2,5,4}, {2}, {6,2,8,4}}

Row 3: {{9,7,5}, {5,4,6}, {1}, {6,7,8,9}}

I've tried AppendTo and Join in this way ...

Nodes = AppendTo[Nodes,List[{8,3,2},{1,2,3},{8},{8,3,3,2}]];

Nodes = Join[Nodes,List[{8,3,2},{1,2,3},{8},{8,3,3,2}]];

and they both take quite a long time to run.

I also tried creating a huge Nodes array ahead of time and then replacing each row, something like this ...

Nodes = Range[200000];
For[ji = 1, ji < 200001, ji++, Nodes[[ji]] = List[{}, {}, {}, {}]];

followed by

Nodes[[nodesrownumber]] = List[{8,3,2},{1,2,3},{8},{8,3,3,2}]];

and this seems to take even longer.

Any suggestions for improving the speed of this? Thanks, very much, in advance.

Welcome to Mathematica.SE! I hope you will become a regular contributor. To get started, 1) take the introductory Tour now, 2) when you see good questions and answers, vote them up by clicking the gray triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge, 3) remember to accept the answer, if any, that solves your problem, by clicking the checkmark sign, and 4) give help too, by answering questions in your areas of expertise. — bbgodfrey, Jul 31 '15 at 03:57
Iteratively appending/joining will kill performance. Also, you don't need to set the variable when using AppendTo. In any case, where do the elements come from? Are you generating them randomly, from prior results in list, from some other list, from... you need to specify this question much more precisely. — ciao, Jul 31 '15 at 04:11
AppendTo Is probably your problem because it generates a copy of its argument before appending. Look into using e.g. Reap and Sow instead (http://blog.wolfram.com/2011/12/07/10-tips-for-writing-fast-mathematica-code/ and http://mathematica.stackexchange.com/questions/70149/lookup-construction-using-data-matrix-and-reap-sow-versus-appendtos). We will also need to know how you are generating your lists in order to help you. — MarcoB, Jul 31 '15 at 04:15
@Marco, I forgot about that blog entry; thanks for linking to it. However, tip #3 seems to be becoming a bit less applicable with each new version, unfortunately. :( — J. M.'s missing motivation, Jul 31 '15 at 05:01
@Guess Indeed. I was most recently shocked by MinimalBy and TakeSmallestBy, both really new functions (2014 and 2015 respectively), and both really slow at doing the very thing their name implies... — MarcoB, Jul 31 '15 at 15:25

m_goldberg · Answer 1 · 2015-07-31T18:25:01.243

One approach is to construct the rows with a head other than list, say row, then accumulate the rows as a linked list, and finally transform the linked list into a simple list of sublists of the form wanted. Here is an example.

Row constructor.

 newRow[] := 
   With[{rs = RandomSample[Range[11]]},
     row[rs[[;; 3]], rs[[4 ;; 6]], {rs[[7]]}, rs[[8 ;;]]]]

On my system, a not very fast, aging iMac, I get the following timing for making a table of 100000 rows as I described.

timer[n_] :=
  Module[{data = {}},
   AbsoluteTiming[
     Do[data = {data, newRow[]}, {n}];
     List @@@ Flatten@data][[1]]]

timer[100000]

0.866738

That's less than a second for a 100000 rows, so maybe it will be fast enough for you.

And just to show that the process actually produces the structures described, consider

Module[{data = {}},
  Do[data = {data, newRow[]}, {3}];
  Column[{data, List @@@ Flatten@data}]]

slick. This could probably use a little more explanation. – george2079 Jul 31 '15 at 19:02 — george2079, Jul 31 '15 at 19:02

score 1 · Answer 2 · answered Jul 31 '15 at 20:24

Maybe there's something I do not understand because this seems fast to me. I used Do instead of For, since it is faster:

(nodes = ConstantArray[0, {200000}];
  Do[
   nodes[[ji]] = List[{8, 3, 2}, {1, 2, 3}, {8}, {8, 3, 3, 2}],
   {ji, 200000}];) // AbsoluteTiming
(*  {0.13, Null}  *)

But it's not even twice as fast as For:

(nodes = ConstantArray[0, {200000}];
  Block[{ji},
   For[ji = 1, ji < 200001, ji++,
    nodes[[ji]] = List[{8, 3, 2}, {1, 2, 3}, {8}, {8, 3, 3, 2}]
    ]];) // AbsoluteTiming
(*  {0.246, Null}  *)

If the calculation of a row can be done independently, either as a function of some parameters or none, then Table will be faster still:

calcrow[i] := List[{8, 3, 2}, {1, 2, 3}, {8}, {8, 3, 3, 2}];
(nodes = Table[
     calcrow[i],
     {i, 200000}];) // RepeatedTiming
(*  {0.086, Null}  *)

nextrow[] := List[{8, 3, 2}, {1, 2, 3}, {8}, {8, 3, 3, 2}];
(nodes = Table[
     nextrow[],
     {i, 200000}];) // RepeatedTiming
(*  {0.018, Null}  *)

Some baseline comparisons for iteration and memory storage:

(* all 1's *)
(#[1, {200000}] // RepeatedTiming // First) & /@ {Do, Table, ConstantArray}
(*  {0.0031, 0.0034, 0.0000663}  *)

(* rows of 11 elements *)
(#[Evaluate@ Internal`PartitionRagged[Range@11, {3, 3, 1, 4}], {200000}] // 
     RepeatedTiming // First) & /@ {Table, ConstantArray}
(*  {0.0062, 0.0012}  *)

Notes: Do does not produce a table, so it gives some idea of a lower bound on iteration. ConstantArray is a rather specialized & optimized table-generator; timings with Table probably reflects the lower bound in generating a table.

Efficient List of Lists

2 Answers2

Linked