4

I have the following two lists (each containing over 500,000 elements). Here is a sample:

lis1 = { {1.86582, 1.70162, 1.25256}, {1.82707, 1.29901, 1.10659}, 
  {1.76547, 1.21544, 1.09433}, {1.18306, 1.28322, 1.75524}, 
  {1.12555, 1.98011, 1.53359}, {1.10584, 1.12299, 1.88411}, 
  {1.83799, 1.5275, 1.76179}, {1.42352, 1.45163, 1.45318}, 
  {1.63669, 1.78145, 1.60307}, {1.61749, 1.44287, 1.57405} 
};

and 

lis2 = {0.826095, 0.73286, 0.918137, 0.937434, 0.506525, 
        0.562795, 0.664915, 0.789321, 0.6559, 0.398447}

They both have equal Lengths. I want to combine both Lists to obtain a new list that looks like this

{ {1.86582, 1.70162, 1.25256, 0.826095}, 
{1.82707, 1.29901, 1.10659, 0.73286}, 
{1.76547, 1.21544, 1.09433, 0.918137}, 
{1.18306, 1.28322, 1.75524, 0.937434}, 
{1.12555, 1.98011, 1.53359, 0.506525}, 
{1.10584,1.12299, 1.88411, 0.562795}, 
{1.83799, 1.5275, 1.76179, 0.664915}, 
{1.42352, 1.45163, 1.45318, 0.789321}, 
{1.63669, 1.78145,1.60307, 0.6559}, 
{1.61749, 1.44287, 1.57405, 0.398447} }

That is, each element of the combined list contains the corresponding elements of lis1 and lis2, with the first three elements being from lis1 and the fourth element from lis2. Here is what I did to combine them in this fashion:

Transpose[{lis1, lis2}] //. {{a_, b_, c_}, d_} :> {a, b, c, d}

Is there a faster way to achieve this as my lists are huge ?

Artes
  • 57,212
  • 12
  • 157
  • 245
RunnyKine
  • 33,088
  • 3
  • 109
  • 176

1 Answers1

4

There is also Transpose : 

Transpose @ Append[ Transpose[lis1], lis2]

Transpose is a really fast approach, and you need to append only one list lis2. So this probably can be faster than Rojo's approach and :

Transpose @ Append[ Transpose[lis1], lis2] == Join[lis1, List /@ lis2, 2]

 True

Edit

Leonid Shifrin provided in the comments another neat approach :

Join[ large1, Transpose[{large2}], 2]

which appears to be even faster on one-core machines and sometimes on multi-core ones. Following his idea we can use this testing function :

test := ( large1 = RandomReal[{0, 2.}, {10^6, 3}];
          large2 = RandomReal[{0, 2.}, 10^6];
         { Do[ Join[ large1, List /@ large2, 2], {100}]// AbsoluteTiming //First,
           Do[ Transpose @ Append[ Transpose[large1], large2],
                {100} ] // AbsoluteTiming // First, 
           Do[ Join[ large1, Transpose[{large2}], 2], {100}]// AbsoluteTiming// First})

Here are some benchmarks :

0n Win 7 64 bit AMD Phenom II x6 2.8 GHz, V9_0 :

 test

{10.413086, 8.477539, 5.245117}

while RunnyKine on his machine :

Win 8 64bit Intel Core i7-2600 3.4GHz 16GB RAM, V9.0, got

test

{6.006038, 1.794012, 1.825212}

Mr.Wizard on v7.0, Win7-64, i5-2500K 8GB RAM:

test

{6.2400087, 2.5000035, 2.2700032}

on one-core old Pentium(R) 3.0 GHz 1GB RAM, V9.0

 test

{16.812500, 14.328125, 9.203125}

All the above benchmarks seem to suggest that Transpose remarkably accelerates on multi-core machines and then Transpose @ Append[ Transpose[lis1], lis2] might be faster than Leonid's solution.

Artes
  • 57,212
  • 12
  • 157
  • 245
  • This is fast. +1. It's even better as the List size grows. – RunnyKine Mar 05 '13 at 20:26
  • Good one, +1. See my comment on @Rojo's code, where I suggest an improvement, which makes it about twice faster than yours. – Leonid Shifrin Mar 05 '13 at 20:43
  • @LeonidShifrin, actually as the List gets bigger, his is actually faster (2x) at least on my computer. – RunnyKine Mar 05 '13 at 20:47
  • @RunnyKine Did you see my comment? I suggested an improvement on Rojo's code, and it is that improved version that, at least on my machine, is about 4 times faster than the original one, and twice faster than Artes's version, for large lists. I tested for the length 10^5 and 10^6. Heuristically, it seems that the reason my version is twice faster is that it has only one Transpose operation. – Leonid Shifrin Mar 05 '13 at 20:53
  • @LeonidShifrin, yes I saw your comment, I used the improved version you suggested. I tested it for 10^5 elements, looping over 1000 times and his is faster on my computer. This is weird. – RunnyKine Mar 05 '13 at 20:58
  • @RunnyKine Weird indeed, as I get the opposite result (Win 7 64 bit,V9_0). – Leonid Shifrin Mar 05 '13 at 21:04
  • @LeonidShifrin and RunnyKine, Thanks for upvotes. It's a pitty that the question was marked as a duplicate, but if you could be so kind, feel free to edit my answer adding some benchmarks. Unfortunately, at the moment I can't do any reliable tests with my computer at hand. – Artes Mar 05 '13 at 21:30
  • @LeonidShifrin Thanks for adding your benchmarks. – Artes Mar 05 '13 at 21:57
  • 1
    @LeonidShifrin, thanks for adding the benchmarks, we were editing at the same time. On my computer the corresponding times are: {6.006038, 1.794012, 1.825212}. – RunnyKine Mar 05 '13 at 22:01
  • @RunnyKine Thanks, it's interesting. Could you provide information on the system and the processor you are using when getting these timings ? Do you get them for the same (Leonid's) code ? Perhaps it migth be because of some default paralellization on your multi-core machine. – Artes Mar 05 '13 at 22:07
  • @Artes, yes I'm using Leonid's code. Win 8 64bit Intel Core i7-2600 3.4GHz 16GB RAM, V9.0 – RunnyKine Mar 05 '13 at 22:18
  • {6.2400087, 2.5000035, 2.2700032} here; v7, Win7-64 – Mr.Wizard Mar 05 '13 at 22:21
  • I think it's not single-core vs. multi-core, but rather that the multi-core machines have more memory bandwidth and by using Transpose instead of doing it at the top level one makes better use of this resource. Of course, if the memory manager is multithreaded, there might be benefits on multi-core machines for that reason. – Oleksandr R. May 27 '13 at 15:56