Fast way to add a column to a large matrix

Question

I have used the (apparently fastest) approach of

Do[MapThread[Prepend, {m, v}];

for prepending and appending v to a matrix m from question What is the most efficient way to add rows and columns to a matrix? but my matrix is quite large and it is too slow for my purposes. (Not at home now but rough performance guide is ~30 seconds to append a 5,000 row vector to a 5000 row matrix to 2 cols)

The specific context: a matrix of time series data (imported from CSV) of 50,000 rows (could be MUCH longer) and 2 columns (col 1 timestamp; col 2 value). I want to add a third column and re-export to CSV.

Is there a better approach than the method quoted?

It is unclear from your description: do you actually use the imported data to create the new column? If not, there seems no need to even create the matrix. And even if you do need to actually import the data into a matrix, you need not create the bigger matrix to export the data. — Alan, Mar 24 '17 at 14:17

score 2 · Accepted Answer · answered Mar 24 '17 at 16:23

2

Adding the column shouldn't be too time consuming. Here's a 1,000,000 x 2 matrix of real numbers getting a column added, with two Transpose[ ] calls thrown in. Too cheap to meter. The file writing is the expensive part.

In[107]:= n = 1000000;

In[108]:= res = Table[RandomReal[20], {i, n}, {j, 2}];

In[109]:= vec = Table[RandomReal[20], {i, n}];

In[110]:= Timing[tres = Append[Transpose[res], vec] // Transpose;]

Out[110]= {0., Null}

In[111]:= Timing[Export["test.csv", tres]]

Out[111]= {19.2193, "test.csv"}

answered Mar 24 '17 at 16:23

MikeY

7,153
18
27

(Working to a deadline so no time to got back and investigate but to acknowledge and thank...) Yes, that seems fast enough ;) I think the problem may have been that one column was of reals and the other of DateObjects. I've bypassed the conversion of timestamps to DateObjects for now and with only text/numbers it is plenty fast. Thx again – Julian Moore Mar 25 '17 at 09:41

score 0 · Answer 2 · answered Mar 24 '17 at 15:28

mat = RandomInteger[100, {10, 2}];  (* fake matrix *)
src = "/temp/temp99.csv";  (* location for fake data *)
Export[src, mat];
vec = RandomInteger[100, 10];  (* fake vector *)
tmp = StringTemplate["``,``"];  (* string template for conversion *)
fOld = OpenRead[src]; (* open your data file to read *)
fNew = OpenWrite["/temp/temp98.csv"] (* target for new data *)
(* each item of vec, read old line & write new line *)
Scan[WriteLine[fNew, tmp[ReadLine[fOld], #]] &, vec];
Close[fNew]; Close[fOld];

Fast way to add a column to a large matrix

2 Answers2