Using PutAppend to write large data files

Question

I have a loop which computes a table, tab, of results at each iteration. This table is usually very large and I would like to use PutAppendor anything of the sort to save the data in a file and therefore saving memory. An example would look like the following:

Put["test.csv"];

Do[tab=RandomReal[{0,1},10];PutAppend[tab, "test.csv"],{5}]

When I import the data in test.csv for later use I do this via data=Flatten[Import[".../test.csv","Data"],1]. However, the list data is not ready to use because typical elements look e.g. like

In[45]:= data[[1]]

Out[45]= "{0.3105121460958056"

In[43]:= data[[2]]

Out[43]= 0.0657094

In[44]:= data[[4]]

Out[44]= " "

Which is a mixture of strings and reals.

So my question is: how can I append the table tab at each iteration of the loop to a file which later is ready to use as a list of rank two of reals?

Thanks for your help.

Answer 1

This appears to be working:

Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, "test123.csv"], {5}]

ReadList["test123.csv", Expression]

You may also wish to look at WriteString which will allow for complete control of what is written.

---

In a comment Ajasja makes an excellent point regarding performance. If your application is anything like the dummy above, with little data being written in each pass of a fast loop, you should be using streams. This is somewhat less convenient in that you must remember to Close the stream when you are done but it is much faster.

You can use either PutAppend or WriteString with streams, though use with PutAppend is not well documented. I'll show both and perform a three-way timing for comparison.

Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, "testAppend1.csv"], {5000}] // AbsoluteTiming

{4.8828086, Null}

str = OpenAppend["testAppend2.csv"];
Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, str], {5000}] // AbsoluteTiming
Close[str];

{0.2340004, Null}

str = OpenAppend["testAppend3.csv"];
Do[tab = RandomReal[{0, 1}, 10]; WriteString[str, ToString[tab, InputForm], "\n"], {5000}] //
  AbsoluteTiming
Close[str];

{0.2028004, Null}

Notice that with WriteString I needed to use ToString -- this is because WriteString will convert into OutputForm by default, which truncates digits.

For the sake of convenience I would use PutAppend myself, despite the lack of documentation and the slightly slower speed.