Making data persistent, as in xmgrace

Question

I've been using xmgrace for quite a long time as my favorite solution for scientific data manipulation and plotting. These data often come from home-made fortran programs, in the form of, e.g., a formatted file with x y z f columns.

I discovered that Mathematica (v9 here) is quite remarkable for this purpose.

In xmgrace, the data you are plotting is stored (made persistent) in the xmgrace file itself (file.agr), which is very convenient because sharing the xmgrace file both shares the plot and the data being plotted.

Is it possible to make the data stored inside the Mathematica (.nb) file?

Note: the answer should be compatible with any quantity of data. For illustration purpose, my xmgrace file can exceed 500 MB.

Related. But I'd think twice before storing 500MB of data in the notebook and suggest you export the data in a separate file and share that as well. — ssch, Jan 18 '13 at 16:06
Agree with @ssch. You can do it but with 500MB of data it will be quite annoying to have to wait with a laggy computer every time you save your file. — Rojo, Jan 18 '13 at 16:36
Please do not focus on the size of the file, which are often of the order of 0.1 to 10 MB. Indeed I could just export when things get laggy, but that's dependent on my computer, hard drive etc so please do not focus on file size. — max, Jan 18 '13 at 16:54
It's not really a good idea to store a large amount of data in the notebook file. The notebook is loaded by the Front End, which is not made for having a large amount of data in memory, like the kernel is. It easy to make a notebook which will load the data from a file which is in the same directory as the notebook. If the data is not large, you can store it literally in the notebook in the form of data = {...}, then put this in a separate section (cell group) that can be collapsed so it doesn't take up visual space. — Szabolcs, Jan 18 '13 at 17:03
In addition to the answers below you may want to also consider DatabaseLink. Have all you data in a database and read/write as required. I have only tried DatabaseLink with MySQL but I've used it regularly for nearly 4 years and in my experience works well. — Mike Honeychurch, Jan 18 '13 at 21:40

score 6 · Accepted Answer · edited Apr 13 '17 at 12:55

For not so big data, what I do often is the following

data = RandomReal[{-1, 1}, 100000];

ByteCount@data

(* 800144 *)

Now, you write data = Interpretation["hidden data", Evaluate@data], then select ONLY THE INTERPRETATION EXPRESSION, and evaluate IN PLACE.

As a result, you get a cell which can be used to load the data, that is fully stored in the notebook.

(I think I read this somewhere suggested by The Futz)

You could also share a package similar to this answer once with your collegues, and then use it

score 3 · Answer 2 · answered Jan 18 '13 at 18:38

Assuming that your data is stored in data, you could also create a button like this:

With[{expr = data}, Button["Set 'data'", data = expr]]

Mathematica graphics

This will output a button that you can copy and paste anywhere in the notebook. Pressing the button will set the value of the variable data.

Another thing to look at is the SaveDefinitions options of Manipulate.

score 1 · Answer 3 · answered Jan 18 '13 at 18:12

For big data, using notebooks with their dynamic interactivity toys is impractical, not just because of slowness but because of stability (i.e., crashes).

So by far the best option in my opinion is to run the Kernel directly. With big datasets in ASCII form you can then easily include the Mathematica code to process them in the same file. E.g., your file could be called dataProcessor.m and it could contain the lines

data = {123.1, 123.4, ... };  (* some huge array *)
processData[d_] := Module[  (* define the processing of your data here *) ]

output = processData[data];

Export["output", output, "Text"]  (* Choose the desired format here *)

For graphics this has become more complicated because the Kernel can't do it on its own in most cases. So output would ideally just contain some boiled-down data (e.g., interpolation functions) that are small enough for realistic visualization to be performed in the notebook.

To execute the file, you would run something like

math -run "dataProcessor.m"

assuming that math is the command to launch the Kernel on your system.

So a .m file with embedded data is the closest you can get to a self-contained structure, I think.

Making data persistent, as in xmgrace

3 Answers3