Is *.dat file suitable for storing high (>2) dimensional matrix?

Question

The typical process goes as first one exports a numerical matrix mat

Export["test.dat",mat,"Table"]

and then imports it back by using

Import["test.dat","Table"]

This is necessary, especially when the evaluation of mat is relatively time-consuming and the value of mat is expected to be used many times.

When mat is a 1-dimensional vector or 2-d matrix, one gets what s/he wants (what is imported is what exactly has been exported). However, to one's surprise, the procedure fails when mat is a 3-d matrix (of Depth 4) --- what imported becomes many strings.

It has been found that *.mat file may do the job. But, is there any way to make *.dat to do the job as well?

worth a note, the "dat" extension has no bearing on what you did there (or any of the answers here). The last argument to Import/Export is a directive to ignore the extension. (the dat extension is so generic it is meaningless anyway) — george2079, Sep 17 '16 at 14:07

score 7 · Accepted Answer · answered Sep 08 '16 at 08:59

7

Use Put and Get.

Having

l1 = {{a1, {a, b, c, d, e}}, {a2, {f, g, h, i}}}

one exports using

Put[l1,"out.dat"]

and imports next with

Get["out.dat"]

This works for arbitrary depth lists.

answered Sep 08 '16 at 08:59

corey979

23,947
7
58
101

@Feyre: where do you see temp files here? – Albert Retey Sep 08 '16 at 16:14
@AlbertRetey I misread the documentation. I was surprised Put[] works so much faster. I don't think Export[] is better than Put[] in this case, in fact Put[] seems to be faster. I have concordantly upvoted this answer. – Feyre Sep 08 '16 at 17:48
Marvelous & remarkable! – Αλέξανδρος Ζεγγ Sep 10 '16 at 06:13
Another question. Can we set where to put things? – Αλέξανδρος Ζεγγ Sep 17 '16 at 13:46
@Alexander, look at SetDirectory[]. – J. M.'s missing motivation Oct 08 '16 at 13:31

Feyre · Answer 2 · 2016-09-08T08:44:43.523

5

dat = RandomReal[10, {3, 3, 3}];
Export["test.dat", dat, "List"];
dat2 = ToExpression[Import["test.dat", "List"]];
Simplify[dat == dat2]

True

dat // MatrixForm
dat2 // MatrixForm

edited Sep 08 '16 at 08:44

answered Sep 08 '16 at 08:39

Feyre

8,597
2
27
46

It seems that "ToExpression" is indispensable? – Αλέξανδρος Ζεγγ Sep 08 '16 at 09:18
@AlexanderZeng Unless you want to work with temp files, yes. – Feyre Sep 08 '16 at 09:33
Can you explain why you think this is better than just using Put and Get? – Albert Retey Sep 08 '16 at 16:15

score 4 · Answer 3 · answered Sep 08 '16 at 16:24

If you just need to store an expression to file to later reload it to Mathematica the by far most efficient way is this:

Export["mat.mx",mat]
mat = Import["mat.mx"]

it will work for every Mathematica expression containing whatever you want and that of course includes arrays of arbitrary depth. The drawback is of course that these files can only be reloaded with Mathematica and no other program. And AFAIK there is no guarantee that they are compatible between different (major) Mathematica versions.

(note that for some newer data structs like graphs, datasets, neural networks, mesh regions... there might be bugs for some versions, so you better check whether what you want to safe can be reloaded, but the same is true for any other file format of course)

score 2 · Answer 4 · answered Sep 08 '16 at 09:45

2

I have been using

Export["file.txt", Compress[whatever]]
Uncompress[Import["file.txt"]]

answered Sep 08 '16 at 09:45

Coolwater

20,257
3
35
64

Is *.dat file suitable for storing high (>2) dimensional matrix?

4 Answers4

Linked