2

I have tried to use OpenRead for my application as it appears to be less of a burden on the memory.

I OpenRead a .csv file to extract coordinates in the form {x,y} in the following way

ClearAll[f];
f = OpenRead["mathfile.csv"];
g = ReadList[f, String, RecordLists -> True];
Close[f];

I can access each record seperately by just calling g[[1]]. However the format is not as numbers and so a function like ListPlot[g[[1]]] doesn't work.

If I use "Number" or "Real" as an option within ReadList I get an error:

"Read:readn: Invalid real number found when reading from 'f'."

Using "String" or "Record" within ReadList doesn't cause a problem. However I would have to convert the "String" or "Record" into a number format.

What am I doing wrong ? I would like to call g[[1]] and have the co-ordinate in a form that can be read by ListPlot

Thanks

Dan Oak
  • 638
  • 3
  • 15
BRS
  • 41
  • 1
  • A little more information, please. When you say "{x,y}", do you mean a sample list entry would look like "{1,2}" for example? That's not a number, it's an expression. You can map ToExpression[] after loading g, not sure you can do it while reading in the CSV. –  Jul 29 '17 at 16:31
  • I Have two columns in Excel (as an XLSX) which I have saved as .csv to enable speedy handling of a large data set. By {x,y}, I refer to a co-ordinate given by calling g[[1]] in my code above. I am trying to use ToExpression at the moment , but after I close 'f' .. will try before that after loading g.... I keep getting the error "ToExpression::sntx: Invalid syntax in or before "0.000,-5.750E-11"....so I think it is almost doing what I want it to but there must be something (format) that gives the erro r...as you can see the {x,y} is being reported in the error – BRS Jul 29 '17 at 16:58
  • I have also checked the CSV file in Notepad and there is nothing 'in or before' in terms of format that is strange .... the OpenRead import may be placing an additional character perhaps ?? – BRS Jul 29 '17 at 17:25
  • I'm pretty sure Mathematica doesn't recognize "E" in numbers. In others words, "1E6" does NOT mean 10^6. Try search/replacing E with *10^ –  Jul 29 '17 at 18:43
  • Mathematica explicitly accepts E format in Read and ReadList. Answer to question coming shortly. – ZachB Jul 29 '17 at 18:52
  • The problem seemed to resolve itself by converting the XLSX file into a .txt file . Mathematica didn't have a problem with reading in the numbers with ReadList like that... E was recognised ...but now I have the problem with trying to read large portions of the file in as the length has seemed to max out at 1501 points – BRS Jul 29 '17 at 18:54

3 Answers3

4

Mathematica doesn't have configurable separators for Read and ReadList, except when reading Words and Records, so importing a CSV of numbers is trickier. Some workarounds:

  1. Use TSV format, which will "just work."

  2. Read a Character to get rid of the comma separator:

    ReadList[stream, {Number, Character, Number}, RecordLists -> True][[All, {1,3}]]

  3. Set WordSeparators -> {","}, the read Words and apply ToExpression. This is maybe the slowest.

BTW, there are a few nice (albeit maybe hefty) answers in this related question that you might want to look at.

ZachB
  • 1,200
  • 9
  • 19
0

If you use OpenRead as described above you'll have to iterate over your records a second time in order to convert strings to pairs of numbers. This is not memory-efficient. Also, if your input stream will no longer be needed after you load the records, it's better to invoke With[{f=OpenRead["mathfile.csv"]},...] so as to localize the variable as a constant.

If you have a CSV with records of the form:

x1, y1
x2, y2
...

you can simply use the Import function's CSV support, like so:

g = Import["mathfile.csv","CSV"]

This will return a two-dimensional list of records of the form {{x1,y1},{x2,y2},...}. Then you get

g[[1]] == {x1,y1}

Use the Import option HeaderLines->n to skip n lines if you have named columns etc.

In my opinion, the capabilities of Import are quite robust and efficient-- even for large files-- and should be considered before falling back to low-level streams, particularly for simple record formats like 2-tuples.

terrygarcia
  • 236
  • 1
  • 5
0

I got some good performance in the following manner. My data was originally an XLSX. I saved it as a MSDOS txt file. Unfortunately, the {x,y} data had long runs punctuated by a text saying 'Segment 1:' e.t.c.. This was stopping the ReadList from working..So after removing the text and whitespace within Excel, it was saved as .txt.I didn't use Import, as it just takes too long and eats up memory (I have a total 300 000 rows of {x,y} data that I want to build a dynamic animation with) with a CSV....

So I basically open the txt for reading and then use ReadList to extract a number of entries in blocks... this is quite quick as my data has a high resolution and so not every data point is necessary for a good looking plot.

ClearAll[str];
str = OpenRead["mathfile1.txt"];
g = Table[ReadList[str, {Real, Number}, 1000], {100}];
Close[str];

Entries can be accessed as g[[1,x]]

I found that if ReadList reads over 15000 entries, my computer starts to seriously hang. I have 8 GB of RAM

BRS
  • 41
  • 1