3
filedata={"    1       2578Hot Pink            47:21:17  9512971  ",
          "    2        294Pale Green          24:57:18  4437289  ",
          "    3       2497Light Yellow        39:40:57  4987642  ",
          "    4       2010Turquoise           16:04:17  3290855  "}

There is an external file.
You open a Stream.
You Skip to a far postion.
It takes a long time for a large file.
How can one conceive a more direct method for a special
straightforward highly regular file structure like this?
We got an index and know the file size anyway.
A low level file function approach?

Update

Records in file: 100.000.000
Set StreamPosition to: 60.000.000
Time needed: 40 seconds

That is: Skip and SetStreamPosition takes the same chunk of time.

Hp Radojewski Schäfer Von
  • 1,283
  • 7
  • 14
  • Break the file up in many small separate files. Even add a directory tree structure to the chunks if you like. – image_doctor Jan 18 '13 at 15:28
  • 3
    If your records are truly all fixed length as your question suggests, it should be easy to use SetStreamPosition to go to the n-th record. – m_goldberg Jan 18 '13 at 15:38
  • To break the file up and adding a directory is what a database is about. . .Here its just the file as it is. . .The use of SetStreamPosition is the very time consuming issue. . .See some timing figures in update above. – Hp Radojewski Schäfer Von Jan 19 '13 at 07:45

1 Answers1

8

Are you on a Windows machine? If so, try OpenReading the file with BinaryFormat->True.

When you open a file on a Windows machine in "text mode," with BinaryFormat->False (which is the default), the system has to translate the Windows newline sequence \r\n (two bytes which mark the end of the line) into the single \n character. In the C language and in Mathematica, \n (one byte) marks the end of a line.

As a result, there is a translation step that must happen. Even if you use SetStreamPosition[stream, n] it must work out where character n is, which requires looking at all the intervening characters. For gigantic files, this can take some time. (It builds a table in memory so it only has to work it out once.)

If you open the file with BinaryFormat->True, you will be responsible for handling the difference between \n and \r\n yourself. But in that case, there is no translation, and SetStreamPosition is essentially instantaneous. Since you know the exact length of each line (adding 1 extra byte for each newline on Unix, 2 extra bytes on Windows!) you can compute exactly where SetStreamPosition should go.

Another way, which doesn't require BinaryFormat->True, is to "pre-load" the translation step. After you open the file for the first time, do SetStreamPosition[stream, Infinity] to set the stream position to the end of the file. In the process, all the newlines will be scanned along the way. That's slow at the beginning, but after that point any calls to SetStreamPosition will be fast, because it will have already built the table.

Don't use Skip when you have fixed size lines -- that's a waste. Skip is really just Read without saving the data that's read. It's appropriate when you don't know how long each record will be and you actually have to process the record to find the next one.

If you're not on a Windows machine I have no idea why SetStreamPosition would be slow.

librik
  • 1,526
  • 9
  • 12