How can I use list processing functions on data in a file?

Question

How might I work with the following functions on a list, if the list is a disk based file?

Partition[]
DeleteDuplicates[]
Split[]  
SortBy[]

The file is too large to ReadList[] or Import[] the data in the first place.

You might find information that will help you by reading this question and its answers. — m_goldberg, Mar 23 '13 at 21:47

score 2 · Answer 1 · edited Mar 23 '13 at 21:37

If your data set is too large to read in one big chunk, it seems unlikely that you'd be able to use those functions on it since they need to hold the entire set in memory.

However, if there are a lot of duplicates, you may be able to work with it if you read the data line-by-line instead. Use Read[] on a stream you Open[] on the file. Prune the duplicates inside a function that reads the data that way.

How can I use list processing functions on data in a file?

1 Answers1