How can I validate this dataset in an efficient manner?

Question

I have a large dataset in a single file, consisting of one positive integer per line. The file is more than 20GB in size, and before I do anything with the data I need to validate that everything is as I expect it. For this, I need to check that the number on line $n$ is always strictly greater than the one on line $n-1$.

How can I achieve this in an efficient manner?

This is how I am loading the file into memory:

(* Load the data file *)
SetDirectory[NotebookDirectory[]];
Print[DateString[], " - Loading dataset..."]
dataset = ReadList["dataset.txt"];

And I can simply access the element at index $i$ with

dataset[[i]]

Thanks!

PS. I am using Mathematica 11.1 and I have a lot of RAM, so space is not an issue.

Have a look to this answer that uses OpenRead and Read to digest data line by line. Would something like that help? — rhermans, Dec 08 '22 at 10:24
@DanielHuber with d = {4, 2, 3, 4} your AllTrue returns True instead of False. — Edmund, Dec 08 '22 at 22:59
You are right. Here is the correct version: d = {4, 2, 3, 4}; AllTrue[Transpose[{Rest[d], Most[d]}], Greater @@ ## &] — Daniel Huber, Dec 09 '22 at 09:21

score 1 · Answer 1 · answered Dec 08 '22 at 22:58

1

You can use Thread and And.

l1 = {1, 2, 3, 4};
And @@ Thread[Rest[l1] > Most[l1]]

True

l2 = {4, 2, 3, 4};
And @@ Thread[Rest[l2] > Most[l2]]

False

Hope this helps

answered Dec 08 '22 at 22:58

Edmund

42,267
3
51
143

How can I validate this dataset in an efficient manner?

1 Answers1