More elegant way

Question

I read ReplaceOrRemoveInvalidOrMissingData Mathematica documentation, and I tried to apply the same approach to my dataset.

I've got two problems, but one is only a style problem!

1. There is one more elegant way to select numeric data in a large dataset? I mean that I've got a dataset o 29 possible numeric value and 2 columns that are not numerical. So I tried this way.

   baddata[entry_] :=  Not[MatchQ[ entry, 
     {_?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
      _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, 
      _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
      _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ, _?NumberQ,
      _?NumberQ, __}]]
   
   DeleteCases[data, _?baddata];
   

It works, but it is not so elegant!

2 . I would accept some non numerical (e.g. blank values) values in some columns (e.g. the first, the twelfth and the twentieth). How can I modify this algorithm? Thanks in advance.

---

EDIT

Your answers are very useful for me, and I'm wondering how all of you use Mathematica in a very smart way! For my first question it's very clear all your solutions, but, it is not clear for me (surely I didn't explain in a right way my problem) if I've got this kind of dataset

mydataset={{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}, {20, 30,
40, ""}, {100, 200, 300, ""}};

I want to select the row in this way: I can admit blank entries for the second column, but not for the fourth column. So the result would be:

result={{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}

Can I modify your algorithm in this way? Thanks again, and sorry!

Answer 1

Shorter at least...

data = Cases[data, {Repeated[_?NumberQ, {29}], _,_}];

Answer 2

For the first example Repeated, as already shown by Ymareth, is the simplest method:

MatchQ[
 {1, 2/3, 3.14, 4, 5.0, "x", "y"},
 {Repeated[_?NumericQ, {5}], _, _}
]

True

For the case of arbitrary positions it seems to me that there are two natural approaches:

Method 1: Build a pattern

You can programmatically build a pattern like the one shown in you question. There are many possible methods, but since building the pattern itself will be very fast I'll use the first one that comes to mind: SparseArray:

pat = Normal @ SparseArray[1 | 3 | 7 -> _, 10, _?NumericQ]

{_, _?NumericQ, _, _?NumericQ, _?NumericQ, _?NumericQ,
 _, _?NumericQ, _?NumericQ, _?NumericQ}

This can easily be turned into a function, e.g.:

fn[non_, len_] := Normal @ SparseArray[non -> _, len, _?NumericQ]

fn[2 | 4, 6]

{_?NumericQ, _, _?NumericQ, _, _?NumericQ, _?NumericQ}

This kind of pattern has early exit behavior: as soon as one element fails a test the pattern fails to match, and no additional time is spent on needlessly testing the other elements.

Method 2: Extract the elements to test

The other approach is to extract (or delete) only the elements that are to be handled specially. For the first example you could write a test function like this:

test = Length[#] == 31 && MatchQ[Take[#, 29], {__?NumericQ}] &;

Join[Range[29], {"foo", "bar"}] // test

True

For your second example you could include Delete:

test2[non_, len_][x_] := 
  Length[x] == len && MatchQ[Delete[x, List /@ non], {__?NumericQ}]

{"a", 2, "c", 4, 5, 6, "g", 8, 9, 10} // test2[{1, 3, 7}, 10]

True

This also has early exit behavior without extraneous testing.

---

Application examples

Addressing the addendum to your question using this data:

mydataset = {{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}, {20, 30,
40, ""}, {100, 200, 300, ""}};

First method:

Cases[mydataset, fn[2, 4]]

{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}

Second method:

Select[mydataset, test2[2, 4]]

{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}

Cases is used for the first method because it builds a pattern, while Select is used for the second because it builds a test function.

Answer 3

You say you want to be able to decide which columns have to be numeric and which ones don't. There are many approaches that aren't based on patterns and this is probably not the best one, but it will do the job. Let's define

checkNumericColumns[entry_, blanks_] := 
 Module[{shouldBeNumeric, areNumeric},
  shouldBeNumeric = Complement[Range@Length@entry, blanks];
  areNumeric = Union @@ Position[entry, _?NumberQ];
  shouldBeNumeric == areNumeric
  ]

so that

checkNumericColumns[{1, 2, 3, "a", 4, 5, "b"}, {7}]
(* Out: False *)
checkNumericColumns[{1, 2, 3, "a", 4, 5, "b"}, {4, 7}]
(* Out: True *)

Now you can select the elements you want to keep from your original data set like this:

Select[data, checkNumericColumns[#, blanks] &]

where blanks is a list of columns that you represented with blanks.

Answer 4

For the case with non numerical columns that are allowed.

sel[dat_, nonNum_] := With[{n = Length@dat[[1]]}, 
  Pick[
   dat, 
   And @@@ Map[NumericQ, dat[[;; , DeleteCases[Range[n], Alternatives @@ nonNum]]], {2}]
   ]
  ]

so for the edit test:

sel[mydataset, {2}]

{{1, 2, 3, 4}, {3, 4, 5, 4}, {1, "", 3, 4}, {10, "", 12, 13}}