3

I am using the Weka workbench to train a protein fold classifier. I imported my training data into Weka and performed PCA-based feature selection. This seems to have worked fine, but now I cannot evaluate my trained classifier on the test data because the test data contains all the original attributes. Of course, if I try to run the feature selection on the test data, I will come up with a different set of features.

In Weka, after you have applied feature selection to a training set, how do you pull those same features out of a test set?

Daniel Standage
  • 663
  • 5
  • 17
  • Any reason why you can't use R for this job? One predict there and you are done. – mbq Dec 09 '11 at 13:20
  • ...or classify in Matlab. But we can't really answer every question by telling people to use our analysis package of choice :) – Colin K Dec 09 '11 at 16:21
  • @ColinK That's why it is a comment not a close request (-; – mbq Dec 09 '11 at 19:49

2 Answers2

3

With any such modelling thing, you are going to have to recalculate the model using the new training set (ie. the original minus its test set).

The usual approach is to randomly extract a subset for testing. Then train using all of the remaining data points.

Of course there will be some random variability according to which are extracted, so you can repeat the process to get some statistical significance.

Your final model will not be trained on all of your data. You either have to live with that (what I usually see in the Natural Language Processing field), or once you have determined your best parameters compute the final model using all of your data - with the understand you won't be able to test it.

winwaed
  • 658
  • 5
  • 13
  • Unfortunately, I am constrained by the fact that the method I am comparing against has already separated training and test data. If I were to combine all of the data and use the approach you suggested, it would not allow me to make an accurate comparison against that method. – Daniel Standage Dec 12 '11 at 14:51
1

It appears that the PCA-based feature selection generates pseudo-features that are linear combinations of the original features. After performing feature selection on the training set, I have not found a way to pull out the same pseudo-features from the test set.

However, if you're not tied to a particular feature selection method (PCA in this case), you can alternatively use a feature selection method that is more straightforward. For example, I ended up using a method that sorts the features according to information gain. Using this method, it is easy to identify the top n features that give the best information gain and extract only those features from the training and test data.

As suggested in the comments, there may be additional alternatives implemented in different languages (which I would be willing to consider, given a more detailed response), but this is the best I could find in the Weka environment.

Daniel Standage
  • 663
  • 5
  • 17