1

I read the paper but found nothing talking about how to implement incremental learning.

Can someone share some basic or deep knowledge? not in coding way.

I know how to write code snippet to train incrementally.

When new data comes in, how to train incrementally if I use XGBRegressor? Reserve the old trees and train new data with new trees?

I found nothing talking about this in detail

code_worker
  • 163
  • 1
  • 1
  • 10

1 Answers1

3

In the fit method, parameter xgb_model can be specified to continue training an old model.

https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor.fit

Edit: Yes, as I understand it, the old trees are preserved in this method. The new trees will fit the residuals of whatever data you pass: you could continue with only the new data, which will first be run through the existing trees and have their residuals fit by new trees; or continue with all the data, which will do the above but also use the old data when splitting (probably better, so that the model doesn't lose its fit to the old data, but maybe your use case cares about the new data substantially more?). In some cases, it may be better to just retrain from scratch (if the existing trees do a poor job on the new data).

Ben Reiniger
  • 11,770
  • 3
  • 16
  • 56
  • 1
    I know, I am curious about the detail, how to determine the split point? or should save what in model? – code_worker Mar 19 '19 at 01:45
  • thx for updating. My data may too much to store in server, so I train new data incrementally and remove old data. After iterative training incrementally with new data, it predict result with the newer trees rather than old trees? – code_worker Mar 19 '19 at 05:07
  • 1
    No, after incremental training the model retains all the trees; any new predictions will get run through the entire set of trees.

    See also (1) https://stackoverflow.com/questions/38079853/how-can-i-implement-incremental-training-for-xgboost , (2) https://datascience.stackexchange.com/questions/25348/how-to-train-a-xgboost-model-on-data-that-is-too-big-for-the-memory , (3) https://github.com/dmlc/xgboost/issues/3055

     – Ben Reiniger Mar 19 '19 at 11:50
  • It seems train Data_A incrementally -> train Data_B incrementally isn't equal to train (Data_A + Data_B) ? – code_worker Mar 20 '19 at 05:02
  • @code_worker Correct. – Ben Reiniger Mar 20 '19 at 11:39
  • Is there any algo/tool can reach Data_A incrementally -> train Data_B incrementally is equal to train (Data_A + Data_B)?? – code_worker Mar 21 '19 at 09:53
  • 1
    That's probably better as a new question. – Ben Reiniger Mar 21 '19 at 13:33
  • I would be really interested in the question asked by @code_worker. Did you open a new question? If yes, can you link it here? – Mattia Surricchio May 02 '21 at 22:15
  • @MattiaSurricchio Nope, cuz I'm busy for another domain know. – code_worker May 03 '21 at 06:30