Boosting in Machine learning

When comparing in bagging, in bagging we used high variance and low bias as our base models. Then we applied randomisation techniques like column sampling ,row sampling and then we used aggregation of base learners.

In the case of boosting we use our base models as high bias and low variance models as our base models. In order to reduce bias we use the concept called additive combining of the base models.

High bias means we are generalising the output. that means we expect to have large training error.

Idea how boosting reduces bias:

First we split the entire data to training data and testing data. We take the training data. Suppose the the training data contains points {xi,yi} where xi’s are the features and yi are the class labels.

Stage -0: We build a model called ‘Model0' on the complete training data say ‘D-train’. As we assumed in the boosting that the model should have high bias and low variance.

Say ‘Model-0’ generates a function h0(x) when seeing the training data.

y=h0(x). Training error = actual class label value (-) predicted value

Now we have { training feature points , actual class label, error}for all the points in training data.

Stage -1: We again train a model on the training data={training feature points, error} not on the actual class label.

the model learns a function h1(x) which predicts the error

The output of stage 1 = α0*h0(x)+α1*h1(x).

Similarly we compute the error in stage 1:

actual class label (-) predicted value of model in stage 1

Similarly we built model 2 with {feature points ,error in the 1st stage}.

Output of stage 2=α0*h0(x)+α1*h1(x)+α2*h2(x). where h2 computes error in the 1st stage.

Similarly for stage k the function would be like:

output= α0*h0(x)+α1*h1(x)+α2*h2(x)+…….+αk*hk(x).

each of the model is used to train the residual error at the end of previous stage.

The output at K model has less training error. This implies we have less bias at the end of K level.

Gradient boosted decision tree comes under boosting technique.