Gradient Boosting and Sequential Correction

By the end of this lesson, you will be able to...

• trace how successive small trees change an ensemble prediction by correcting current error;

• explain how learning rate, tree complexity, and number of stages control the size and number of corrections;

• contrast sequential boosting with the independent voting used by a random forest.

Idea in one sentence: Gradient boosting builds a predictor as a sequence of small corrections, so each new tree must respond to the mistakes already made by the earlier trees.

Core Insight

Consider a subscription service that has tried a random forest to flag accounts likely to cancel. Its many trees vote independently, which helps when one tree is unstable. Yet the error review shows a pattern the vote does not repair well: accounts with a failed payment but recent activity are often missed. The forest can contain trees that notice this pattern, but it does not make later trees concentrate on it. Each tree is trained as another independent voter.

A tempting next move is “add more trees.” That works for a forest only when more diverse votes reduce instability. Here the pressure is different: the current predictor has a visible, systematic leftover error. We want the next learner to look specifically at what the current ensemble still gets wrong.

Gradient boosting does that. It starts with a simple prediction, measures the remaining error under a chosen loss, fits a small tree to a correction direction, and adds only a scaled part of that correction. The trees are not a committee. They are a sequence: tree two is meaningful only because tree one has already changed the prediction.

A Small Score That Leaves Useful Errors Behind

Use a deliberately simplified teaching model. Each account has an outcome y: 1 means it later cancelled and 0 means it did not. We will use squared-error residuals, y − prediction, because the arithmetic is easy to inspect. Real binary gradient boosting normally optimizes a classification loss such as log loss and transforms an additive score into a probability-like output; this table is not a production probability calculation.

The base prediction F₀ = 0.50 is deliberately plain: before seeing features, predict the same midpoint for everyone. It is not a recommendation to deploy 0.50 as a cancellation threshold. Lesson 008 established that a score, a threshold, and a business action are separate choices.

The residual column is the evidence that earns the next step. Accounts B and D need their scores raised; account A needs its score lowered. More interestingly, C is a cancellation that the base model does not distinguish from A, and E is an inactive account that does not cancel. A single broad correction may help one pattern while leaving another unresolved.

Add a Small Correction, Then Look Again

Plain meaning: do not replace the current model. Add a small rule that moves its predictions toward the outcomes it still misses.

In this scenario: the first shallow tree notices that inactivity of seven or more days is associated with positive residuals overall. Its teaching output is +0.20 for those accounts and -0.20 otherwise.

Technical name: in gradient boosting, the learner at each stage is fit to a quantity derived from the current loss—often described as a residual in this simplified picture or a negative gradient in the general formulation.

Let the learning rate be 0.5. The first tree contributes only half of its proposed correction:

F₁(x) = F₀(x) + 0.5 × tree₁(x)

The first correction helped B and D. It made C worse because C was active recently, and it made E worse because inactivity alone is not enough to explain cancellation. That is not a reason to discard the ensemble. It is the information used by the next stage.

Tree 2 now sees the new residuals, not the original labels in isolation. A small tree can express two remaining patterns: raise the score for a failed payment and lower it for a very inactive account without a failed payment. For this teaching trace, its outputs are +0.60 for C, -0.50 for E, and 0 for the other accounts.

The update sequence is the mechanism:

base score F₀
  -> tree 1 corrects a broad inactivity pattern
  -> inspect the new residuals
  -> tree 2 corrects failed payment and the E-like exception
  -> inspect again before adding another tree

The final score after two stages is the sum of the base score and both scaled contributions. There is no majority vote, no bootstrap crowd, and no later tree overriding an earlier tree. Each stage adjusts the same additive predictor.

So far, we have seen that boosting earns its extra complexity by concentrating capacity on remaining error. This matters because a later tree can be useful even if it would be a poor standalone classifier: its job is narrow correction, not a complete decision rule.

Learning Rate Is a Brake, Not a Guarantee

What if the learning rate were 1.0 instead of 0.5? Tree 2 would move C from 0.40 to 1.00 and E from 0.60 to 0.10 in this toy trace. Those stronger moves reduce the shown residuals more quickly, but they also commit more strongly to a pattern inferred from a small sample. With noisy labels or an unrepresentative split, a large correction can chase a quirk rather than a repeatable relationship.

With a smaller learning rate, each tree changes the ensemble less. The model usually needs more stages to express the same amount of structure. This is the useful trade-off:

larger learning rate -> fewer, stronger corrections; faster fitting; easier to overreact
smaller learning rate -> more, gentler corrections; more training work; finer control

Neither setting wins by definition. Tree depth or leaf count matters too. A shallow tree makes a broad correction, such as “recently inactive.” A deeper tree can isolate an account pattern more precisely, but it can also fit a small subgroup or a label mistake. The number of estimators controls how long the sequence is allowed to continue. These choices must be tuned together on validation data that matches the future decision, not selected from training loss alone.

The important boundary is that boosting does not discover whether a correction is meaningful outside the training data. A falling training loss says the additive model has described its training examples more closely. It does not establish that a feature is available at prediction time, that a label is correct, or that a threshold produces acceptable false-positive and false-negative costs.

Stop When New Corrections Stop Helping the Decision

Because each stage can improve the training objective, it is tempting to keep adding trees until the training curve looks excellent. That is the wrong stopping signal. Monitor a validation metric that reflects the actual decision and its error costs. If validation performance stops improving while training performance keeps improving, later corrections may be learning sample-specific noise.

Some implementations offer early stopping, but it is a procedure, not a magic property of boosting. It still depends on a valid validation partition, a chosen metric, patience rules, and no leakage through repeated tuning. For time-dependent cancellation data, a random validation split can make early stopping look more reassuring than it should; lesson 014 will make that validation choice explicit.

Compare the two ensemble mechanisms directly:

This comparison is a teaching model, not an algorithm-selection rule. A random forest can perform well on a problem with residual structure; boosting can be stable with careful settings. Choose an experiment based on the representation, data size, required latency, error costs, and evidence from held-out cases.

Trace It Yourself

Check: After Tree 1, account C has F₁ = 0.40 and target y = 1. Tree 2 proposes an output of +0.60. With a learning rate of 0.5, what is C's new score?

Think first, then reveal.

Answer: The scaled correction is 0.5 × 0.60 = 0.30, so the new score is 0.40 + 0.30 = 0.70. Tree 2 does not replace the earlier model; it adds a scaled correction to it.

Check: A team fits 800 boosting stages and obtains near-perfect training accuracy, but validation recall falls after stage 230. Which evidence should guide the next decision?

Think first, then reveal.

Answer: Use the validation evidence around the best stage, together with the stated error costs, rather than the near-perfect training result. The divergence suggests later stages are improving fit to the training sample without improving the future-like cases represented by validation.

Practice: Choose the Next Boosting Experiment

A claims-review team predicts whether a case needs manual review. A boosted model with deep trees and a high learning rate has excellent training loss, but on a time-ordered validation set it raises false positives for a newly introduced claim category. A colleague proposes adding 500 more estimators.

What is a stronger next experiment?

Model answer: First inspect whether the new category exists in the training period and whether its features are defined consistently across time. The validation pattern may be a representation or distribution-shift problem, not a shortage of correction stages. Compare a shallower-tree, lower-learning-rate candidate with early stopping on a time-aware validation split, using a metric that reflects the cost of unnecessary manual review. If the category is genuinely new, collect representative data or define a safe fallback rather than treating extra trees as evidence. More stages can intensify a mistaken pattern; they cannot create historical support for a new category.

Resources

• [PAPER] Greedy Function Approximation: A Gradient Boosting Machine — Focus: see the original additive, loss-reduction view of gradient boosting.
• [DOCUMENTATION] scikit-learn: Ensemble methods — Focus: connect gradient-boosting stages, loss functions, and tree controls to the mechanism traced here.
• [DOCUMENTATION] scikit-learn: GradientBoostingClassifier — Focus: inspect the relationship between learning rate, number of estimators, and staged predictions.

Key Takeaways

• Gradient boosting is an additive sequence: each tree makes a scaled correction to the predictor built so far.
• The next tree learns from current error, which is why boosting is different from a random forest's independent vote.
• Learning rate, tree complexity, and number of stages jointly control how aggressively the model follows the training data.
• Validation evidence and error costs, not training loss alone, determine when another correction is worthwhile.