Random Forest and Bagging

By the end of this lesson, you will be able to...

• trace how bootstrap samples, tree diversity, and a vote produce a random-forest prediction;

• explain why averaging helps an unstable tree only when its errors are not too similar;

• distinguish a useful out-of-bag estimate from evidence that the target, features, and decision are trustworthy.

Idea in one sentence: A random forest makes many different versions of a fragile decision tree, then lets their partly independent mistakes cancel through aggregation.

Core Insight

A subscription service wants to offer help before a customer cancels. One decision tree gives an appealing rule: “if days inactive exceed seven, flag the account.” The rule fits this month's training data well. But next month, a few active customers with failed payments cancel, while several inactive customers return after a holiday. A small change in the training rows could have made the tree split on failed payment first instead.

The initial model says: find the one best tree and trust its path. That is attractive because a tree can express nonlinear rules and show one readable explanation. It works when the training sample is large and stable enough that small changes do not alter important splits. It becomes insufficient when several nearly plausible splits compete, a few rows are unusual, or a dominant feature happens to look stronger than it is.

Bagging gives the tree a different job. Instead of asking one tree to be perfectly stable, create many trees from slightly different samples and aggregate their predictions. A random forest adds a second source of difference: each split considers only a random subset of features. The goal is not a large crowd for its own sake. The goal is a crowd whose errors do not all point in the same direction.

The Small Situation: One Rule Moves Too Easily

For the teaching trace, assume the service has six labeled training accounts. The values and outcomes below are illustrative, not observed customer data.

Noor is a new account with 11 inactive days, a failed payment, and one support ticket. A single tree could split on inactivity first and flag Noor. Another reasonable tree could split on failed payment first and also flag Noor. A third could give more weight to support tickets and not flag Noor.

None of these trees is irrational. Each sees a small dataset where several rules can reduce training error. The pressure is that a single chosen tree converts an accidental preference among those rules into the whole model's prediction. We need a way to preserve useful nonlinear rules without letting one sample's quirks decide everything.

Bagging: Resample, Fit, Aggregate

Plain meaning: make several versions of the training set by drawing rows with replacement, fit one tree to each version, and combine their outputs.

In this scenario: a bootstrap sample contains the same number of draw positions as the original training set, but an account can appear more than once and another account may not appear at all.

Technical name: this is bootstrap aggregating, usually shortened to bagging.

Here are three small bootstrap samples. They are deliberately tiny so that the sampling is visible.

Each tree sees a related but not identical problem. Tree A sees account 2 twice and never sees account 3. Tree B sees the failed-payment example twice. Tree C gives extra weight to accounts 2 and 6. A deep-enough tree can still react strongly to the rows it sees, but a particular noisy row is less likely to dictate every tree.

For classification, the final forest commonly uses a majority vote. For regression, it commonly averages numeric predictions. This aggregation is the part that reduces instability: an idiosyncratic split that appears in only some trees has less influence on the final result than a pattern that appears across many resamples.

Why a Forest Adds Feature Randomness

Bagging alone can leave a problem. Suppose days inactive is a strong-looking feature. Many bootstrapped trees may place it at the root. They are different trees on paper, but their predictions still move together because they rely on the same first decision. Averaging highly correlated mistakes does not remove much error.

A random forest limits the features a tree can inspect at each candidate split. The subset is redrawn at each split, not once for the whole tree. In the tiny trace below, the possible splits are simplified. Real tree training evaluates thresholds and impurity reductions rather than following this hand-written rule.

The final vote is now inspectable:

Tree A: yes
Tree B: yes
Tree C: no

yes votes: 2
no votes:  1
forest prediction: yes

The feature subsets did not make all trees equally good. They made them less likely to repeat exactly the same reasoning. That is the stronger model:

bootstrap samples make trees react to different row histories
feature subsets make trees search different split opportunities
aggregation gives repeated patterns more influence than isolated quirks

So far, we have seen that the forest does not discover one superior root-to-leaf story for Noor. It combines multiple conditional stories. This matters because a forest's stability comes from diversity plus aggregation, not merely from its number of trees.

What the Vote Changes—and What It Does Not

Before bagging, a team may try to stabilize a tree by pruning it aggressively until its branches are simple. After bagging, it can often use more variable trees and stabilize their predictions through averaging. This typically reduces the variance-like part of the problem: how much the fitted predictor changes when the training sample changes.

But averaging does not manufacture missing information. If cancellation labels are wrong, if a feature is unavailable when the offer must be made, or if the target definition mixes voluntary cancellation with payment failures, every tree can inherit the same problem. The forest may be stable and still answer the wrong question.

That distinction is especially important for feature importance summaries. An importance value may indicate that a feature was useful to the forest's splits under this data and this fitting procedure. It is not a causal claim, a fairness finding, or a complete explanation of why Noor received one prediction. To review one case, inspect its inputs, the model's output, the decision threshold, and the available evidence rather than treating a global ranking as a case narrative.

The central trade-off is clear:

one tree      -> one readable path, but a fragile prediction
many trees    -> a steadier aggregate, but no single complete path to narrate

More trees also cost training time, memory, and prediction work. Restricting candidate features can make trees more diverse, but if the subset is too small, many trees may miss a genuinely useful predictor and become individually weak. The useful setting is a preference under constraints: choose the number of trees, their depth, and the feature subset through validation with the real error costs, latency budget, and explanation needs in view.

Use Out-of-Bag Cases as a Check, Not a Shortcut

Look again at the bootstrap table. Account 3 was omitted from Tree A's sample. Tree A can predict account 3 without having trained on it. Such omitted rows are called out-of-bag (OOB) for that tree.

For a larger forest, an account has predictions from the trees whose bootstrap samples omitted it. Aggregating only those predictions produces an OOB estimate for that account. Repeating this across training rows gives a convenient internal check while fitting the forest.

This is useful evidence, not a replacement for an evaluation design. OOB predictions can help compare a limited set of forest settings or reveal that a model is unstable. They do not repair leakage in the features, account for a future time shift, choose a business threshold, or replace a final test set reserved from tuning. If the service has multiple rows for the same customer or a changing cancellation policy over time, lesson 014's grouped or time-aware validation design is still needed.

The signal to watch is disagreement between the reassuring OOB estimate and an evaluation split that actually resembles deployment. That disagreement asks a concrete question: are the resampled rows too similar to the held-out future cases, or has the representation leaked information? Adding more trees is not the answer until that evidence is understood.

Trace It Yourself

Check: Tree D sees a bootstrap sample in which account 5 appears twice and account 1 is absent. Is Tree D trained on seven distinct accounts?

Think first, then reveal.

Answer: No. A bootstrap sample draws positions with replacement. Repeating account 5 increases its presence in that tree's fitting data; omitting account 1 leaves it available as an out-of-bag case for Tree D. The sample has the original number of draw positions, not necessarily the original number of distinct accounts.

Check: Every tree in a forest selects days inactive at its root and makes the same mistake for accounts after a holiday. Why is this a problem even if the forest has 1,000 trees?

Think first, then reveal.

Answer: The trees' errors are highly correlated. Voting cannot cancel an error that nearly every voter repeats. Investigate the feature representation, labels, and feature-subset setting; additional copies of the same error are not independent evidence.

Practice: Defend a Forest Choice

A retention team compares a single shallow tree with a random forest. The forest has a better OOB score and better validation recall, but it generates 30% more false-positive outreach offers. Those offers are costly and annoy customers. The forest's feature-importance chart ranks a recent-support-contact feature first.

What should the team do next?

Model answer: Do not choose the forest merely because its OOB score is higher or because one feature ranks first. Compare both models at decision thresholds that reflect the cost of unwanted offers and missed cancellations. Inspect false positives by customer segment and timing, and check whether recent support contact is available before the offer and represents a useful signal rather than a policy artifact. If the forest remains preferable under those constraints, document that it trades a simple per-case rule for lower variance and a more complex explanation. If the extra false positives are unacceptable, adjust the threshold or choose a different model; adding trees cannot resolve an error-cost decision.

Resources

• [DOCUMENTATION] scikit-learn: Ensemble methods — Focus: compare bagging, random forests, feature randomness, and out-of-bag evaluation terminology.
• [PAPER] Random Forests — Focus: read the original account of random feature selection, ensemble correlation, and generalization error.
• [DOCUMENTATION] scikit-learn: RandomForestClassifier — Focus: connect bootstrap sampling, max_features, and OOB scoring to the model's behavior.

Key Takeaways

• Bagging fits many trees to bootstrap samples and aggregates their outputs so that sample-specific quirks have less influence.
• A random forest adds random feature subsets at each split to reduce correlation between trees; diversity, not sheer quantity, makes voting valuable.
• OOB predictions are a useful fitting-time check, but they do not replace leakage checks, future-like validation, or a cost-aware decision threshold.
• A forest can reduce instability without fixing a weak target, unavailable feature, biased label, or costly decision policy.