Overfitting, Underfitting, and Regularization

By the end of this lesson, you will be able to...

• diagnose a plausible underfitting or overfitting pattern from training and validation evidence;

• explain how a regularization penalty changes which fitted solution the learner prefers;

• choose a next experiment without confusing regularization with a fix for leakage, poor labels, or an invalid split.

Idea in one sentence: A model generalizes when it learns structure that survives new data, and regularization is one way to make fragile explanations less attractive during fitting.

Core Insight

Consider an online retailer predicting whether an order will be returned. The model scores an order at checkout so the support team can offer help before shipment. The team has already written a feature contract and uses chronological development folds, so each score describes a plausible past-to-future rehearsal.

An engineer adds product-category indicators, customer-history features, and many interaction features. Training recall at the fixed support capacity rises from 0.81 to 0.94. That looks like a clear win.

The held-out folds tell a different story: recall rises only from 0.76 to 0.71. The additional training fit did not travel. The model may be responding to accidental details in the development periods—rare product codes, noisy interactions, or chance combinations that are unlikely to recur.

The first model, “the largest training improvement is the best improvement,” is reasonable when the training sample is a faithful miniature of everything the model will see. It breaks because a finite sample also contains quirks. The goal is not to erase every training error. It is to choose a rule whose useful patterns survive beyond the rows that taught it.

The Confusion This Concept Solves

Training and validation scores answer different questions.

Plain meaning: a model can become very good at the worksheet it practiced without becoming good at the next worksheet.

In this scenario: a rare product-code interaction may predict returns in the January–June orders that contain it, but have no useful effect in July–August orders.

Technical name: when a model fits sample-specific patterns that do not transfer, it is overfitting. When it cannot capture useful structure even in its training data, it is underfitting.

These labels are diagnoses, not conclusions drawn from one number. They only have meaning after the feature pipeline and split have respected the prediction-time boundary. A high training-validation gap caused by a feature fitted on all rows is leakage, not evidence that a penalty is needed. The prior lesson supplies that guardrail.

A Small Comparison

The retailer compares three logistic models on the same chronological folds. The scores below are illustrative recall at the team’s fixed daily capacity, averaged across those folds.

Candidate A may be underfitting. Its simple representation or capacity could be missing a real distinction, such as how return risk changes jointly with item type and delivery promise. Candidate B has a recognizable overfitting pattern: it learns much more from the training rows than can be reused later. Candidate C is the useful correction. It accepts a worse training score in exchange for a better validation score.

This table does not prove the cause of every gap. Fold-to-fold variation, label noise, a recent catalog change, or an overly small validation period can also affect the result. But it gives the team a specific next question instead of a ritual: which change improves held-out performance under the fixed evaluation contract?

The Precise Meaning of Regularization

For a linear or logistic model, fitting normally minimizes a data-loss term: predictions that disagree with observed labels cost more. Regularization adds a second preference.

objective = data loss + λ × penalty on the coefficients

The first term asks, “which coefficients explain the training examples?” The second asks, “among similarly useful explanations, which is less extreme according to this penalty?” The parameter λ controls how much the second preference matters. Different libraries use different parameter names and scales, so treat the equation as a teaching model rather than a copy-paste configuration rule.

With an L2 penalty, large coefficients cost more because the penalty includes their squares:

coefficients [5, 4]  -> L2 penalty = 5² + 4² = 41
coefficients [1, 1]  -> L2 penalty = 1² + 1² =  2

These toy values do not say [1, 1] is automatically the better model; it may fit the data far worse. They show the pressure the objective adds. A feature must earn a very large coefficient by improving data fit enough to offset its larger penalty.

L1 regularization applies a penalty based on absolute coefficient size and can drive some coefficients exactly to zero. That can produce a sparse model, meaning fewer active features. L2 usually shrinks coefficients without necessarily removing them. Both are ways to express restraint; neither makes a feature trustworthy or a causal explanation true.

For iterative learners, the same idea can appear as an early stopping rule. Gradient boosting, for example, adds trees sequentially. Stop at the iteration chosen by validation evidence rather than always accepting later corrections that improve training loss. That is a different mechanism from an L1 or L2 penalty, but the shared aim is to limit sample-specific refinement.

A Worked Decision Path

The team keeps candidate B’s feature set but tests increasing levels of L2-style restraint. The numbers are illustrative and the threshold remains fixed by the support-capacity contract.

Starting point: weak restraint lets the many interaction coefficients respond closely to training rows.

Intermediate change: moderate restraint reduces the incentive for a rare interaction to dominate unless it repeatedly helps the data-loss term. Training recall falls. This is expected: the model is giving up some ability to match its practice rows.

Result: validation recall rises. Under this evaluation design, that is evidence that the less extreme solution travels better to later orders.

Boundary: very strong restraint lowers both scores. The model now has too little effective flexibility to represent genuine return-risk differences. More regularization is not “safer” once it begins to cause underfitting.

So far, we have seen that regularization is a trade-off between matching the training sample and preferring a less fragile fitted rule. The useful setting is selected with validation evidence, not by assuming that smaller coefficients are morally better.

What This Is Not

Confusion: Every weak validation score means overfitting.

Why it is tempting: Validation performance is disappointing, and regularization is a familiar response.

Better model: If both training and validation scores are low, stronger restraint often makes the model less able to learn useful structure. Inspect representation, capacity, label quality, and the chosen metric before adding a penalty.

Confusion: Regularization repairs leakage.

Why it is tempting: A penalty can shrink the apparent impact of an overly predictive feature.

Better model: A future-derived feature remains invalid even if its coefficient is small. Remove or redefine it, then refit each transformation inside the training portion of every fold.

Confusion: A sparse model has discovered the only important features.

Why it is tempting: L1 can set coefficients to zero, which looks like an explanation.

Better model: correlated or differently scaled features can share or swap coefficient weight. Feature selection is a modeling choice under a particular dataset and penalty, not a proof of causation or fairness.

Consequences, Trade-offs, and Limits

Regularization can improve held-out performance when the model has enough flexibility to chase details that are not stable. It costs training fit and can make a model miss real but subtle relationships when applied too strongly. It also adds a parameter to tune, which means more development experiments and a greater need to preserve the final test period.

It does not solve a target that measures the wrong decision, labels that arrive with systematic error, feature availability problems, or a future that differs sharply from the data used for development. Those are different failure modes.

Three signals tell the team what to inspect:

• Training score is high and validation remains lower across credible folds: test a less flexible model, stronger restraint, or a cleaner representation.
• Training and validation are both low and close: test a missing feature relationship, a different model family, or whether the target and metric are workable.
• A chosen regularization setting wins only in one fold or disappears in the final future test: inspect fold variation and data shift before announcing a general rule.

The next lesson turns these patterns into learning curves: instead of comparing one training size, it will examine how training and validation change as the amount of evidence grows.

Check Your Understanding

Check: A model has training recall of 0.96 and validation recall of 0.69 after a credible time-aware split. The team then removes a leaked post-delivery feature, and both scores become 0.74. Should it add stronger regularization first?

Think first, then reveal.

Answer: Not from these numbers alone. The original gap was partly an invalid feature, not necessarily excess model flexibility. After removing leakage, the team should rerun the complete comparison and diagnose the new pattern. If training and validation are both near 0.74, stronger regularization is more likely to reduce useful capacity than to fix the main problem.

Check: Why can a model with lower training recall be the better candidate?

Think first, then reveal.

Answer: The decision is about future or held-out performance under the agreed metric and split. A model that gives up some training fit but improves validation recall is showing evidence of better transfer, subject to checking fold variation and a final untouched test.

Practice: Choose the Next Experiment

A subscription service uses a valid chronological evaluation and a complete pipeline. Its current model has training AUC of 0.67 and validation AUC of 0.65. A teammate proposes much stronger L2 regularization because “regularization always helps generalization.”

What should the team test first, and why?

Model answer: The scores are both low and close, so the immediate pattern is more consistent with underfitting than with a large generalization gap. First inspect whether the representation misses useful relationships, whether the model family is too limited, and whether AUC matches the decision. A small regularization sweep can be part of a controlled comparison, but much stronger regularization is not the leading hypothesis because it reduces effective flexibility. Keep the feature and split contracts fixed while testing so the results remain interpretable.

Resources

• [DOCUMENTATION] scikit-learn: Linear models — Focus: compare coefficient penalties in Ridge, Lasso, and Elastic Net.
• [DOCUMENTATION] scikit-learn: Validation curves — Focus: connect model-complexity choices to training and validation scores.
• [DOCUMENTATION] scikit-learn: Cross-validation — Focus: keep regularization selection inside a valid development loop and preserve a final test set.

Key Takeaways

• Overfitting means training-specific fit fails to transfer; underfitting means useful structure is not captured even in training data.
• These diagnoses require a valid feature pipeline and split; regularization cannot repair leakage or an invalid evaluation design.
• A penalty changes which training solutions are preferred by charging for coefficient size or other forms of complexity.
• Select the amount of restraint from held-out evidence and its variability, not from the training score or a universal preference for simpler models.