Polynomial Features and Regularization

By the end of this lesson, you will be able to...

• explain how adding polynomial features lets a linear model represent a curved pattern;

• calculate a simple L2-regularized objective and describe what its penalty prefers;

• use training and validation evidence to distinguish a useful curve from underfitting or overfitting.

Idea in one sentence: Polynomial features give a model more shapes it can express, while regularization makes each extra-flexible fit earn its complexity with better evidence on new data.

Core Insight

The support team from the previous lessons wants to estimate a student's week-four quiz score from completed practice sessions. A straight line is an honest first model: every additional session changes the prediction by the same amount.

But the small historical pattern suggests something else. Early sessions appear to help more than later sessions. A line may miss that bend.

The obvious correction is to make the model more flexible. Add x², fit the training rows more closely, and celebrate the lower loss.

That correction works only halfway. A flexible model can capture a real curve; it can also use its extra freedom to chase four noisy points. The stronger model is not “always use a more complicated equation.” It is:

choose a representation that can express the plausible pattern
-> constrain needless complexity during fitting
-> check the result on examples that did not choose the fit

Polynomial features and regularization are two parts of that sequence. One enlarges the model's vocabulary; the other makes extreme use of that vocabulary expensive.

A Pattern a Straight Line Cannot State

Suppose these toy rows come from earlier cohorts. They are illustrative, not a causal study of practice.

The score rises, but the gains get smaller. From zero to one session, the score rises by 14; from three to four, it rises by 2.

A one-feature line has this form:

ŷ = b + w₁x

For a fixed w₁, every additional session changes the prediction by the same amount. That is a useful assumption when the relationship is roughly straight. Here it is too rigid: a line can tilt up, but it cannot bend toward a plateau.

The first model did not fail because “linear regression is bad.” It failed because its representation gave it only one building block: x. The missing building block is a way for the effect of x to change as x grows.

Add a Feature, Not a New Training Loop

Create a second feature from the first one:

x        = practice sessions
x²       = practice sessions squared

Now fit:

ŷ = b + w₁x + w₂x²

This is a polynomial model of degree two. It is curved as a function of x, but it is still linear in its parameters: the optimizer adjusts b, w₁, and w₂ by adding weighted feature contributions, just as in lesson 002. Gradient descent from lesson 003 can still minimize its loss.

Consider this illustrative model:

ŷ = 40 + 16x − 2x²

The negative w₂ bends the curve downward. It does not say that practice eventually harms learning in the real world. It says that, within this simplified model and this range of inputs, the incremental predicted gain gets smaller.

That distinction matters. A model coefficient describes the function fitted to its data. It does not prove a causal mechanism, and it should not be trusted far outside the observed range. At x = 10, this equation predicts 0; that is a warning about extrapolating a toy polynomial, not a credible educational claim.

More Capacity Creates Two Different Failures

Adding x² is a change in capacity: the set of patterns the model can represent becomes larger.

Capacity helps only when it matches signal that persists beyond the training rows. Think about three model choices:

degree 1: one straight line
degree 2: one smooth bend
degree 10: many possible turns and wiggles

• A degree-1 model can underfit: it has high error even on training examples because it cannot describe the main bend.
• A degree-10 model can overfit: it can drive training error very low by bending around accidental variation that will not repeat.
• A degree-2 model may be a useful middle candidate, but only validation evidence can support that claim.

The names describe evidence, not just visual style. A model is underfit when it does poorly even on the data it is allowed to learn from. A model is overfit when its training performance looks much better than its performance on relevant unseen examples.

training error alone cannot distinguish
"I learned a repeatable curve"
from
"I memorized quirks in these rows."

That is why “lower training loss” is not the selection rule. It is only one observation in the selection process.

A Penalty That Makes Large Coefficients Earn Their Place

Suppose the model uses squared error, as in lesson 002. One common regularized objective adds an L2 penalty:

objective = training MSE + λ(w₁² + w₂²)

Read each part literally:

• training MSE rewards predictions close to training labels;
• w₁² + w₂² is a simple measure of coefficient magnitude in this two-feature example;
• λ (lambda) is the regularization rate, chosen before fitting; it controls how strongly large coefficients matter.

For clarity, this teaching objective does not penalize the intercept b. Many practical implementations make that same choice, but details vary by method and library.

Now compare two invented candidate fits on the same training sample:

With λ = 0.01:

curved objective = 1.0 + 0.01(260) = 3.60
simple objective = 3.0 + 0.01(144) = 4.44

The curved candidate wins this objective. Its better fit earns the extra penalty.

With λ = 0.05:

curved objective = 1.0 + 0.05(260) = 14.00
simple objective = 3.0 + 0.05(144) = 10.20

Now the simple candidate wins. A stronger penalty makes large coefficients expensive enough to reject the additional bend.

This is what regularization does: it does not remove the model's ability to curve; it changes which parameter values the optimizer prefers. For L2 regularization, coefficients are encouraged toward zero, not generally set to exactly zero.

The Penalty Is a Preference, Not a Verdict

Trade-off: Polynomial features can represent a real curved relationship; regularization can reduce fragile, extreme fits. This costs bias: a penalty that is too strong can flatten a useful curve and increase both training and validation error. It can still fail when the transformed features are irrelevant, the validation split is unrepresentative, or the loss ignores the decision's actual costs. The signals to watch are training and validation behavior together, plus coefficient magnitudes and predictions near the edge of the observed range.

λ = 0 means no penalty. Raising λ strengthens the preference for smaller coefficients. Neither extreme is automatically correct. The appropriate regularization rate is data-dependent and must be selected without using the final test set as a tuning surface.

Regularization also does not make a feature valid. It cannot repair leakage, a missing label definition, or a prediction that arrives too late. It only changes the optimization preference among the models the representation already makes possible.

A Small Model-Selection Checkpoint

The support team now evaluates three candidates on a held-out validation set. These MSE values are synthetic and exist only to make the diagnosis concrete.

Read the table in order:

1. A has similarly high training and validation error. It likely lacks capacity for the main pattern, so it is a plausible underfit candidate.
2. C nearly memorizes training rows but fails badly on validation rows. The large gap is evidence consistent with overfitting.
3. D restrains the same high-degree representation. It reduces the gap, but it still validates worse than B in this toy comparison.
4. B has the best validation MSE here. Under the assumption that the validation set represents intended future use and MSE matches the goal, B is the best current choice.

Notice what this conclusion does not say: B is not the universally best model, its coefficient effects are not causal, and it has not passed a final untouched test. It is simply the best-supported next candidate under the present evidence.

So far, the first four lessons form one loop:

define a prediction contract
-> choose a representation and loss
-> optimize its parameters
-> compare capacity and restraint on unseen evidence

Check Your Understanding

Check 1: In ŷ = 40 + 16x − 2x², what is the predicted score for x = 3?

Think first, then reveal.

Answer: 40 + 16(3) − 2(9) = 70. The squared feature lets the contribution change as x grows.

Check 2: A degree-6 model has training MSE 0.2 and validation MSE 25, while a degree-2 model has 6 and 7. Which evidence most strongly suggests overfitting?

Think first, then reveal.

Answer: The degree-6 model's large training–validation gap. Its very low training error did not carry to held-out examples. The degree-2 model is the stronger current candidate if the validation setup is representative.

Check 3: If λ increases in an L2-regularized objective, must validation performance improve?

Think first, then reveal.

Answer: No. A larger penalty shrinks coefficients more strongly, which can reduce overfitting or can oversimplify the model. Validation evidence decides whether the change helped.

Practice: Choose the Next Experiment

A different tutoring program predicts a final project score from completed practice modules. It compares these models using the same validation process:

Write a two-sentence recommendation. A good answer should:

• identify the strongest current candidate and cite the relevant evidence;
• name the likely failure of the other two candidates;
• state one boundary or next check before deployment.

Model answer: The quadratic with moderate L2 is the strongest current candidate because it has the lowest validation MSE (9) without the large gap shown by the degree-8 model. The line likely underfits, while degree 8 likely overfits; before deployment, verify that the validation split represents future students and inspect predictions at the range boundaries.

Connections and Next Step

Polynomial features show that a “linear” model can become nonlinear in its inputs through representation. Regularization shows how the same optimization loop can prefer a less extreme fit. Together they replace the rule “minimize training loss” with a more disciplined model-selection question.

The next lesson changes the target type. Logistic Regression Fundamentals starts with a linear score too, but turns it into a class probability and separates probability estimation from threshold-based decisions.

Resources

• [TUTORIAL] Google Machine Learning Crash Course: Model complexity — Focus: why fitting data and keeping a model simple are competing goals.
• [TUTORIAL] Google Machine Learning Crash Course: L2 regularization — Focus: coefficient penalties, the regularization rate, and why stronger L2 pushes weights toward zero.
• [TUTORIAL] Google Machine Learning Crash Course: Overfitting — Focus: interpreting training versus validation behavior as evidence about generalization.

Key Takeaways

1. Adding x² changes what a model can represent while preserving the familiar weighted-sum training structure.
2. Underfitting and overfitting are diagnosed from training and unseen-data evidence, not from model degree alone.
3. L2 regularization adds a price for large coefficients; λ controls how strongly that price changes the fitted model.
4. Choose capacity and regularization from representative validation evidence, then keep final test evidence untouched.