Linear Regression Fundamentals

By the end of this lesson, you will be able to...

• turn one numerical feature into a linear prediction with a baseline and a weight;

• calculate residuals and compare two candidate lines using squared error;

• explain what a lower training loss establishes—and what it does not establish.

Idea in one sentence: Linear regression makes a numerical prediction by adding a baseline to weighted inputs, then judges candidate parameter values by how far their predictions are from observed labels.

Core Insight

The previous lesson asked the support team to define an honest prediction contract. Suppose it has now done so: at the end of week three, it wants to estimate each student's score on the week-four quiz so it can offer optional study support before the quiz.

To keep the first model inspectable, use one feature: the number of practice sessions a student completed by Friday. The label is the later quiz score. A model has to turn the feature into a number.

The simplest useful rule is a line:

predicted quiz score = baseline + weight × practice sessions

The rule is not useful merely because it can output a number. We also need a way to compare one proposed line with another. That comparison is the small loop underneath much of supervised learning:

examples -> prediction rule -> predicted labels -> residuals -> loss -> choose better parameters

This lesson makes every arrow visible. The following lesson will ask how an algorithm searches for better parameters. For now, we will calculate what “better” means.

Start with a Baseline, Then Notice What It Misses

Assume a small historical sample produced these records. The values are invented for teaching; they are not evidence about real students.

One perfectly reasonable first guess is the cohort's average score: predict 60 for everyone. That is a baseline: a simple reference rule that does not use the feature.

It is better than guessing blindly, but it misses the visible pattern. Students with more completed practice sessions tend to have higher scores in this tiny sample. A line lets the prediction respond to that input instead of making the same estimate for every student.

Do not overread the pattern. Four rows cannot prove that practice sessions cause a higher score. The feature could stand in for prior knowledge, available time, or other factors. At this stage, we are building a predictive approximation, not a causal explanation.

The Smallest Linear Model

With one numerical feature, write a linear regression model as:

ŷ = b + wx

Read the symbols as follows:

• x is the feature value for one example: completed practice sessions.
• ŷ (“y hat”) is the model's predicted label: predicted quiz score.
• b is the intercept or baseline prediction when x = 0.
• w is the weight: how much the prediction changes when x increases by one unit.

Consider this candidate rule:

ŷ = 48 + 5x

For a student who completed three sessions, substitute x = 3:

ŷ = 48 + 5 × 3
ŷ = 63

The model predicts a quiz score of 63. The intercept 48 is the model's starting point; the weight 5 adds five predicted score points per extra session. These values are model parameters, not facts about students.

With several features, the same idea becomes a sum of contributions:

ŷ = b + w₁x₁ + w₂x₂ + ... + wₚxₚ

Each feature gets its own weight. Starting with one feature matters because it makes the arithmetic—and the limits—easy to see.

Residuals Show What a Prediction Missed

For student C, the actual score was 65 and the candidate predicted 63. The signed difference is the residual:

residual = actual label − predicted label
residual = y − ŷ
residual = 65 − 63 = 2

The positive sign says the model predicted too low. A residual of -2 would say the model predicted too high by two points. The sign is helpful for diagnosis, but signs cancel if we simply add residuals across many rows. We need a loss that measures their size.

Here are all four predictions for the candidate ŷ = 48 + 5x:

The final column uses squared error. Squaring removes the sign and makes larger misses count more heavily. The total is the sum of squared errors (SSE) for this toy dataset:

SSE = Σ(y − ŷ)² = 10

Dividing this total by the number of examples would give the mean squared error (MSE): 10 / 4 = 2.5. For comparing two models on the same four rows, SSE and MSE choose the same winner because one is a fixed multiple of the other.

Compare Candidate Lines Instead of Eyeballing One

The line above feels plausible, but “feels plausible” is not a fitting method. Compare it with a second candidate:

Candidate A: ŷ = 48 + 5x
Candidate B: ŷ = 44 + 7x

Candidate B predicts 51, 58, 65, and 72 for students A–D. Its squared residuals are 1, 1, 0, and 36, for an SSE of 38.

On these rows and under squared error, candidate A is the better fit because 10 is lower than 38. Candidate B is exact for student C, but its large miss for student D is expensive after squaring. A model is judged across the dataset, not by its most flattering single example.

This is the meaning of fitting in linear regression: choose the intercept and weights that minimize a chosen loss on the training examples. In ordinary least-squares linear regression, the objective is to minimize the residual sum of squares. The exact search procedure can be solved directly in some simple cases; the next lesson introduces gradient descent, a repeated-update method used much more broadly.

What the Loss Is—and Is Not—Telling You

A lower training loss says that, on the examples used for fitting and for this particular loss, one set of parameters predicts closer to the labels than another. It does not say all of the following:

• that the learned relationship is causal;
• that the model will work as well for next month's cohort;
• that squared error represents the real cost of every mistake;
• that the model should be used outside the range it has seen.

Trade-off: Squared error gives a crisp, optimizable comparison and strongly penalizes large misses. This costs sensitivity to unusual observations: one bad label or genuinely exceptional case can dominate the fitted line. It can still fail when the loss does not represent the decision's real costs. The signal to watch is whether the largest residuals are data errors, important cases, or a sign that the model is missing structure.

The loss choice itself carries a preference. Squared error treats a miss of 6 as 36 units of loss, whereas two misses of 3 contribute 18 in total. That may be appropriate when large errors should matter much more. If the business cost is closer to typical absolute error, another measure such as mean absolute error may be more informative.

For the support team, a quiz-score estimate also must connect back to an action. A slightly lower MSE is not automatically useful if staff cannot decide what kind of support follows from a predicted score. The prediction contract from lesson 001 remains the boundary around this math.

Where a Straight Line Breaks

Linear regression is deliberately constrained. With ŷ = 48 + 5x, every additional session changes the prediction by exactly five points. Real learning patterns can plateau, jump after a threshold, or differ because another feature changes the effect.

The toy line also extrapolates mechanically. At x = 12, it predicts 108, even though the observed scores are on a 0–100 scale and training data covered only one to four sessions. A formula can compute outside its evidence; that does not make the answer credible.

These limitations do not make the model worthless. A linear model is often a strong baseline because its assumptions, inputs, errors, and parameters are visible. Compare it with a simple reference rule, evaluate it on unseen relevant examples, and then decide whether more complexity earns its cost.

Check Your Understanding

Check 1: The rule is ŷ = 48 + 5x. What score does it predict for x = 2.5 practice sessions?

Answer: 48 + 5 × 2.5 = 60.5. The prediction can be non-integer even if the observed scores are whole numbers.

Check 2: A model has residual -4 for a student. Did it predict too high or too low?

Answer: Too high. Because y − ŷ = -4, the prediction is four points larger than the actual label.

Check 3: Candidate A has a lower SSE than candidate B on the four training rows. Is A now proven to be the right model for future students?

Answer: No. It is the better candidate under this loss on these rows. It still needs evaluation on examples that did not choose its parameters, and the use case may demand a different loss or decision policy.

Practice: Make the Comparison Auditable

An after-school program wants to estimate a student's next assignment score from the number of completed practice exercises. It has these three historical rows:

Compare these rules:

Rule P: ŷ = 44 + 6x
Rule Q: ŷ = 47 + 4x

1. Calculate the three predictions for each rule.
2. Make a residual and squared-residual column for each rule.
3. Sum the squared residuals. Which rule fits these three rows better under SSE?
4. Name one reason not to claim that the better rule proves exercises caused the scores.

Self-check: Rule P predicts 50, 56, and 62, so its SSE is 1. Rule Q predicts 51, 55, and 59, so its SSE is 1 + 1 + 4 = 6. P is the lower-SSE rule on this sample. A different cause, such as earlier mastery or tutoring access, could affect both practice and scores.

Connections and Next Step

Linear regression sits at the meeting point of prediction and optimization. The prediction function supplies ŷ; residuals compare it with y; a loss aggregates those comparisons; fitting searches for better b and w values.

That loop transfers beyond straight lines. Later models use different prediction functions and losses, but they still need an objective and evidence about generalization. The immediate next question is operational: instead of manually proposing candidate weights, how can an algorithm move them toward lower loss? That is gradient descent.

Resources

• [COURSE] Google Machine Learning Crash Course: Linear regression — Focus: prediction equations, weights, bias, loss, and the training loop.
• [ARTICLE] Google Machine Learning Crash Course: Loss — Focus: residual-based loss and why squared loss emphasizes larger misses.
• [REFERENCE] scikit-learn: LinearRegression — Focus: ordinary least squares as minimizing residual sum of squares.

Key Takeaways

1. A one-feature linear regression model has the form ŷ = b + wx: an intercept plus a weight times the input.
2. Residuals show each miss; squared-error loss turns many misses into a quantity that can compare candidate fits.
3. Lower training loss is local evidence about a chosen objective, not proof of causation, future performance, or decision usefulness.