Logistic Regression Fundamentals

By the end of this lesson, you will be able to...

• turn a linear score into a bounded binary-risk estimate with the sigmoid function;

• explain why log loss distinguishes uncertain correct predictions from confident correct ones;

• trace how one probability threshold changes which cases receive a positive action.

Idea in one sentence: Logistic regression first computes a linear score, converts it to a probability estimate, and leaves the final positive/negative action to a threshold chosen for the real cost of errors.

Core Insight

The support team has moved from predicting quiz scores to a different question: at the end of week two, should it invite a student to optional mentoring because the student may withdraw before the cohort ends?

The tempting model is a line that predicts 0 for “will not withdraw” and 1 for “will withdraw.” But an unconstrained line can output -0.4 or 1.3. Neither is a usable probability.

The team needs three things, in order:

features -> a score that combines evidence
score -> a bounded estimate of withdrawal risk
risk estimate + policy threshold -> mentoring invitation or no invitation

Logistic regression supplies the first two. The threshold supplies the third. Keeping those jobs separate prevents a common mistake: treating a model's probability estimate as if it were already the decision or a statement of certainty about a student.

A Linear Score Is Useful, but It Cannot Be a Probability

Use two time-valid binary features for a small teaching model:

• x₁ = 1 when week-two attendance is below 70%, otherwise 0;
• x₂ = 1 when the first assignment is missing, otherwise 0.

Suppose training has produced this linear score:

z = -1 + 1.2x₁ + 1.0x₂

The weights say how the model combines the selected inputs into a score. They do not prove that low attendance or a missing assignment causes withdrawal.

For a student with low attendance and a submitted assignment, x₁ = 1 and x₂ = 0:

z = -1 + 1.2(1) + 1.0(0) = 0.2

The value 0.2 is useful as a ranking score, but it is not yet a probability. Another combination can produce z = 3.4; a linear formula has no built-in 0-to-1 boundary.

This is where linear regression and logistic regression part ways. Linear regression can predict an unrestricted numerical label such as a quiz score. A binary outcome needs an output that can represent uncertainty without falling below 0 or above 1.

The Sigmoid Gives the Score a Bounded Meaning

Logistic regression passes z through the sigmoid function:

p = 1 / (1 + e⁻ᶻ)

Here, p is the model's estimated probability for the positive label, and e is a mathematical constant. The formula matters because it changes the score's range:

very negative z -> p close to 0
z = 0         -> p = 0.5
very positive z -> p close to 1

The output approaches 0 or 1 but does not reach either exactly. That lets the model express different strengths of evidence without producing impossible probabilities.

For A, p = 0.269 means the fitted model assigns an estimated 26.9% chance to the positive label under this simplified representation. It does not mean A is 26.9% likely to withdraw as an intrinsic fact, nor that a mentoring invitation will change that outcome. Calibration and evaluation are needed before using such estimates as reliable probabilities.

The model still has a linear boundary in feature space: it adds weighted inputs into z. The sigmoid changes the interpretation of the output, not the fact that the underlying score is a weighted sum. Polynomial or other engineered features can change what shapes that score can represent, as lesson 004 showed.

Why Correct Labels Are Not Enough

Suppose the true label is 1 for two historical students who later withdrew. Both receive a positive classification at a threshold of 0.5:

Both labels are “correct” after thresholding, but E is much better supported by the model than D. A training procedure that only counted correct labels would throw away that difference.

Logistic regression instead commonly minimizes log loss:

log loss = -[y ln(p) + (1 − y) ln(1 − p)]

For one row, y is the observed label (0 or 1) and p is the predicted probability. When y = 1, the second term disappears and the loss is -ln(p), as in the table. When y = 0, the loss is -ln(1 − p).

This makes confident mistakes expensive. If the true label is 0 but the model gives p = 0.95, the loss is -ln(0.05), about 2.996. The model assigned high probability to the wrong outcome.

Log loss does not certify that all probabilities are perfectly calibrated. It gives the optimizer a loss that rewards useful probability estimates and strongly penalizes certainty in the wrong direction. Validation must still test whether scores behave reliably on relevant unseen students.

A Threshold Turns an Estimate into an Action

The support team has capacity for mentoring invitations but does not want unnecessary outreach. It chooses a rule:

if p >= threshold: invite to optional mentoring
otherwise: do not invite now

Consider four validation examples. The outcomes are shown only because they arrived later and allow retrospective evaluation.

At a threshold of 0.50, F and G receive invitations. That yields one true positive (F), one false positive (G), one false negative (H), and one true negative (I).

Raise the threshold to 0.70, and only F receives an invitation. The false positive disappears, but H is still missed. In a larger dataset, changing the threshold would typically shift several cases across the boundary:

lower threshold -> more invitations, usually more true positives and more false positives
higher threshold -> fewer invitations, usually fewer false positives and more false negatives

The best threshold is therefore not contained in the sigmoid formula. It is a policy choice shaped by capacity, likely benefits and harms of outreach, and the relative cost of missed students versus unnecessary invitations. Lesson 008 will make these trade-offs explicit with classification metrics.

Probability, Prediction, and Intervention Are Different Claims

This lesson creates a useful boundary.

• A probability estimate is the model's numerical output for a defined label and feature set.
• A classification is a thresholded decision derived from that output.
• An intervention effect asks whether inviting a student causes a better outcome.

Logistic regression can support the first two jobs. It does not answer the third. A high-risk score can prioritize a conversation; it cannot prove why a student may withdraw or that mentoring will prevent withdrawal.

Trade-off: Logistic regression provides a compact, inspectable score and a bounded output that can rank cases. This costs expressive power: without useful feature transformations, one global linear score may miss nonlinear patterns or interactions. It can still fail when probabilities are poorly calibrated, features leak future information, or the threshold encodes an unfair or impractical policy. Signals to watch are calibration and error patterns on held-out data, plus whether the invitation policy has capacity and a defensible purpose.

Check Your Understanding

Check 1: The score for a student is z = 0. What probability does the sigmoid return?

Think first, then reveal.

Answer: 0.5. The formula becomes 1 / (1 + e⁰) = 1 / 2. This is the center of the sigmoid curve, not a universal decision threshold.

Check 2: A true negative case receives p = 0.95. Is this a small or large log-loss contribution?

Think first, then reveal.

Answer: Large: -ln(1 − 0.95) = -ln(0.05), about 2.996. The model was confidently wrong.

Check 3: If the mentoring threshold is lowered from 0.70 to 0.40, what must happen to the number of invitations?

Think first, then reveal.

Answer: It can only stay the same or increase, because every case already at or above 0.70 is still at or above 0.40. Whether this is better depends on the error costs and capacity.

Practice: State the Policy, Not Just the Model

A program predicts missed-deadline risk for four students: 0.81, 0.62, 0.47, and 0.18. It has capacity to offer a time-intensive one-to-one check-in to only the two highest-risk students.

1. Give a threshold that selects exactly two cases.
2. State what the model has provided and what the threshold policy added.
3. Name one piece of evidence needed before claiming this check-in improves completion.

Model answer: A threshold of 0.62 selects the first two cases if the rule is p >= threshold. The model supplied estimated risks; the policy converted them into two check-in offers under a capacity constraint. To claim the intervention improves completion, compare outcomes under a design that can separate the effect of the check-in from pre-existing differences between students.

Connections and Next Step

Logistic regression keeps the parameterized-score and optimization loop from the first cluster, but changes the target from a numeric value to a binary label. The sigmoid and log loss make probability estimation visible; a threshold turns that estimate into a decision.

The next lesson, Decision Trees and Rule-Based Splits, introduces a contrasting classifier. Instead of one global score, it creates local rule-based regions by splitting examples on features.

Resources

• [TUTORIAL] Google Machine Learning Crash Course: Logistic regression — Focus: why binary-outcome prediction uses a probability model rather than a raw linear output.
• [TUTORIAL] Google Machine Learning Crash Course: Sigmoid function — Focus: calculate a bounded probability estimate from a linear score.
• [TUTORIAL] Google Machine Learning Crash Course: Log loss and regularization — Focus: why confident wrong probabilities receive a larger loss.
• [ARTICLE] Google Machine Learning Crash Course: Thresholds and the confusion matrix — Focus: convert probabilities into categories with a policy threshold.

Key Takeaways

1. Logistic regression converts a weighted linear score into a 0-to-1 probability estimate with the sigmoid function.
2. Log loss rewards probability estimates that support the observed label and heavily penalizes confident mistakes.
3. A threshold converts a probability estimate into a classification or action; it is a policy choice, not a fact of nature.
4. A risk model can prioritize support but cannot establish causal explanations or intervention effects.