Classification Metrics and Trade-Offs

By the end of this lesson, you will be able to...

• read a confusion matrix as a record of operational outcomes, not just a score source;

• choose a primary metric and a threshold from the costs of false positives and false negatives;

• write a small decision memo that states the capacity, trade-off, and evidence needed before acting on classifier scores.

Idea in one sentence: A classifier's score becomes a useful decision only after we choose which mistakes matter, how much action we can take, and the threshold that makes those choices visible.

Core Insight

Suppose a support team can call only 60 students this month. The model ranks students by evidence, but the team still has to decide where to act. That decision cannot be read from accuracy alone because it spends a limited resource and exposes people to two different kinds of mistake.

The Decision Hiding Behind an Accuracy Score

A learning platform wants to offer a short support call to students likely to leave a course. There are 1,000 active students this month. From later outcomes, 80 were genuinely at high risk. The support team can complete at most 60 calls.

The first tempting rule is simple: fit a classifier, label every score above 0.50 as high risk, and report accuracy. That is a reasonable starting point when labels are balanced and the two error types cost about the same.

Here it is not enough. A model that labels every student “not at risk” is correct for 920 of 1,000 students: 92% accuracy. It also finds none of the 80 students the program meant to help.

Accuracy did not lie. It answered a much broader question: “What fraction of all labels were correct?” The operating question is narrower: “Which students should receive the 60 calls, and how many of the scarce calls are likely to help?”

Metrics make that question inspectable. The threshold is then a policy choice that turns a model score into an action.

Start With the Outcome Ledger

Suppose the model gives every student a risk score and the team initially uses a threshold of 0.70: offer a call when score >= 0.70. On a held-out evaluation set of 1,000 students, this produces the following illustrative counts.

This is the confusion matrix. It is an outcome ledger: it keeps the two kinds of wrong decision separate.

• A true positive is a call offered to a student who was genuinely at risk.
• A false positive is a call offered to a student who was not at risk.
• A false negative is a student at risk who was not offered a call.
• A true negative is a student correctly left out of this particular intervention.

The names can feel backwards at first. Read them in two parts: positive or negative decision, then true or false compared with the later outcome. “Positive” does not mean good; it means the model triggered the action.

The initial accuracy score is (TP + TN) / total = (42 + 902) / 1000 = 94.4%. It improved on the all-negative baseline, but it still hides the main decision: 38 of the 80 at-risk students were missed.

What Each Metric Lets the Team Ask

Different metrics divide the same four counts in different ways. Each answers a different operational question.

precision = TP / (TP + FP)
recall    = TP / (TP + FN)
accuracy  = (TP + TN) / (TP + FP + FN + TN)

For the 0.70 threshold:

precision = 42 / 60 = 70%
recall    = 42 / 80 = 52.5%
accuracy  = 944 / 1000 = 94.4%

Precision asks: “Of the students we call, how many are actually at risk?” It connects directly to the team’s limited time and to the cost of unnecessary outreach.

Recall asks: “Of all students actually at risk, how many did we reach?” It connects directly to missed opportunities for intervention.

Accuracy includes all outcomes. It can be useful as a coarse supplementary check when classes are balanced and error costs are similar. In this scenario, the 92% all-negative baseline shows why it cannot be the primary decision metric.

There is also a useful secondary count: alert volume, here TP + FP = 60. Precision alone does not tell the team whether it can act. A model might have good precision while still generating 5,000 alerts. Capacity is not a mathematical afterthought; it constrains the valid threshold choices.

Check: At threshold 0.70, does “70% precision” mean that 70% of all at-risk students were called?

Think first, then reveal.

Answer: No. It means 70% of the 60 called students were at risk. The fraction of all at-risk students called is recall: 42 out of 80, or 52.5%.

A Lower Threshold Changes the Service

The model has not changed. Only the action threshold changes. At 0.40, the team sees this held-out result:

Lowering the threshold labels more students positive. It captures 22 more at-risk students, so recall rises. It also creates 58 more false alarms and needs 140 calls, far beyond the team’s capacity.

This is the central trade-off. There is no universally “correct” threshold of 0.50, 0.70, or any other number. A threshold is appropriate only relative to the action it triggers, the available capacity, and the harm of each miss and false alarm.

For this program, a reasonable policy might be:

Constraint: no more than 60 calls this month.
Primary goal: maximize recall among feasible thresholds.
Safety check: inspect precision and the experience of contacted students.
Decision: use the highest-recall threshold whose predicted call volume fits capacity.

That is a situated preference, not a universal metric rule. If a false positive meant incorrectly denying a loan, the cost and threshold policy would be different. The same score can support different actions in different settings.

The Scores Need Evidence Too

A threshold orders scores into actions. It does not prove that a score of 0.70 is a literal 70% chance of dropout. That interpretation requires calibration: among many students assigned about 0.70, roughly 70% would need to have the positive outcome under comparable conditions.

For the decision above, ranking may be enough if the team always takes the highest-risk students until capacity is full. If a policy says “call only when risk exceeds 70%,” calibrated probabilities matter much more. These are different product claims and should be evaluated differently.

Likewise, metrics must be calculated on data not used to choose the model or repeatedly tune its threshold. The table is a teaching model of a held-out evaluation set. In a real workflow, keep a final test set untouched until the policy and model choices are stable, or use a validation process that separates threshold selection from final reporting.

The observed rate of high-risk students can also drift. A new course format, term timing, or definition of “at risk” may change prevalence. The old precision and recall then may no longer describe the new operation. Re-measure outcomes before claiming that last term’s threshold is still justified.

Curves Compare Possible Operating Points

A precision-recall curve summarizes precision and recall across many thresholds. It is especially useful here because the positive class is uncommon and the team cares about positive predictions and missed positives.

A ROC curve instead plots true-positive rate (recall) against false-positive rate across thresholds. It can describe ranking performance over possible thresholds, but it does not itself choose a threshold or encode the team’s capacity. A model can have a respectable aggregate curve and still be a poor fit for the small region where only 60 calls are possible.

F1 combines precision and recall into one number:

F1 = 2 × precision × recall / (precision + recall)

It is useful when precision and recall genuinely deserve similar weight and a compact comparison is needed. It is not a substitute for the outcome ledger. F1 does not know whether a false negative costs ten times a false positive, whether the team can make only 60 calls, or whether the scores are calibrated.

Common confusion: “The threshold with the best F1 is the best deployment threshold.”

Why it is tempting: F1 gives one clean maximum on a chart.

Better model: Choose it only when its equal weighting matches the decision. Otherwise use the confusion matrix, cost, and capacity to define the operating point.

A One-Page Threshold Decision Memo

Before deploying a classifier, write a short memo that makes the decision reviewable:

This memo makes a hidden assumption visible: the labels themselves must be meaningful. If “at risk” is inferred from an unfair or incomplete outcome measure, no metric repairs that target definition. Metrics evaluate agreement with labels; they do not prove that the labels represent a fair or causal intervention policy.

Practice: Choose the More Defensible Policy

The team can choose between two feasible monthly thresholds evaluated on a comparable held-out set:

Which option is more defensible if every call has the same cost, the team can make 60 calls, and the stated goal is to reach as many at-risk students as possible without increasing false alarms?

Model answer: Choose B. Both options have 15 false positives, but B reaches five more true positives and remains within capacity. Its precision is also higher (45 / 60 = 75% versus 40 / 55 ≈ 72.7%), while recall rises from 50% to 56.25%. This conclusion depends on the stated constraints; it would need revisiting if calls became harmful or capacity changed.

Resources

• [TUTORIAL] Google ML Crash Course: accuracy, precision, recall, and related metrics — Focus: connect each confusion-matrix count to a metric and its error-cost interpretation.
• [TUTORIAL] Google ML Crash Course: thresholds and the confusion matrix — Focus: inspect how one score cutoff changes the four outcome counts.
• [DOCUMENTATION] scikit-learn: Metrics and scoring — Focus: locate precision, recall, F1, ROC, and precision-recall evaluation tools after the decision logic is clear.

Key Takeaways

• A confusion matrix is an outcome ledger: it keeps false alarms and misses visible before any summary metric hides them.
• Precision asks whether positive actions are well targeted; recall asks whether real positive cases are being reached.
• A threshold is policy, not a model default. Its choice must name the action, cost, capacity, and evidence.
• Accuracy and F1 can be useful summaries, but neither chooses an operating point for an imbalanced, capacity-limited decision by itself.
• Evaluation measures agreement with a label. It cannot make an unclear, stale, or unfair target definition trustworthy.