Learning Curves and Model Diagnosis

By the end of this lesson, you will be able to...

• review a supervised prediction system from decision contract through model evidence;

• read learning curves without confusing a plausible diagnosis with a proven cause;

• write a short recommendation for the next experiment, including what must remain fixed and what evidence could change the decision.

Idea in one sentence: A learning curve turns “try something else” into a narrower question: as the model sees more valid evidence, is the main limit capacity, unstable fit, representation, data volume, or the decision design itself?

Core Insight

Consider a clinic that can make only 80 reminder calls each morning and must decide whom to call at booking time. One score cannot tell that clinic whether a model is missing useful structure, memorizing temporary detail, or being judged by an impossible evaluation. A learning curve makes one part of that uncertainty visible by repeating the same valid comparison at several training sizes. The important word is same: if the target, threshold, feature set, preprocessing boundary, or validation period changes at the same time, the curve no longer isolates the effect of more evidence.

The team is not looking for a curve shape to memorize. It is looking for a defensible next question. A low, flat validation trace may shift attention toward representation; a persistent training-validation gap may justify a restraint experiment; an upward validation trace may make additional labels worth their cost. In every case, the curve earns a limited inference, and the broader system contract decides whether that inference is useful.

The Scenario

Consider a clinic that wants to offer a limited number of reminder calls for appointments likely to be missed. Every booking morning, the clinic can call at most 80 patients. A model assigns a no-show risk score, and staff call the 80 highest-scored bookings.

The team has a dashboard full of scores but no agreed next step. One person asks for a bigger gradient-boosted model. Another asks for more labeled appointments. A third proposes adding whether the patient rescheduled after receiving a reminder. All three ideas can sound plausible in isolation.

The initial model is that the team should pick the idea with the largest training improvement. That works only if training fit is the decision the clinic cares about. It is not. The clinic needs to find missed appointments among future bookings, using information available at booking time and without exceeding 80 calls.

The stronger model is an evidence review. It connects the target, feature boundary, metric, split, model behavior, and learning curve before choosing one next intervention. A learning curve is not the review by itself. It is one diagnostic trace inside it.

Constraints

The team writes down the system it is actually evaluating.

This is not paperwork around the model. It determines what a useful score means. A model with high overall accuracy can still be unhelpful if it misses no-shows in the top 80. A model whose best feature arrives after the reminder cannot make the booking-time decision honestly.

The team holds October through December aside as the final future test period. On January through September, it uses several past-to-future development folds. Every fold refits the complete pipeline: imputation, category handling, feature transformations, and model. The resulting scores are development evidence, not a result to advertise as final performance.

Design Goal

The goal is not “maximize a chart.” The team needs a recommendation that another reviewer can inspect:

Given a booking-time feature contract,
a fixed 80-call capacity,
and chronological development folds,
which next experiment is most likely to improve future recall?

That question keeps four earlier ideas connected.

• A supervised system starts with a target, features, loss or metric, and decision context—not an algorithm name.
• A model family supplies a bias: a linear model prefers a global additive rule; a tree ensemble can represent thresholded interactions, but can also fit detail more aggressively.
• Validation must simulate the information and population available in the real decision.
• Regularization is chosen from held-out evidence. It cannot repair a leaked feature or an invalid metric.

Learning curves add one variable: training size. They ask how training and validation behavior change when the same valid pipeline receives more examples.

Proposed Models

The clinic compares three candidates. All use the same allowed booking-time inputs, the same chronological folds, and the same top-80 decision rule.

This is a comparison of hypotheses, not a ranking of prestigious algorithms. The baseline may win if broad effects carry most of the signal. The ensemble may win if interactions such as appointment type and short booking lead time matter. Neither result establishes a causal story about why a patient misses an appointment.

Walkthrough: Build the Curves Without Moving the Goalposts

For each training size, the team takes an earlier portion of each development fold, fits the full pipeline only on that portion, and evaluates the fixed top-80 policy on the subsequent validation period. The values below are illustrative averages across the valid folds.

Start with the logistic baseline. Its training and validation recall are close, but both flatten around 0.69. That is evidence that more examples of the same representation and model may offer little improvement. It does not prove that data collection is useless: new data could cover an underrepresented clinic or appointment type. It does say that “collect anything” is not the most direct explanation of this curve.

Now look at weakly restrained boosting. With 600 appointments it nearly memorizes training examples while validation recall is poor. As data grows, training recall falls and validation recall rises. The gap narrows, but remains visible at 6,000 appointments. This is a high-variance-like pattern: extra flexibility is finding both genuine interaction structure and sample-specific detail.

The moderately restrained version gives up training recall at every size. Its validation recall is higher at 3,600 and 6,000 appointments. That is the decisive observation in this teaching example. The penalty did not make the model “better” by definition; it changed the trade-off so that more of what it learned survived later validation periods.

same feature contract + same split + same decision threshold
        |
        +--> train much higher than validation? inspect excess flexibility
        +--> both low and close? inspect representation, capacity, target, or metric
        +--> validation rises with size? more valid examples may still help
        +--> validation flat? more of the same data may have limited value

The last line is deliberately cautious. A flat curve can reflect a real ceiling, noisy labels, a metric that hides the useful change, or an evaluation period that is too short to resolve a difference. It narrows the investigation; it does not announce a law.

Failure Review

Before accepting the curve diagnosis, the team asks whether its evidence could be misleading.

Could the feature contract be wrong? Yes. If rescheduled_after_reminder or a future schedule update entered any training subset, the curve would describe a temporally impossible system. Remove the feature and recompute the whole trace.

Could the split be wrong? Yes. If the clinic will score later bookings, random folds can let future practice patterns shape earlier validation. Use the chronological split. If the intended population changes to a new clinic, add a group boundary that tests that deployment promise.

Could the metric be wrong? Yes. Recall at 80 calls fits the stated capacity, but it should be read alongside the number of unnecessary calls and the consequences of a missed patient. If staffing changes from 80 to 20 calls, the ranking threshold and evidence must be recomputed.

Could a curve hide instability? Yes. A mean curve can conceal that one fold collapsed after a policy change. Keep the individual fold scores and inspect the period, class rate, missing values, and new categories behind a large deviation.

These checks prevent a common failure: using a sophisticated diagnostic plot to make a basic evaluation error look more scientific.

Trade-offs

The evidence supports a provisional choice: continue development with the moderately restrained boosted pipeline. It has the highest validation recall at the clinic’s fixed capacity, and its curve shows that additional valid appointment data may still help.

That choice costs computation and interpretability compared with the logistic baseline. It also creates a more demanding review task: the team must verify that feature values, preprocessing, and threshold selection are repeated exactly in each fold. It can still fail when an appointment policy changes, when new clinics differ, or when no-show labels are incomplete.

There is also a data-investment trade-off. The rising validation curve makes additional labeled bookings plausible evidence for future improvement, but it does not specify how many labels are worth their cost. The clinic should compare the cost of waiting for another quarter of labels with cheaper experiments: inspect high-error appointment types, validate a booking-time feature proposal, or tune restraint only within the development data.

The signal to watch is not merely the next aggregate score. Watch whether validation recall at 80 calls improves consistently across later periods without widening the training-validation gap or increasing the burden of unnecessary calls.

Evidence and Readiness

Before reading the final October–December test, the team records a small decision memo.

System: booking-time no-show ranking
Decision: 80 reminder calls per morning
Feature boundary: booking-time fields only; no post-reminder events
Development evidence: chronological folds and learning curves
Selected candidate: moderately restrained boosted pipeline
Reason: highest validation recall at fixed capacity with a smaller gap than weak restraint
Next experiment: collect the next valid booking period and test one pre-booking feature proposal inside the same pipeline
Do not change: target definition, capacity metric, time boundary, held-out test period

Only after this memo is fixed does the team fit the selected pipeline on all January–September development data and evaluate once on October–December. That test result checks the selected system; it should not be repeatedly reopened to settle new model ideas.

Readiness is not a claim that the model is safe or finished. It means the team can explain what it is predicting, which evidence it used, why the split is credible, what the curve suggests, and which observation would make it reconsider.

There is one final discipline in this review. Change one explanatory element at a time. If the team adds a new feature, switches model family, changes the call capacity, and collects a new period of data together, an improved score will not tell it which investment produced the benefit. A small, pre-registered experiment can be slower than changing everything at once, but it creates knowledge the next decision can reuse. That is the capstone skill: turn a model score into a reasoned experiment rather than a guess with a chart attached.

Check Your Understanding

Check: A model’s validation recall rises from 0.61 to 0.74 as training data grows, but its training-validation gap remains 0.20. Does the curve say “collect more data and do nothing else”?

Think first, then reveal.

Answer: No. Rising validation recall makes more valid data a plausible investment, but the persistent gap also suggests that flexibility or representation stability deserves inspection. Compare a restrained version under the same pipeline and splits, and keep the final test untouched until the choice is fixed.

Check: The clinic changes its capacity from 80 calls to 20. May it reuse the previous curve and selected threshold without revision?

Think first, then reveal.

Answer: No. The decision policy and metric have changed. Recompute recall or precision at the new capacity using the same honest feature and split boundaries. A model’s ranking may be useful at 80 calls but poor at 20.

Final Challenge: Write the Review

A school district wants to prioritize 50 tutoring invitations each week for students at risk of missing an assignment. Its draft model uses current attendance, assignment type, and teacher_followup_completed. It reports high training recall, lower validation recall, and a validation curve that is still rising. The team used random folds even though it will predict next week’s assignments.

Write a four-part review before recommending a larger model or more data:

1. Name the decision moment, capacity, and metric that should replace the vague phrase “model accuracy.”
2. Identify at least one feature or split problem that makes the current evidence unreliable.
3. State what the curve suggests and one competing explanation it does not settle.
4. Propose one next experiment and name what must remain fixed.

Model answer: The model should score each assignment before the tutoring invitation is sent, rank students, and be judged by recall or a related cost-aware measure among the 50 available invitations. teacher_followup_completed is suspicious because it may be recorded after the prediction moment; its availability must be checked or it must be removed. Random folds can train on later assignments and validate on earlier ones, so use chronological development windows and fit every learned transformation inside each training window. The rising validation curve makes more valid examples plausible, but the training-validation gap could also reflect excess flexibility or leakage. First rerun the existing pipeline with a valid time split and a modest regularization comparison; keep the target, 50-invitation capacity, feature contract, and final future test period fixed. Only then decide whether more labels or a larger model are justified.

Resources

• [DOCUMENTATION] scikit-learn: learning_curve — Focus: see how training and test scores are computed across chosen training sizes.
• [DOCUMENTATION] scikit-learn: Cross-validation — Focus: keep preprocessing, split design, and final-test discipline inside the diagnostic workflow.
• [DOCUMENTATION] scikit-learn: Validation curves — Focus: contrast data-size diagnosis with model-complexity diagnosis.

Key Takeaways

• A learning curve is useful only inside a valid prediction, feature, metric, and split contract.
• Training and validation traces narrow plausible bottlenecks; they do not prove a single cause or prescribe a universal fix.
• The best next experiment changes one justified element while keeping the decision boundary and final test evidence intact.
• A trustworthy model review can defend both a chosen candidate and the observation that would make the team change its mind.