Machine Learning Fundamentals

By the end of this lesson, you will be able to...

• frame a supervised prediction problem using examples, features, a label, a prediction time, and an action;

• trace the path from historical examples to a model and then to predictions on unseen examples;

• recognize when a clean rule, missing evidence, or an unusable prediction makes machine learning the wrong tool.

Idea in one sentence: Machine learning is useful when examples can shape a prediction rule, but the rule is trustworthy only when the target, available evidence, decision, and test on new data are defined together.

Core Insight

A learning platform wants to help students before they leave a course. At the end of week two, a support team can contact a small number of students. The first proposal sounds reasonable:

Train a model to find students who need help.

The proposal mentions a model, but it does not yet define a learning problem. What counts as “need help”? When must the prediction exist? Which information is available at that moment? What will the support team do with the result?

If those questions stay vague, a technically accurate model can answer the wrong question. It might predict a label that arrives too late, use information from the future, or produce a score that nobody can act on.

The important shift is small but fundamental:

machine learning problem
≠
algorithm + data

machine learning problem
=
prediction contract + examples + learning procedure + evaluation

This lesson uses supervised learning because the track begins with prediction from labeled examples. Other learning settings exist, but choosing among them is not the first job here. The first job is to make one prediction problem honest and inspectable.

Before Choosing a Model, Define the Decision

The support team can send an optional mentoring invitation to 40 students each Monday. The invitation is helpful but not free: mentors have limited time, and unnecessary messages can feel intrusive.

That gives the prediction a purpose:

Goal:
Prioritize optional support invitations.

Prediction time:
Friday at the end of week two.

Outcome to predict:
Whether the student withdraws before the cohort ends.

Action:
Invite some high-risk students to speak with a mentor.

This is narrower than “find students who need help.” Narrowness is useful. It lets us ask whether each row of historical data represents the right thing and whether every input would truly exist on Friday of week two.

The first reasonable model is:

If historical examples contain patterns, a learning algorithm can discover those patterns better than a growing pile of hand-written rules.

That model is often useful. A fixed rule such as “contact anyone with attendance below 70%” cannot easily combine attendance, assignment completion, and recent changes in activity. A learned model can combine several signals and adjust their influence from data.

But the initial model is incomplete. Examples do not decide what the system should predict. They only contain recorded facts. We still have to choose the label, eligible features, prediction time, action, and evaluation boundary.

The Smallest Supervised-Learning Vocabulary

We can now name the pieces without making them mysterious.

• An example is one case the model can learn from or predict. Here, one example is one student enrolled in one cohort.
• A feature is an input available to the model at prediction time, such as week-two attendance.
• A label is the historical answer we want the model to predict, such as whether the student later withdrew.
• A model maps features to a prediction. Training shapes that mapping from labeled examples.
• Training is the process that adjusts or selects the model using historical examples.
• Inference is using the trained model to predict for a new example whose label is not yet known.
• Generalization is useful performance on relevant examples that did not shape the model.

For this lesson, treat the model as a simplified teaching object:

features -> model -> prediction

Later lessons will open that box. Linear regression will expose parameters and error. Gradient descent will expose repeated updates. Here, we need only the boundary: training may use historical features and labels; inference receives current features and produces a prediction.

Where the Initial Model Breaks

Suppose the dataset contains these columns:

student_id
week_2_attendance
week_2_assignments_completed
final_exam_score
withdrew_before_end

At first glance, final_exam_score looks valuable. Students with a missing or low final score probably withdrew more often. A model using that column may look excellent in an offline test.

But the support team predicts at the end of week two. The final exam has not happened. The feature contains information from after the prediction time.

This is target leakage: information unavailable when the real prediction must be made enters model development. Leakage gives the model an unfair view of the answer and produces optimistic evidence.

The correction is auditable:

We expected:
More informative columns should improve the model.

The timeline shows:
final_exam_score does not exist at the prediction time.

The stronger model says:
A feature is eligible only if the real system can know it at inference time.

There is a second break. The label withdrew_before_end records an outcome, not a cause. A model may estimate withdrawal risk. It does not tell us why a student left, whether outreach will help, or whether the student “deserves” attention.

Prediction and intervention are different questions. This track begins with prediction. Causal questions about whether an action changes an outcome belong to later study.

A Worked Prediction Contract

The following values are illustrative. They are a toy dataset, not observed measurements.

Rows A–D will stand in for training examples. Rows E–F will stand in for unseen evaluation examples. A real project would need far more data, a defensible split, quality checks, and careful analysis of how cohorts differ. The small table exists only to make the roles visible.

Step 1: Fix the unit and prediction time

One example means one student-cohort enrollment. The prediction is made at 18:00 on the Friday ending week two.

This prevents quiet changes in meaning. A row is not “a student forever,” and the model cannot use Saturday activity for a Friday prediction.

Step 2: Choose the label

The label is:

1 if the student formally withdraws before the cohort ends
0 otherwise

This is a direct historical outcome, but it is still limited. A student who stays may struggle. A student who withdraws may have reasons unrelated to course engagement. The label is useful only for the specific prediction contract.

Step 3: Admit only time-valid features

The first candidate features are week-two attendance and assignment completion. The final score is rejected because it lies in the future. A staff note written after outreach would also be rejected.

An eligibility question catches many mistakes:

Could the production system compute this exact value at the prediction timestamp without seeing the future?

Step 4: Train a model

The training procedure receives the features and labels from historical rows:

historical features + historical labels
                    |
                    v
                 training
                    |
                    v
                  model

Training searches for a mapping that performs well under a chosen objective. Different model families search in different ways. We do not need to select one yet.

Step 5: Use the model on new examples

For a new cohort, the label is unknown at prediction time:

current week-two features -> trained model -> risk score

Assume the toy model gives E a score of 0.35 and F a score of 0.72. These numbers are illustrative outputs, not measured probabilities. A policy can use scores to prioritize limited outreach, but the model does not choose the policy by itself.

Step 6: Evaluate after outcomes arrive

When the cohort ends, the team compares predictions with outcomes on examples that did not shape the model. The evaluation must reflect the future use case closely enough to provide evidence about generalization.

The full path is now visible:

define outcome and action
-> freeze prediction time
-> build time-valid examples
-> separate development from evaluation evidence
-> train
-> predict for unseen examples
-> act under a policy
-> observe later outcomes
-> evaluate and revise

So far, the algorithm has been the least important choice. The contract determines what every later number means.

What Generalization Does—and Does Not—Prove

Low training error shows that a model fits the examples used to shape it. That is not the product promise. The product promise concerns future students.

Evaluation on held-out examples provides stronger evidence because those labels did not guide training. It still does not prove permanent real-world performance. A new cohort may differ because the course format, student population, calendar, or withdrawal process changed.

Generalization therefore depends on a relationship:

development data
must be relevant to
evaluation data
which must be relevant to
future use

A random split is not automatically honest. If the same student appears in several rows across cohorts, related rows can cross the split. If behavior changes over time, older and newer cohorts may differ. Lesson 014 will examine validation design in detail.

For now, keep one question:

What future situation is this evaluation set standing in for?

If the answer is unclear, the score is unclear too.

When Machine Learning Is the Wrong Tool

Machine learning earns its complexity when examples contain a reusable pattern, the label is meaningful, prediction-time evidence exists, and a prediction can improve a decision.

Prefer a hand-written rule when the rule is exact and stable:

If a payment amount exceeds an account's configured limit, reject it.

That is a policy, not a pattern to estimate.

Pause the ML project when:

• the desired outcome cannot be defined or observed;
• the only strong features arrive after the decision;
• historical examples do not represent intended use;
• no safe or useful action follows from the prediction;
• the cost of mistakes is unacceptable without stronger safeguards;
• a simple rule already meets the need more clearly.

This boundary is not anti-ML. It is part of competent ML work.

Consequences, Costs, and Limits

A learned model can combine weak signals that are awkward to encode as rules. The cost is a new dependence on data and evaluation.

The model can inherit measurement errors and historical patterns. A label records how a process behaved, including earlier policies and access differences. A high score is not a moral judgment about a person.

The support policy also creates consequences. Too many false alarms waste mentor capacity and may annoy students. Missed high-risk students lose an opportunity for support. Later lessons will make those error trade-offs precise.

Finally, prediction does not solve intervention design. Even a well-evaluated risk model cannot establish that mentoring causes retention. The model helps prioritize a question; it does not settle every decision around that question.

Signals that the boundary is near include:

• offline performance that collapses on a later cohort;
• features that cannot be reproduced at prediction time;
• large performance differences across relevant groups;
• predictions that arrive after the action window;
• a support team that cannot explain what action a score triggers.

Check Your Understanding

Check 1: The team adds days_until_withdrawal as a feature. It is strongly predictive. Can the feature be used for the week-two model?

Think first, then reveal.

Answer: No. The value depends on a future withdrawal date, so it cannot exist at the week-two prediction time. Its apparent strength is leakage, not usable predictive evidence.

Check 2: The model performs well on historical cohorts. Does that prove mentoring will prevent withdrawal?

Think first, then reveal.

Answer: No. The evaluation may support a prediction claim: the model can identify higher-risk cases in similar data. Whether mentoring changes withdrawal is a causal intervention question.

Check 3: A deterministic policy already says every student who requests a mentor receives one. Should a model replace it?

Think first, then reveal.

Answer: Not for that decision. The policy is explicit and directly follows a request. A model could introduce avoidable uncertainty. A different constrained problem—such as prioritizing outreach when demand is hidden—would need its own contract.

Practice: Frame a Nearby Problem

A support desk wants to predict, when a ticket is created, whether it will miss a 24-hour response deadline. Draft the smallest prediction contract.

A good answer should identify:

• one example;
• prediction time;
• label;
• two eligible features;
• one tempting but invalid future feature;
• the action triggered by the prediction;
• what unseen data should represent.

Model response:

Example:
One newly created support ticket.

Prediction time:
Immediately after ticket creation.

Label:
Whether the first human response arrives more than 24 hours later.

Eligible features:
Queue length at creation; ticket category.

Invalid future feature:
The actual response time.

Action:
Escalate high-risk tickets to a triage queue.

Evaluation evidence:
Tickets from a later time window, with no future-derived features and with
the same response-time definition used in training.

This answer is not complete because it names an algorithm. It is complete because every future model must obey the same decision and evidence boundary.

Connections

• Linear Regression Fundamentals opens the model box and shows how features, parameters, predictions, and errors fit together.
• Classification Metrics and Trade-Offs asks how the costs of false alarms and missed cases should shape evaluation and thresholds.
• Feature Engineering and Representation returns to feature availability, leakage, and reproducible transformations in greater depth.

Resources

• [COURSE] Supervised Learning — Focus: examples, features, labels, training, evaluation, and inference as one workflow.
• [ARTICLE] Datasets: Labels — Focus: direct labels, proxy labels, and why label choice constrains what a model can mean.
• [ARTICLE] Datasets: Dividing the Original Dataset — Focus: why development and evaluation need separate evidence.
• [TUTORIAL] Common Pitfalls and Recommended Practices — Focus: prediction-time feature availability, leakage, and keeping training separate from testing.

Key Takeaways

• A supervised-learning project begins with a prediction contract, not an algorithm.
• Features must exist at the moment the real prediction is made; future information creates leakage.
• Training shapes a model from labeled examples, while inference applies that model to examples whose labels are not yet known.
• Held-out evaluation is evidence about generalization only when it represents the future use case.
• A prediction is not a causal explanation, a policy, or a moral judgment; its value depends on the action and boundary around it.