Hypotheses Need Risk

By the end of this lesson, you will be able to...

• Distinguish a research question, hypothesis, prediction, and revision rule.

• Rewrite a vague scientific claim so that observations can genuinely count against it.

• Explain why a failed prediction challenges a package of claims and assumptions, not always one hypothesis in isolation.

Idea in one sentence: A useful hypothesis rules out some possible observations before we see the result, so evidence has a real chance to change our mind.

Core Insight

A team has built a new retrieval component for an AI support assistant. The team says:

The new component makes answers more reliable.

The sentence sounds reasonable. It may even be true. But what would make the team admit that the component did not improve reliability?

Suppose the benchmark score rises. The team points to the score. Suppose the score stays flat. The team says the benchmark missed subtle improvements. Suppose factual errors fall but unanswered questions rise. The team says cautious silence is also reliability.

Every result can be turned into a success story.

That is the problem this lesson solves. Science needs claims that can lose contact with the evidence. A hypothesis becomes informative when it exposes itself to observations that would weaken it, narrow it, or force its replacement.

Here, risk does not mean danger. It means exposure to correction.

The Claim That Cannot Lose

The team's original sentence has several escape routes:

• Reliable has not been defined.
• The population of support questions is unknown.
• There is no comparison with the old system.
• There is no expected direction or size of change.
• The team has not said which side effect would cancel the improvement.
• There is no result that would trigger revision.

This flexibility feels safe. It is also epistemically weak. If a claim agrees with every possible result, then the result cannot distinguish that claim from its competitors.

Compare these two statements:

1. “The new retrieval component helps somehow.”
2. “On a locked set of English support questions, the new component will reduce unsupported factual claims by at least 20% relative to the current system, while increasing the unanswered-question rate by no more than 3 percentage points.”

The second statement may fail. That is exactly why success would teach us more.

Check: Which statement takes more scientific risk?

Think first, then reveal.

Answer: The second statement. It names a population, comparison, outcome, expected change, and cost boundary. Several observations could conflict with it. The first statement can absorb almost any outcome.

From a Question to a Risky Hypothesis

Four pieces are easy to blur together.

Plain meaning:

A hypothesis is a proposed answer that commits us to an observable pattern.

In this scenario:

The proposed answer is that filtering weak sources before generation reduces unsupported claims. If that mechanism matters, the new system should behave differently from the old one on relevant questions.

Technical name:

The claim is testable because it has potential observations that conflict with its prediction. In a stricter Popperian vocabulary, a claim is falsifiable when some conceivable observation is incompatible with it.

Testability is not the same as truth. A false claim can be testable. A true claim can be stated so vaguely that the current test teaches little. Testability describes the relationship between a claim and possible evidence.

Build the Risk Before Looking at the Result

Let us turn the team's idea into an inspectable hypothesis package.

Step 1: Name the scope

The team limits the claim to:

• English customer-support questions;
• product behavior documented in the approved knowledge base;
• a frozen set of 600 questions sampled before the new component is evaluated;
• the current Model A and proposed Model B configurations.

The team is not yet claiming that the result generalizes to every language, every domain, or every future model.

Step 2: Name the mechanism

The mechanism claim is:

Retrieval gating rejects low-support passages before generation, so the model has fewer opportunities to convert weak evidence into confident factual statements.

This is stronger than saying “Model B is better.” It says what is expected to change and why.

Step 3: Name the observable prediction

Before running the benchmark, the team records:

Model B will reduce the proportion of answers containing at least one unsupported factual claim by 20% or more relative to Model A. Its unanswered-question rate may rise, but by no more than 3 percentage points.

The second condition matters. A system could eliminate unsupported claims by refusing to answer everything. That would improve one metric by destroying the service.

Step 4: Name the assumptions that connect claim to observation

The prediction does not follow from the hypothesis alone. It also depends on assumptions:

• evaluators can identify unsupported claims consistently;
• the benchmark represents the stated population;
• neither model has seen the answer labels;
• both systems use the same generation settings except for the retrieval component;
• 600 questions provide a stable enough comparison for the planned conclusion.

These are auxiliary assumptions. They form the bridge from the mechanism claim to the expected observation.

Step 5: Name the revision rule

The team records three possible responses:

• If both thresholds are met, the result supports this limited claim.
• If factual errors fall but abstention rises too much, the design has traded one failure for another.
• If the factual-error threshold is missed, the planned claim is not supported. The team must inspect the hypothesis, the scope, and the auxiliary assumptions before proposing a new test.

Writing the revision rule before seeing the data closes the easiest escape routes.

A Worked Evidence Path

Now run the locked comparison.

The unsupported-claim rate falls from 18% to 12%. The relative reduction is:

(18 - 12) / 18 = 0.33, or 33%

The unanswered rate rises by 2 percentage points.

Both planned boundaries are met. The result supports the limited prediction. It does not prove that retrieval gating is the only possible explanation, that the evaluator is perfect, or that Model B will improve every domain.

Now change one result:

The relative reduction is about 17%. The prediction is missed.

The team should not quietly replace “20%” with “any reduction” after seeing 17%. It should report that the planned threshold was missed. The 17% result may motivate a new hypothesis, but it cannot retroactively become the prediction that was tested.

So far, the path is:

question
-> mechanism hypothesis
-> prediction plus auxiliary assumptions
-> locked observation
-> comparison with the revision rule
-> support, narrowing, or repair

The naive path looks different:

collect results
-> choose the most flattering metric
-> invent a matching explanation
-> call the explanation predicted

The second path can generate ideas. It cannot provide the same test of those ideas on the data that generated them.

Prediction, Accommodation, and Honest Exploration

Scientists often discover patterns after looking at data. That is not a defect. Exploration is how many useful hypotheses begin.

The important distinction is between prediction and accommodation:

• A prediction is recorded before the relevant result is known.
• An accommodation is an explanation shaped to fit a result already observed.

An accommodated explanation can be insightful. It becomes risky when it faces new evidence that did not shape it. The practical sequence is:

observe an unexpected pattern
-> propose an explanation
-> derive a fresh prediction
-> test that prediction on new data or a new setting

Preregistration is one tool for recording hypotheses, methods, outcomes, and analysis choices before a study. It does not make a weak hypothesis strong, and it does not forbid exploration. It makes the timeline of the reasoning easier to inspect.

Why One Failed Prediction Is Not Always the End

A simple slogan says: one conflicting observation falsifies a hypothesis. Real tests are less tidy because observations depend on a package:

core hypothesis
+ scope conditions
+ measurement model
+ experimental setup
+ background assumptions
-> prediction

If the prediction fails, at least one part of the package needs attention. Several diagnoses are possible:

1. The mechanism may be wrong. Retrieval gating may not reduce unsupported claims.
2. The effect may be smaller than claimed. The direction may be right, but the 20% threshold may be too strong.
3. The scope may be wrong. The effect may appear only for some question categories.
4. The measurement may be weak. Evaluators may disagree about what counts as unsupported.
5. The comparison may be contaminated. A configuration difference other than retrieval may have changed.
6. Ordinary variation may matter. A noisy sample can miss or exaggerate a real effect.

This does not give the team permission to rescue its favorite idea forever. Every repair creates a new commitment. If each failed test produces another untested exception, the claim is losing risk again.

Check: The planned threshold is missed, and the team discovers that half the answer labels were generated by Model B itself. What should change first?

Think first, then reveal.

Answer: The evaluation assumption must be repaired before the benchmark can strongly challenge or support the retrieval hypothesis. The result still matters—it exposed a broken evidence chain—but it does not cleanly isolate the proposed mechanism.

Trade-offs and Limits

Risky hypotheses improve learning because they reduce interpretive freedom. They also impose costs.

The central trade-off is between exposure and flexibility. A more exposed claim can teach us more from a result, but it gives us less freedom to reinterpret that result afterward.

• Specificity can become brittleness. A threshold chosen without domain knowledge may turn a useful question into an arbitrary pass/fail game.
• Not every field can run a clean intervention. Astronomy, geology, ecology, and historical sciences often test risky predictions through observations that nature produces rather than experiments researchers control.
• Probabilistic hypotheses rarely forbid one outcome absolutely. Their risk comes from assigning different expected patterns or probabilities, not from claiming an event is impossible.
• A passed prediction is not proof. Competing hypotheses may predict the same result.
• A failed prediction is not permission for endless repair. Revisions should become new, independently testable commitments.

The best question is not merely, “Can this claim be falsified?” Ask:

Which observations would discriminate this hypothesis from serious alternatives, and what would I revise if those observations appeared?

That question keeps the useful pressure of falsifiability without pretending that scientific judgment is a one-bit machine.

Common Confusions

Confusion: A risky hypothesis is an unlikely hypothesis

Why it is tempting:

Everyday language associates risk with low odds.

Better model:

Scientific risk means exposure to conflicting evidence. A highly plausible hypothesis can still be risky if it makes clear predictions.

Confusion: Falsifiable means false

Why it is tempting:

The words sound similar.

Better model:

Falsifiable means that conceivable observations can conflict with the claim. It says nothing by itself about whether the claim is currently true.

Confusion: Any precise sentence is a good hypothesis

Why it is tempting:

Numbers create an appearance of rigor.

Better model:

A precise claim can still use a bad measure, irrelevant threshold, or biased sample. Precision exposes a claim to a test; it does not guarantee a good test.

Confusion: A post-hoc explanation is useless

Why it is tempting:

Explanations fitted after seeing data can overfit that data.

Better model:

Post-hoc explanations are useful for discovery. Their fresh predictions need fresh evidence before they receive the evidential credit of successful prediction.

Practice: Write a Risk Card

A greenhouse team claims:

Fertilizer F helps tomato seedlings grow better.

Rewrite the claim as a small risk card with five fields:

1. Scope.
2. Mechanism hypothesis.
3. Observable prediction.
4. Important auxiliary assumptions.
5. Revision rule.

A good answer might look like this:

• Scope: Tomato seedlings of one variety, grown for four weeks under the stated light, soil, and watering conditions.
• Mechanism hypothesis: Fertilizer F supplies a limiting nutrient, increasing early stem growth without increasing plant death.
• Prediction: Compared with the standard fertilizer, F will increase median stem height after four weeks by at least 10%, while the death rate rises by no more than 2 percentage points.
• Auxiliary assumptions: Height is measured consistently, plants are assigned fairly, watering and light are comparable, and the chosen seedlings represent the stated population.
• Revision rule: If the growth threshold is missed or deaths cross the boundary, do not claim the planned improvement. Inspect the nutrient mechanism, scope, and study assumptions, then test any revised claim with new plants.

Your numbers may differ. The important test is structural: can you name an observation that would make the team revise the claim?

Connection to the Next Lesson

A risky hypothesis tells us what evidence could matter. It does not tell us whether a comparison isolates the cause.

The next lesson adds experiment design, controls, and confounding. That is where we ask whether the observed difference came from Fertilizer F, retrieval gating, or something else that changed at the same time.

Resources

• [ARTICLE] Scientific Method — Stanford Encyclopedia of Philosophy - How testability, confirmation, falsification, and actual scientific practice fit together.
• [ARTICLE] Science and Pseudo-Science — Stanford Encyclopedia of Philosophy - Popper's falsifiability criterion and why no single criterion captures every feature of science.
• [PAPER] Strong Inference - Competing hypotheses, decisive tests, and the discipline of exposing explanations to alternatives.
• [ARTICLE] Preregistration — Center for Open Science - Recording hypotheses, methods, outcomes, and analysis plans before observing results.

Key Takeaways

• A hypothesis teaches more when it rules out some possible observations before the result is known.
• A research question asks; a hypothesis proposes; a prediction exposes; a revision rule says how evidence should change the claim.
• Predictions depend on auxiliary assumptions about scope, measurement, comparison, and ordinary variation.
• A failed prediction demands diagnosis, but repeated ad hoc rescue removes the hypothesis's scientific risk.
• Exploration can generate valuable hypotheses; fresh evidence is what turns those hypotheses into serious tests.