Statistical Evidence Needs a Comparison

LESSON

Probability, Random Processes, and Statistical Thinking

009 25 min beginner

Statistical Evidence Needs a Comparison

By the end of this lesson, you will be able to...

Build a simple reference model for an observed success count.

Measure how surprising an observation is under that reference without treating surprisingness as proof.

Separate statistical evidence, practical effect size, and causal conclusions.

Idea in one sentence: An observation becomes evidence only when we say what would have happened under a credible comparison model.

Core Insight

Consider a notification team testing a small client change. Before the change, comparable requests acknowledged eventually at about 90%. After the change, a sample of 100 requests contains 96 acknowledgements.

The observed rate is:

\hat p=96/100=0.96

That is a six-percentage-point improvement over the reference rate of 90%. Is it evidence that the change helped?

The point estimate alone cannot answer. A sample can move above or below its process rate by chance. We need a comparison:

If the true success rate were still 90%, how often would a sample of 100 requests produce 96 or more successes?

This “what would happen if nothing meaningful changed?” model is a reference model. When it is used as a null hypothesis, it is often called a null model.

The comparison does not decide the truth for us. It tells us whether the observation is ordinary or unusual under a clearly stated alternative explanation. This is the central trade-off: a simple reference makes evidence inspectable, but every shortcut in that reference can distort the conclusion.

The Naive Evidence Claim

A teammate says:

“The new rate is 96%, so the change improved reliability.”

There are at least three missing questions:

Compared with what rate?
How much variation would a no-change process produce in 100 requests?
Could another change, such as traffic mix or an outage ending, explain the difference?

Without a reference, “96%” is a description. With a reference, it can become evidence. With a design that controls other explanations, it may support a causal conclusion. These are different levels of claim.

Build the Reference Model

Let $$Y$$ be the number of eventual acknowledgements in 100 requests. Under the no-change reference:

Y\sim\operatorname{Binomial}(n=100,p_0=0.90)

This model says:

there are 100 comparable trials;
each trial has success probability $$0.90$$ ;
the trials are independent enough for the binomial approximation;
the question is about the count of successes, not latency or retry work.

The expected count under the reference is:

\[ E[Y]=np_0=100(0.90)=90 \]

The observed count, 96, is six above that reference expectation. The effect size is therefore:

\[ 0.96-0.90=0.06 \]

The effect size tells us how large the observed difference is in the units the team cares about. It is not replaced by a tail probability.

A Worked Tail Calculation

To ask whether 96 is surprising under the reference, calculate:

P(Y\geq96\mid p_0=0.90)

For the binomial model:

\begin{aligned} P(Y\geq96) &=\sum_{k=96}^{100} \binom{100}{k}(0.90)^k(0.10)^{100-k}\\ &\approx0.0237 \end{aligned}

So, if the true rate were 90% and the model assumptions held, a result of 96 or more successes would occur about 2.4% of the time.

This tail probability is often called a p-value for a one-sided test. Its precise meaning is:

How often would data at least this extreme appear under the reference model?

It is not:

the probability that the reference model is true;
the probability that the change caused the improvement;
the probability that the result will repeat tomorrow;
the size of the improvement.

The result is evidence against the reference model at a commonly used 5% threshold, but it is not a complete explanation. The sample could be unrepresentative, requests could be dependent, or another operational change could have occurred at the same time.

Simulation Makes the Comparison Concrete

The same comparison can be made without summing a formula.

Set the reference success rate to $$0.90$$ .
Generate 100 independent success/failure outcomes.
Count the successes.
Repeat this many times.
Record the fraction of simulated samples with at least 96 successes.

A large simulation should produce a tail frequency near 0.024. The simulated distribution might look like:

Success count	Interpretation under $$p_0=0.90$$
85	ordinary lower variation
90	near the reference expectation
94	somewhat high
96 or more	uncommon upper tail

The exact counts vary with the random seed. The point is to build a reference distribution of possible observations before deciding how unusual the real observation is.

This is the same simulation habit used in lesson 005, now with a comparison question:

Do not simulate the observed result after the fact. Simulate what the reference process would produce.

Choose the Comparison Carefully

A reference model is not automatically credible because it has a formula. Possible references include:

a historical rate from comparable traffic;
a control group measured during the same period;
a service-level target such as 90%;
a randomized assignment that receives no change;
a mechanistic model derived from earlier probability assumptions.

The comparison should match the claim's scope. A 90% rate from last year may be a poor reference if traffic, clients, or server architecture changed. A control group can be stronger, but it must be comparable and measured with the same outcome definition. A target is useful for policy, but it is not evidence that the system naturally produces that rate.

Write the reference before inspecting the result when possible. Otherwise, it is easy to choose a baseline that makes the observed number look impressive.

Statistical and Practical Significance

The 6-point improvement is practically meaningful only if six percentage points matter for users, capacity, or risk. A very large sample can make a tiny difference statistically surprising:

90.00% versus 90.10% may be detectable with millions of observations.
90% versus 96% may matter operationally even with a modest sample.

Report both:

Effect size: how much the observed metric changed.
Evidence against the reference: how unusual that change is under the model.

Do not let a threshold replace judgment. A p-value just below 0.05 is not a command to ship, and one above 0.05 is not proof that two systems are equivalent. The decision also needs cost, risk, measurement quality, and a plan for further evidence.

Direction and Optionality

The question determines the tail.

If the team only cares whether the change improved success, use an upper-tail question such as $P(Y\geq96)$ . If any change matters, including a serious regression, use a two-sided comparison that treats unusually low and unusually high results as evidence.

Choosing the direction after seeing the data makes the evidence look stronger than the original question justified. Pre-registering or writing down the comparison first is a practical way to reduce this flexibility.

Dependence and Repeated Peeking

The binomial reference assumes independent comparable trials. It can fail when requests share:

a degraded service state;
a tenant, region, or deployment;
a queue burst;
a retry chain where one failure causes another observation;
a user or device that contributes many correlated requests.

Dependence can make the observed 96 look more or less surprising than the independent model predicts. A p-value from the wrong reference distribution is not repaired by more decimal places.

Repeatedly checking the dashboard and stopping when the number crosses a threshold creates another problem. The more opportunities we give a random process to produce an extreme result, the more often some result will look surprising. Define the sampling window and stopping rule before reading the result, or use a method designed for sequential monitoring.

Common Confusions

Confusion: A p-value is the probability that the null is true

Why it is tempting: the number is between 0 and 1 and is described as evidence.

Better model: it is a probability of data under the reference model, not a probability of the model itself.

Confusion: Statistical significance proves causality

Why it is tempting: an unusual result feels like an explanation.

Better model: a comparison can show that data are unusual under one reference. Causal support also needs control of alternative explanations, measurement quality, and a sound intervention design.

Confusion: A non-significant result proves no effect

Why it is tempting: the comparison did not cross a chosen threshold.

Better model: the sample may be too small, noisy, biased, or poorly measured to distinguish the effect from ordinary variation.

Confusion: The reference must be zero

Why it is tempting: “no effect” sounds like no successes.

Better model: the reference is the expected process without the claimed change. For reliability, it may be 90%, not 0%.

Check Your Understanding

Check: What question does $P(Y\geq96\mid p_0=0.90)$ answer?

Think first, then reveal.

Answer: It asks how often a no-change process with a 90% rate would produce at least 96 successes in 100 trials. It does not give the probability that the change caused the result.

Check: Why report the six-percentage-point effect size as well as the tail probability?

Think first, then reveal.

Answer: The tail probability describes unusualness under a reference. The effect size describes the magnitude in operational units. A decision needs both.

Check: What happens if the 100 observations are strongly clustered in one outage or tenant?

Think first, then reveal.

Answer: The independent binomial reference may understate uncertainty. The observations carry less independent information than their row count suggests.

Practice: Compare a Batch Policy

A batch team changes a scheduler. Before the change, 75 of 100 jobs met the deadline. After the change, 82 of 100 meet it.

Write a short evidence report that includes:

A reference model.
The observed effect size.
A comparison question.
One reason a simple binomial calculation might be misleading.
One follow-up measurement or experiment.

Model answer

Use $$p_0=0.75$$ as the reference if the pre-change jobs are comparable and the outcome definition is unchanged. The observed effect is $$0.82-0.75=0.07$$ , a seven-percentage-point increase. The comparison question is: “Under a 75% no-change process, how often would 82 or more on-time jobs appear in 100?” A simple binomial calculation may be misleading if jobs share a queue, workload burst, or infrastructure incident. Follow up with a comparable control period or randomized scheduler assignment, and record workload, queue depth, missing jobs, and job identity so dependence and selection can be checked.

The report should not say that the scheduler caused the improvement solely because 82 exceeds 75. It should state what the reference makes unusual and what alternative explanations remain.

Connections

Lesson 008 showed how an interval describes uncertainty around an estimate. This lesson adds a reference distribution and asks whether an observation is surprising relative to it. Lesson 010 will stress the comparison further: heavy tails and mixtures can make a single average or reference summary fragile.

The track is building a disciplined sequence:

\text{model}\rightarrow\text{sample}\rightarrow\text{estimate}\rightarrow\text{interval}\rightarrow\text{comparison}

Each step adds information, but each step also adds assumptions that must be checked.

Resources

[COURSE] MIT OpenCourseWare: Introduction to Probability - Focus: Connect binomial models, sampling variation, and reference probabilities.
[REFERENCE] MIT 18.05: Probability Terminology and Examples - Focus: Review null models, tail probabilities, and the limits of p-value interpretation.
[COURSE] MIT OpenCourseWare: Fundamentals of Probability - Focus: Follow the optional formal path for hypothesis tests and comparison distributions.

Key Takeaways

Statistical evidence is relational: compare the observation with a credible reference model.
A tail probability measures how unusual the data are under that reference; it is not the probability that the reference or a causal story is true.
Report effect size and evidence together.
Reference quality, dependence, stopping rules, and measurement boundaries can dominate the arithmetic.
A result that crosses a threshold is a reason to investigate and decide, not a substitute for explanation or judgment.

← Back to Probability, Random Processes, and Statistical Thinking

← Back to Mathematics And Formal Foundations

← Back to Learning Hub