Samples Change Even When the Process Does Not

LESSON

Probability, Random Processes, and Statistical Thinking

007 25 min beginner

Samples Change Even When the Process Does Not

By the end of this lesson, you will be able to...

Distinguish a process parameter from a statistic computed from one sample.

Explain sampling variation, estimator bias, and standard error with a small example.

Predict how sample size and dependence change the reliability of an estimate.

Idea in one sentence: A sample statistic is evidence about a process, not a transparent window into it, so different samples can give different answers even when the process stays fixed.

Core Insight

Consider the notification service model from the previous lessons. Under the healthy/degraded state model, the probability of eventual acknowledgement is:

\[ p=P(S=1)=P(A)+P(TA)=0.86+0.088=0.948 \]

The team cannot observe the whole process at once. It sees samples of requests. One engineer inspects 20 requests and finds 20 acknowledgements. Another inspects a different 20 and finds 18. Their estimates are:

\hat p_1=20/20=1.00,\qquad \hat p_2=18/20=0.90

The two samples disagree. That disagreement does not automatically mean the service changed between observations. The samples may simply contain different random outcomes.

This creates a new layer of uncertainty:

The process has a parameter, such as the long-run success probability $$p$$ .
A sample is a finite set of observed runs from that process.
A statistic is a calculation from that sample, such as $\hat p$ .
An estimator is a statistic used to learn about an unknown parameter.

The process can remain stable while the estimator moves.

Process, Parameter, Sample, Statistic

Keep the four objects separate.

Object	Meaning	Notification example
Process	The uncertain mechanism producing requests	Healthy/degraded service plus retry rule
Parameter	A fixed property of that mechanism	$$p=0.948$$ , the long-run acknowledgement rate
Sample	The finite runs we happened to observe	20 requests from one time window
Statistic	A number calculated from those runs	18 acknowledgements out of 20

The parameter is fixed inside a model, even if we do not know its value in a real system. The statistic is random because a new sample can produce a new value.

For binary outcomes, the sample proportion is:

\hat p=\frac{\text{number of successes}}{n}

The hat means “this is an estimate,” not “this is the parameter itself.”

Plain meaning:

$\hat p$ reports what happened in this sample.

Technical name:

An estimator is a rule that maps a sample to an estimate of a parameter. The sampling distribution is the distribution of that estimator across repeated samples of the same size from the same process.

Thinking about the sampling distribution prevents a common mistake: treating one observed statistic as if it were the only value the process could have produced.

A Worked Path: Two Samples From One Process

Assume requests are comparable and independent for this calculation, with the process parameter $$p=0.948$$ . Take two samples of $$n=20$$ .

Sample 1

All 20 requests are acknowledged:

\hat p_1=20/20=1.00

This is possible even though the underlying rate is 0.948. A lucky sample contains no failures.

Sample 2

Eighteen requests are acknowledged:

\hat p_2=18/20=0.90

This is also possible. A less lucky sample contains two failures.

Neither sample proves that the parameter equals its observed proportion. The estimates differ because the sample is finite.

The naive interpretation says:

“The service was perfect in the first window and only 90% reliable in the second.”

The stronger interpretation says:

“These are two noisy estimates of a stable process parameter. We need more evidence and an assumption check before calling the difference a process change.”

The worked path is:

State the process and parameter.
Draw a fixed-size sample.
Compute $\hat p$ .
Repeat the sampling mentally or with simulation.
Describe how much $\hat p$ can move before claiming that the process changed.

Sampling Variation Has a Scale

For an independent binary sample, the standard error of the sample proportion is approximately:

\operatorname{SE}(\hat p)=\sqrt{\frac{p(1-p)}{n}}

Using $$p=0.948$$ and $$n=20$$ :

\operatorname{SE}(\hat p) =\sqrt{\frac{0.948(0.052)}{20}} \approx0.050

The value $$0.050$$ is the typical scale of movement for the estimator across samples of 20 under this model. It is not the probability that the estimate is wrong, and it is not a guarantee that every estimate lies within 0.05 of $$p$$ .

In practice, the true $$p$$ is usually unknown. A common plug-in estimate uses the observed $\hat p$ :

\widehat{\operatorname{SE}}(\hat p) =\sqrt{\frac{\hat p(1-\hat p)}{n}}

For $\hat p=0.90$ and $$n=20$$ , this gives approximately $$0.067$$ . The plug-in value is itself uncertain, especially for small samples. Later lessons will use intervals to communicate this uncertainty more honestly.

The sample size changes the scale:

Sample size	Approximate standard error using $$p=0.948$$
20	0.050
80	0.025
320	0.0125

Multiplying the sample size by four roughly halves the standard error. Doubling the sample does not halve it. More data helps, but the improvement follows a square-root relationship.

Bias and Variation Are Different Problems

An estimator can miss the parameter for two different reasons.

Variation

The estimator is centered correctly over repeated representative samples, but each finite sample moves around that center. The sample proportion is unbiased under independent, representative sampling:

E[\hat p]=p

That does not mean every $\hat p$ equals $$p$$ . It means that the average of the estimator over many same-sized samples equals the target parameter.

Bias

The estimator is systematically shifted because the sampling procedure favors some outcomes or the measurement rule is wrong.

Suppose the team estimates reliability using only requests sent during healthy-looking periods. Degraded periods are missing from the sample. The resulting estimate may be high again and again, even if the sample is very large. More observations from the same biased window reduce random variation but do not remove the bias.

This is the key distinction:

More data can reduce variance.
More data from the wrong sampling process can preserve bias.

The phrase “large sample” is therefore incomplete. Ask: large and representative of what?

Standard Deviation Is Not Standard Error

The individual success indicator $$S$$ varies from request to request. Its standard deviation under $$p=0.948$$ is:

\operatorname{sd}(S)=\sqrt{p(1-p)} =\sqrt{0.948(0.052)}\approx0.222

The standard error of $\hat p$ is smaller because $\hat p$ averages $$n$$ observations:

\operatorname{SE}(\hat p)=\frac{\operatorname{sd}(S)}{\sqrt n}

For $$n=20$$ , $0.222/\sqrt{20}\approx0.050$ .

Use the words carefully:

Standard deviation: spread of individual outcomes.
Standard error: spread of an estimator across repeated samples.

Confusing them can make a report sound much more certain or much more uncertain than it is.

Dependence Changes the Effective Sample

The simple standard-error formula assumes comparable, independent observations. Lesson 004 showed why requests can share a degraded service state. If ten requests arrive during the same outage, they carry related information. Counting them as ten independent pieces of evidence exaggerates how much the sample has taught us.

The sample may contain 100 rows but less information than 100 independent rows. A rough symptom is that failures cluster in time, region, tenant, or deployment state.

This creates another trade-off:

Collecting more observations is cheaper than modeling dependence, but pretending correlated observations are independent produces an error bar that is too small.

Possible responses include sampling across independent time windows, grouping by incident, adding state variables, or using a model that accounts for correlation. The right response depends on the operational question; simply deleting inconvenient observations is not a repair.

Common Confusions

Confusion: The parameter changes whenever the sample statistic changes

Why it is tempting: the dashboard number moved, so the process seems to have moved.

Better model: a statistic is random across samples. Investigate time, state, and measurement changes before inferring a parameter change.

Confusion: Unbiased means accurate on every sample

Why it is tempting: “unbiased” sounds like “correct.”

Better model: unbiased describes the center of the sampling distribution, not each individual estimate. A particular sample can still be far from the parameter.

Confusion: A large sample removes bias

Why it is tempting: more observations usually make estimates look stable.

Better model: more data reduces random variation only when the sampling and measurement process is appropriate. A large biased sample can be confidently wrong.

Confusion: Standard error is the probability of an error

Why it is tempting: both are reported as a single number.

Better model: standard error is a scale for estimator variation. It needs an interval or probability statement before it can support a decision.

Check Your Understanding

Check: Two teams measure 20 requests and report 1.00 and 0.90 success rates. What is the first explanation to consider?

Think first, then reveal.

Answer: Sampling variation from two finite samples of the same process. Check the sampling windows and dependence before declaring a process change.

Check: What happens to standard error when the sample size is multiplied by four?

Think first, then reveal.

Answer: Under the simple independent model, it is roughly halved because standard error scales as $1/\sqrt n$ .

Check: A team collects a million requests only during healthy periods. What problem can remain?

Think first, then reveal.

Answer: Sampling bias. The estimate can be very stable while excluding the degraded states the reliability claim is supposed to cover.

Practice: Compare Two Sampling Plans

A batch system has a true on-time probability of $$p=0.75$$ . Compare:

Plan A: observe 40 jobs chosen across the whole month.
Plan B: observe 400 jobs from the first hour after each successful deployment.

For each plan, identify one source of variation and one possible source of bias. Which plan better supports a monthly reliability claim, and what additional evidence would you request?

Model answer

Plan A has sampling variation because 40 jobs are finite, but spreading them across the month can represent different loads and states. Plan B has lower random variation from its larger nominal sample, but it may be biased toward healthy post-deployment periods and miss later degradation or high-load windows. Plan A is closer to the monthly claim if the selection is genuinely representative. I would request the sampling rule, timestamps, load or state labels, missing jobs, and a repeated-sample comparison. If Plan B is operationally important, report it as a post-deployment metric rather than silently treating it as the monthly parameter.

Connections

Lessons 001–006 described a probability process and checked its assumptions. This lesson starts the second layer: evidence is a sample from that process. Lesson 008 will turn estimator variation into intervals, and later lessons will add comparison models, heavy tails, and measurement noise.

The question has changed from “What could happen under this model?” to “What can this finite set of observations justify saying about the model?”

Resources

[COURSE] MIT OpenCourseWare: Introduction to Probability - Focus: Connect random variables and repeated samples to sampling distributions and estimator variation.
[REFERENCE] MIT 18.05: Probability Terminology and Examples - Focus: Review sample proportions, expectation, variance, and standard-error intuition.
[COURSE] MIT OpenCourseWare: Fundamentals of Probability - Focus: Follow the optional formal path for laws of large numbers and sampling.

Key Takeaways

A parameter describes the process; a statistic describes the sample; an estimator uses the statistic to learn about the parameter.
Sampling variation can move estimates even when the process is unchanged.
Bias is systematic displacement from the target; variance is random movement around the target.
Standard error describes estimator spread and usually shrinks like $1/\sqrt n$ under independent sampling.
Larger samples reduce random variation but do not repair biased selection, bad measurement, or hidden dependence.

← Back to Probability, Random Processes, and Statistical Thinking

← Back to Mathematics And Formal Foundations

← Back to Learning Hub