Heavy Tails Make the Average Fragile

LESSON

Probability, Random Processes, and Statistical Thinking

010 25 min beginner

Heavy Tails Make the Average Fragile

By the end of this lesson, you will be able to...

Recognize a mixture with a rare, expensive tail event.

Compare mean, median, quantiles, and tail probabilities on the same data.

Choose a summary that matches a capacity, reliability, or user-impact decision.

Idea in one sentence: When a small number of extreme outcomes carry a large share of the cost, the mean can move dramatically while the typical experience barely changes.

Core Insight

Consider a service that records 1,000 request latencies. Most requests are routine, but a few encounter a slow dependency:

Latency	Number of requests
100 ms	990
200 ms	9
10,000 ms	1

The distribution has a common mode and a rare slow mode. It is a simple mixture: observations come from more than one operating condition.

The mean latency is:

\frac{990(100)+9(200)+1(10{,}000)}{1000} =\frac{110{,}800}{1000}=110.8\text{ ms}

The median is 100 ms. At least 99% of requests finish in 200 ms, while one request takes 10 seconds.

Which number should the dashboard show?

There is no universally correct summary. The mean is useful for total compute or average resource demand. The median describes a typical request. A tail quantile or tail probability is better for an experience guarantee. The rare 10-second event matters greatly if it causes a user-visible timeout, even though it is only one request in 1,000.

This is the pressure behind the lesson:

A summary that is adequate for average work can be unsafe for tail risk.

Where the Mean Starts to Struggle

Remove the one 10,000 ms request and recompute:

\frac{990(100)+9(200)}{999} =\frac{100{,}800}{999}\approx100.9\text{ ms}

One rare observation moved the mean from about 100.9 ms to 110.8 ms, a change of nearly 10%. The median stayed 100 ms in both cases. The routine population did not change; the sample happened to include one extreme event.

This does not make the mean “wrong.” It tells us what the mean is sensitive to. The average includes every cost, so a rare event with a very large magnitude can dominate it.

The same pattern appears in:

incident recovery cost;
file sizes and response payloads;
queue wait times;
job completion delays;
financial loss or demand spikes.

In each case, ask whether a rare mode is part of the process rather than treating it as a measurement error automatically.

A Worked Comparison of Summaries

Use the 1,000 latency observations.

Mean

\bar L=110.8\text{ ms}

Interpretation: if all request time were pooled and divided evenly, the average request accounts for 110.8 ms. This can help estimate total CPU, network, or dependency work.

Median

The 500th and 501st sorted observations are both 100 ms, so the median is 100 ms.

Interpretation: at least half of requests are no slower than 100 ms. It is robust to the one extreme event, but it says nothing about the worst 1% directly.

Quantile

At least 99% of the observations are at or below 200 ms. We can report an empirical 99th-percentile threshold of approximately 200 ms using a stated quantile convention.

Interpretation: the upper tail up to the 99th percentile is visible, but the single 10,000 ms request sits beyond that threshold. If the user promise concerns the worst one in a thousand, p99 may be too coarse; p99.9 or a direct tail count may be more appropriate.

Tail probability

P(L\geq10{,}000\text{ ms})=1/1000=0.001

Interpretation: under this observed window, one in 1,000 requests reached the extreme mode. A tail probability is often easier to connect to an SLO or incident budget than a raw maximum.

The summaries answer different questions. Replacing one with another without naming the decision is a modeling error.

Mixtures Explain Why the Tail Appears

Imagine a hidden operating mode $$M$$ :

$$M=N$$ : normal dependency, producing roughly 100 ms responses.
$$M=S$$ : slow dependency, producing hundreds or thousands of milliseconds.

The observed latency distribution is a mixture:

P(L\in A)=P(L\in A\mid M=N)P(M=N) +P(L\in A\mid M=S)P(M=S)

You do not need to estimate every hidden state to use the idea. The point is that one overall average can combine different regimes:

a normal mode with high frequency;
a slow mode with low frequency but high cost.

If the slow mode becomes more common, the mean, tail probability, and queue behavior can all change. If it is caused by a shared outage, observations in the slow mode will also be dependent. A single “latency distribution” can therefore hide both a mixture and a time-dependent process.

Why Samples of the Tail Are Unstable

Suppose the extreme mode occurs once per 1,000 requests on average. A 100-request window will often contain no extreme event. If it contains one, its mean can jump sharply.

Two windows can therefore report:

Window A: 100 requests, all near 100 ms, mean near 100 ms.
Window B: 99 requests near 100 ms and one at 10,000 ms, mean near 199 ms.

The underlying process may be unchanged. The windows differ because rare events are not evenly distributed across small samples.

This is a different issue from the sampling variation of a proportion. Here, the magnitude of one observation is also extreme. Means and variances can become unstable, and a confidence interval based on a thin-tailed approximation may be misleading.

The practical response is not to hide the tail. It is to collect enough tail observations, stratify by operating mode, and report the summary that matches the risk.

Choose the Summary for the Decision

Decision	Useful first summary	Why
Estimate total average work	Mean	Every unit of work contributes
Describe a typical request	Median	Resistant to a few extremes
Set a user latency objective	p95, p99, or a stated tail probability	Focuses on the upper experience
Protect against catastrophic slowdowns	Tail count, maximum with context, incident-mode rate	Rare events drive harm
Compare modes	Separate distributions by state or segment	Avoids mixing unlike processes

No summary should be reported without its sample window and inclusion rule. A p99 over all traffic can conceal a p99.9 problem for a critical tenant. A mean over successful requests can hide failed requests entirely.

The Trade-off: Robustness Versus Sensitivity

Median and quantiles are less sensitive to extreme values, which makes them stable for typical experience. That robustness can also hide changes in total work or a rare failure mode. The mean preserves those costs but becomes fragile when the tail is heavy.

This trade-off is not solved by choosing one “best” statistic. Report a small set with distinct jobs:

one typical-value summary;
one tail summary;
the count or rate of rare events;
the mean when average resource demand matters.

The extra reporting costs dashboard space and interpretation effort. The benefit is that no single number is asked to carry incompatible promises.

Common Confusions

Confusion: Heavy-tailed means every value is huge

Why it is tempting: the phrase sounds like the whole distribution is large.

Better model: most observations can be ordinary while a small tail contains values much larger than the center. The tail's magnitude and probability both matter.

Confusion: The median ignores the problem

Why it is tempting: the median remains 100 ms even when one request takes 10 seconds.

Better model: the median answers a typical-case question. Pair it with a tail metric when rare harm matters.

Confusion: Removing outliers always improves the data

Why it is tempting: extreme values make the average look inconvenient.

Better model: remove a value only when it is a measurement error or outside the decision boundary. A real slow request may be the event the system must protect against.

Confusion: A high p99 means the mean must also be high

Why it is tempting: both summarize the same latency list.

Better model: the mean weights magnitude and frequency; a quantile ranks observations. They can move in different directions.

Check Your Understanding

Check: In the 1,000-request example, why does one 10,000 ms request move the mean much more than the median?

Think first, then reveal.

Answer: The mean includes the magnitude of every observation, so the extreme value contributes 10,000 ms. The median depends on the middle rank, which remains among the 100 ms requests.

Check: Which is more direct for a promise that at least 99% of requests finish below 200 ms: the mean or the empirical 99th-percentile threshold?

Think first, then reveal.

Answer: The percentile threshold, together with its sample window and uncertainty. The mean does not encode a threshold guarantee.

Check: Should the 10-second request be removed automatically?

Think first, then reveal.

Answer: No. First decide whether it is a real request inside the model boundary. If it is real, it is evidence about tail risk, not a nuisance to delete.

Practice: Incident-Cost Summaries

An operations team records 100 incidents:

90 cost 1 hour of recovery;
9 cost 4 hours;
1 costs 60 hours.

Calculate the mean cost. Identify a robust typical summary and a tail summary. Then write one sentence explaining why a capacity plan and a resilience plan may need different summaries.

Model answer

The mean cost is:

\frac{90(1)+9(4)+1(60)}{100} =\frac{186}{100}=1.86\text{ hours}

The median is 1 hour because the middle incidents are routine. A useful tail summary is that 10% of incidents cost at least 4 hours, with one 60-hour event in the sample. A capacity plan may use the mean to estimate total recovery effort, while a resilience plan should inspect the tail rate and the conditions that produce a 60-hour incident. The extreme incident should be investigated, not silently discarded.

Connections

Lessons 007–009 added sampling variation, intervals, and reference comparisons. Heavy tails show why those tools need distributional checks: a point estimate or interval based on a fragile mean can hide the regime that matters most to users.

Lesson 011 will move from process variation to measurement noise. That distinction matters here: a slow request may be a real tail event, a bad timestamp, or a mixture of both. The measurement boundary must be checked before changing the summary.

Resources

[COURSE] MIT OpenCourseWare: Introduction to Probability - Focus: Review distributions, quantiles, expectation, and the effect of rare outcomes.
[REFERENCE] MIT 18.05: Probability Terminology and Examples - Focus: Compare mean, variance, empirical distributions, and tail probabilities.
[COURSE] MIT OpenCourseWare: Fundamentals of Probability - Focus: Follow the optional formal path for mixtures, dependence, and random processes.

Key Takeaways

Heavy tails combine ordinary outcomes with rare values that can dominate cost or risk.
The mean, median, quantiles, and tail probabilities answer different questions.
A rare real event is not an outlier to delete merely because it makes the average inconvenient.
Small samples can miss a rare mode or overreact when one extreme appears.
Report summaries that match the decision and expose the tail rather than asking one average to carry every promise.

← Back to Probability, Random Processes, and Statistical Thinking

← Back to Mathematics And Formal Foundations

← Back to Learning Hub