Dependence Changes the Meaning of Evidence
LESSON
Dependence Changes the Meaning of Evidence
By the end of this lesson, you will be able to...
Calculate a conditional probability from a small joint table.
Distinguish independence from dependence and explain why shared causes create dependence.
Use Bayes rule as probability bookkeeping when new evidence changes a base rate.
Idea in one sentence: Evidence changes probability through a dependency model, so two observations cannot be multiplied safely until we know whether they share a cause.
Core Insight
Consider the notification client from the previous lessons: it sends an attempt, waits two seconds, and retries once after a timeout. Its possible complete histories are:
| Outcome | Meaning |
|---|---|
| A | first attempt acknowledged |
| TA | first attempt timed out, retry acknowledged |
| TT | both attempts timed out |
Lesson 001 used this toy distribution:
The operations team now sees a first timeout. It asks:
Given that the first attempt timed out, how likely is the second attempt to time out too?
The answer is not the unconditional probability of TT, which is \(0.04\). We have learned something: the run is no longer in the full sample space. It is in the smaller set of histories that begin with T.
That is the role of conditional probability. It tells us how likely one event is after we restrict attention to another event.
The Small Situation: Evidence Arrives in Stages
Define these events:
- \(R\): the client uses the retry, so the first attempt timed out.
- \(F\): the client fails completely, so the outcome is TT.
From the outcome table:
The probability of a retry is:
The probability of a complete failure is:
Once we know that a retry happened, only TA and TT remain possible. Of those retrying requests, \(0.04\) of all requests are TT and \(0.20\) of all requests use the retry. Therefore:
Read the notation as “the probability of \(F\), given \(R\).” The vertical bar means that \(R\) is the information we already have.
Plain meaning:
After seeing a first timeout, one in five retrying requests fails again in this toy model.
Technical name:
For events \(A\) and \(B\), conditional probability is:
The intersection \(A\cap B\) means that both events happen. We divide by \(P(B)\) because the new question is about the world in which \(B\) has already happened.
The Naive Multiplication
Suppose a teammate says:
“A timeout happens 20% of the time. Two timeouts should therefore happen with probability \(0.20\times0.20=0.04\).”
That calculation can be correct, but only under a model. It assumes that the second timeout probability remains \(0.20\) after the first timeout. In symbols:
For the toy distribution, the assumption happens to match the numbers:
But the arithmetic did not prove independence. It used independence as an assumption about the process.
This distinction matters because a shared cause can make observations move together. A congested service, a broken network route, or a regional outage can affect both attempts. After the first timeout, the second attempt may be much more likely to time out than it was before any evidence arrived.
A Worked Dependency Model: Healthy and Degraded Service
Make the shared cause explicit. Let the service be in one of two states during a request:
- \(D\): the service is degraded, with probability \(P(D)=0.20\).
- \(H\): the service is healthy, with probability \(P(H)=0.80\).
Assume the timeout probability for one attempt depends on that state:
| Service state | Probability of state | \(P(T\mid\text{state})\) |
|---|---|---|
| Healthy \(H\) | 0.80 | 0.05 |
| Degraded \(D\) | 0.20 | 0.50 |
The marginal probability of one timeout is found by considering both paths:
So a randomly chosen attempt times out 14% of the time.
Now compare two attempts from the same request. Within each state, suppose attempts are independent once the state is fixed. The service state is still shared, however.
The probability of two timeouts is:
If the attempts were independent based only on their marginal rate, we would predict:
The shared degraded state makes two timeouts much more common: \(0.052\), not \(0.0196\).
The conditional probability makes the change visible:
After seeing one timeout, the chance of a second timeout is about 37%, even though an unconditioned attempt times out only 14% of the time.
The evidence tells us something about the hidden state. A timeout is more plausible during degradation, so observing one timeout raises the probability that the service is degraded:
This is Bayes rule. It is not a magic predictor. It is a rearrangement of the joint-probability definition:
Plain meaning:
Start with a base rate for a possible cause, ask how likely the evidence is under that cause, then normalize across all possible causes.
In this example, one timeout moves the degraded-state probability from 20% to roughly 71%. That is why the next timeout is no longer an ordinary 14% event.
Independence Is a Model Claim
Two events \(A\) and \(B\) are independent when learning one does not change the probability of the other:
When \(P(B)>0\), an equivalent form is:
Independence is not the same as “the events look unrelated” or “the events happen at different times.” It is a claim about probabilities after conditioning.
Examples that may be approximately independent:
- Two random-number draws from a correctly implemented generator.
- Two requests routed to genuinely separate services with no shared capacity or incident.
Examples that are often dependent:
- Two alerts emitted by the same failing database.
- Two measurements taken by the same miscalibrated sensor.
- A retry immediately following a timeout from the same overloaded service.
- Two users affected by the same regional outage.
The word “may” matters. Independence must be justified by the process or checked against data. It should not be inserted merely because multiplication is convenient.
Evidence and Base Rates: A Test Example
Transfer the same bookkeeping to a defect test. Suppose:
- 10% of builds contain a defect: \(P(D)=0.10\).
- The test fails for 80% of defective builds: \(P(F\mid D)=0.80\).
- The test also fails for 10% of clean builds: \(P(F\mid\neg D)=0.10\).
What is the probability that a build is defective after the test fails?
First compute the total probability of a failure:
Then apply Bayes rule:
So a failing test makes a defect substantially more likely, but the probability is about 47%, not 80%. The 80% was the test's sensitivity, \(P(F\mid D)\). The question asks for the reverse direction, \(P(D\mid F)\). The base rate and false-positive path both matter.
The lesson is not “always distrust tests.” It is:
Name the direction of the conditional probability before interpreting evidence.
Trade-offs and Limits
Conditional probability gives a sharper answer after evidence arrives, but it also makes the model more expensive to state and maintain. This trade-off is deliberate: we define the conditioning event, preserve the direction of the question, and track shared causes that may change over time. An independence shortcut is cheaper to calculate, but it can understate correlated failures. Bayes bookkeeping does not identify the true cause by itself; it only updates the probabilities supplied by the model. The boundary appears when the state, base rate, or measurement process changes and the old conditional rates no longer describe current behavior.
Common Confusions
Confusion: \(P(A\mid B)\) is the same as \(P(B\mid A)\)
Why it is tempting: both expressions mention the same two events.
Better model: the condition changes the reference group. \(P(\text{failure}\mid\text{degraded})\) asks about failures among degraded runs; \(P(\text{degraded}\mid\text{failure})\) asks about degraded runs among failures.
Confusion: Multiplication proves independence
Why it is tempting: the product rule is short and familiar.
Better model: multiplication of marginals is valid only when independence is justified. With dependence, use \(P(A\cap B)=P(A)P(B\mid A)\).
Confusion: Independent means mutually exclusive
Why it is tempting: both ideas describe relationships between events.
Better model: mutually exclusive events cannot happen together. Independent events can happen together; learning one simply does not change the probability of the other. Non-trivial mutually exclusive events are dependent.
Confusion: A positive test result gives the test's accuracy as the posterior
Why it is tempting: sensitivity sounds like the probability we want.
Better model: keep the direction visible. Evidence updates a base rate through both true-positive and false-positive paths.
Check Your Understanding
Check: In the notification model, why is \(P(TT\mid R)=0.04/0.20\) instead of \(0.04\)?
Think first, then reveal.
Answer: Once a retry \(R\) is observed, the reference group contains only TA and TT requests. The conditional probability renormalizes the complete-failure cases within that smaller group.
Check: In the healthy/degraded model, why is \(P(T_2\mid T_1)\) greater than \(P(T_2)\)?
Think first, then reveal.
Answer: The first timeout is evidence that the shared service may be degraded. Degradation raises the chance of another timeout, so the conditional probability is larger than the marginal probability.
Check: A test fails. Which quantity answers “what is the probability the build is defective now”: \(P(F\mid D)\) or \(P(D\mid F)\)?
Think first, then reveal.
Answer: \(P(D\mid F)\). The condition is the evidence we observed: a failing test.
Practice: Audit an Alert Rule
An incident detector watches two alerts:
- \(A\): a timeout alert fires.
- \(B\): a queue-depth alert fires.
In a historical set of 1,000 comparable five-minute windows:
- Both alerts fired in 60 windows.
- \(A\) fired in 100 windows.
- \(B\) fired in 200 windows.
Answer:
- Estimate \(P(A)\), \(P(B)\), and \(P(A\cap B)\).
- Compare \(P(A\cap B)\) with \(P(A)P(B)\). Does the independence approximation look plausible?
- If \(A\) has already fired, estimate \(P(B\mid A)\).
- Name one shared cause that could explain the dependence and one additional measurement that would help distinguish it from coincidence.
Model answer
If the alerts were independent, the product of their marginal probabilities would be:
The observed joint rate, 0.06, is three times the independent estimate. That is evidence of dependence in this sample, although it is not by itself proof of a particular cause.
The conditional estimate is:
A shared overloaded database, a traffic burst, or a common downstream dependency could produce both alerts. Useful follow-up measurements include request volume, service state, deployment timing, and whether the alerts came from independent components. The extra measurement should test the proposed cause rather than merely add another correlated symptom.
Connections
Lessons 001–003 built outcomes, random variables, distributions, expectation, and variance. This lesson adds the rule for updating a probability after evidence arrives. Lesson 005 will use simulation to make these differences visible across repeated trials, and lesson 006 will ask the learner to audit the whole model.
The track uses Bayes rule here only as conditional-probability bookkeeping. Priors, posterior computation, and latent-variable inference depth belong in the downstream probabilistic-modeling track.
Resources
- [COURSE] MIT OpenCourseWare: Introduction to Probability - Focus: Review conditional probability, independence, and Bayes rule with small examples.
- [REFERENCE] MIT 18.05: Probability Terminology and Examples - Focus: Practice joint, conditional, and marginal probability tables.
- [COURSE] MIT OpenCourseWare: Fundamentals of Probability - Focus: Follow the optional deeper treatment of conditional expectation and dependence.
Key Takeaways
- Conditional probability changes the reference group after new evidence is observed.
- Independence is a model claim: learning one event leaves the probability of the other unchanged.
- Shared causes such as outages, load, or faulty instruments create dependence and make repeated evidence less surprising than an independent model predicts.
- Bayes rule keeps the direction of evidence visible and combines a base rate with likelihoods.
- Before multiplying probabilities, ask what can connect the events and which measurement would reveal that connection.
← Back to Probability, Random Processes, and Statistical Thinking