Day 377: Causal Inference - Interventions, Counterfactuals, and Confounding

The core idea: Causal inference asks what would happen if we changed an action on purpose, not merely what tends to appear alongside an outcome in historical data.

Today's "Aha!" Moment

In 08.md, Harbor City used sensitivity analysis to learn that West Tunnel debris buildup and household departure delay are the assumptions most capable of flipping the flood-response recommendation. That was useful, but it did not yet answer the mayor's actual question. The city is not choosing which variable looks important inside the model. It is choosing actions: should it fund pre-storm debris crews at the tunnel mouth, send evacuation texts thirty minutes earlier, or spend the money on another pump-maintenance cycle instead?

Historical data seems to offer a quick answer, and it is the wrong one. Days with extra debris crews often still have the worst flooding and the earliest tunnel closures. If the city compares "crew days" to "no-crew days" naively, it can end up concluding that the crews made things worse. In reality, the crews were dispatched precisely when the forecast was severe and when the Seawall District was already most exposed. The bad outcome and the intervention share the same upstream causes.

Causal inference is the discipline of separating those stories. It asks a counterfactual question: for storms with comparable exposure, what would Harbor City have observed if it had dispatched the crews, and what would it have observed if it had not? Because the city never gets to run the exact same storm twice, it has to recover that comparison through design, assumptions, and careful adjustment for confounders.

That shift matters because operations teams, policy teams, and product teams all make decisions about interventions rather than correlations. Sensitivity analysis showed Harbor City's leverage points. Causal inference decides which of those levers are real. The lesson after this, 10.md, picks up from there and deals with the next problem: how to communicate causal estimates honestly when they still come with uncertainty bands and assumption risk.

Why This Matters

Harbor City's resilience budget is finite. If the city mistakes association for causation, it can penalize the very interventions that are helping during the worst storms and overfund variables that only look predictive because they travel with hidden context. That is not a statistical nicety; it is a production decision problem. Public warnings, staffing rules, and capital upgrades all become harder to justify when the evidence cannot distinguish "this happened before the outcome" from "this changed the outcome."

The same failure mode appears outside city planning. A backend team may believe retries improved latency because they were enabled during a quiet week, or conclude that an emergency throttling rule hurts throughput because it is only activated during incidents. In each case the intervention is correlated with the environment that triggered it. If you do not model that assignment process, the metric dashboard tells a misleading story with great confidence.

Causal inference gives Harbor City a disciplined workflow for turning observational evidence into decision support. The city must define the action, name the outcome, identify the confounders that influence both, and choose a design that can support the claim. The reward is not perfect certainty. The reward is that when Harbor City says "earlier alerts reduce households without a safe route by about this much under these conditions," the sentence refers to a defensible intervention effect rather than to a convenient coincidence.

Learning Objectives

By the end of this session, you will be able to:

1. Define a causal question precisely - Distinguish an intervention effect from a descriptive association and tie both treatment and outcome to a concrete decision.
2. Use counterfactual reasoning correctly - Explain why causal claims are about unobserved alternative worlds and how study design approximates them.
3. Diagnose confounding in practice - Identify pre-treatment variables, mediators, and overlap problems that can make an estimate unusable for real decisions.

Core Concepts Explained

Concept 1: Start with the intervention, not the dataset

Harbor City's first causal question is narrower than it sounds: "If the city dispatches a pre-storm debris crew to West Tunnel six hours before landfall, how much later does the tunnel become unsafe for buses?" That wording already improves the analysis. The treatment is specific, the timing is explicit, and the outcome is tied to a real operational threshold. A vague question such as "do better-prepared districts flood less?" mixes many actions together and leaves no stable intervention to estimate.

Once the intervention is clear, the city has to separate causes of treatment assignment from effects of the treatment itself. Severe surge forecasts increase the chance that the city dispatches a crew. Those same forecasts also increase the chance of early tunnel closure. District exposure works the same way: low-lying blocks receive more attention and also suffer worse outcomes. Those are confounders because they push both the intervention and the outcome.

An ASCII graph makes the mechanism visible:

forecast_severity --------> crew_dispatch --------> tunnel_closure_time
       |                         |                          |
       v                         v                          v
district_exposure ----------> debris_level ----------> safe_route_loss
       \__________________________________________________/

This graph explains why raw comparisons fail. If Harbor City estimates the effect of crew_dispatch by comparing all treated storms against all untreated storms, the estimate absorbs both the effect of the crew and the effect of being in worse storms or worse locations. The point of a causal design is to block those backdoor paths by conditioning on the right pre-treatment variables, randomizing when possible, or exploiting a natural experiment with comparable groups.

The trade-off is precision versus relevance. A tightly defined intervention such as "dispatch one tunnel crew six hours before landfall" supports a cleaner estimate than a fuzzy category like "prepared response," but it answers a narrower question. That is usually a good trade in production work. Decision-makers need claims that can survive contact with a real operating procedure.

Concept 2: Counterfactuals are the target; data is only the evidence

Even with the right graph, Harbor City still faces the fundamental causal problem: the city never observes both worlds for the same storm. For a given event, the district either received an early evacuation text or it did not. The city cannot watch the same residents react to both timelines under identical weather, road conditions, and pump status. The missing outcome in the unchosen world is the counterfactual.

That is why causal estimands are written in intervention language rather than in observational language. When Harbor City asks about earlier alerts, the target is the difference between two hypothetical averages:

average effect of earlier alert
= E[households_without_safe_route | do(alert_time = early)]
  - E[households_without_safe_route | do(alert_time = standard)]

The do(...) notation means "set the policy deliberately" rather than "look at rows where the policy happened to be this value." That distinction matters because the observational distribution may have been shaped by hidden triage rules. Perhaps early alerts are only sent when forecasters already expect severe overtopping. In that case the raw average under alert_time = early is not the same thing as the average the city would see if it adopted early alerts as a policy across comparable storms.

Study design is what makes the counterfactual comparison believable. For evacuation texts, Harbor City could run a randomized pilot during low-risk drills or in neighborhoods where ethics and safety allow controlled timing differences. For debris crews, randomization may be impossible during an active storm season, so the city may need a quasi-experimental design based on roster rotations, forecast thresholds, or matched storm events with strong pre-treatment covariates. The estimate is only as strong as the story for why treated and untreated cases are comparable after adjustment.

The trade-off here is realism versus identifiability. Observational data reflects real operations, but it usually comes with messy treatment assignment. Randomized data gives cleaner identification, but only for interventions that can be randomized safely and politically. Mature teams use both: experiments where they can, and explicitly defended observational designs where they cannot.

Concept 3: Confounding is a design problem, not a regression problem

Once Harbor City begins estimating effects, a tempting mistake appears: pour every available column into a model and hope the coefficient becomes causal. That does not work. Some variables are useful confounders, some are mediators on the path from intervention to outcome, and some are colliders that create bias if conditioned on. Causal inference requires deciding which is which before fitting the estimator.

Suppose the city wants the total effect of sending an earlier evacuation text on households_without_safe_route. Forecast severity, neighborhood elevation, road capacity, and pump availability are plausible pre-treatment confounders. Departure delay is different. It sits on the causal path from alert timing to route loss. If Harbor City controls for departure delay while estimating the total effect of the alert, it partially blocks the very mechanism the city is trying to measure. The estimate then answers a narrower direct-effect question, not the operational question leadership asked.

Harbor City therefore needs a workflow that treats identification as an engineering artifact:

1. Define the treatment, outcome, and decision window precisely.
2. Draw the assumed causal graph and mark which variables exist before treatment assignment.
3. Choose the smallest credible adjustment set that blocks confounding without conditioning on mediators or colliders.
4. Check overlap: do comparable treated and untreated cases actually exist across the relevant risk range?
5. Estimate the effect, then stress-test it with robustness checks and hidden-confounding sensitivity analysis.

This is where causal inference reconnects to production relevance. If overlap is poor because every high-risk neighborhood always gets the early alert, the city may not be able to estimate the policy effect for those neighborhoods from existing logs. That is not a software bug in the model; it is a limitation of the data-generating process. Harbor City may need a different estimand, a different policy rollout, or a different source of evidence. Good causal work tells you when the answer is unsupported, not just when it is inconvenient.

The trade-off is organizational discipline. A causal estimate that can survive scrutiny usually costs more thought upfront than a dashboard correlation or a regression summary. But that effort prevents a more expensive failure later: committing money, policy, or operational trust to an action that looked good only because the assignment process was hidden.

Troubleshooting

Issue: The estimated effect of debris crews turns negative even though field staff insist the crews help.

Why it happens / is confusing: Harbor City may be comparing severe-storm deployments against mild-storm non-deployments, so the treatment is acting as a proxy for storm severity instead of being isolated from it.

Clarification / Fix: Rebuild the design around comparable storms or a credible adjustment set. If the city cannot explain why treated and untreated cases are comparable after conditioning, the negative sign is not interpretable as a causal effect.

Issue: Adding more variables to the regression keeps changing the answer.

Why it happens / is confusing: Some of those variables may be mediators or colliders rather than confounders. More controls do not guarantee less bias.

Clarification / Fix: Draw the causal graph explicitly and decide whether the goal is a total effect or a direct effect. In the alert example, do not control for departure delay if the city wants the total impact of earlier alerts on safe-route availability.

Issue: The city has no untreated comparison for the highest-risk neighborhoods because it always sends them the earliest possible warning.

Why it happens / is confusing: This is an overlap problem. The policy is already deterministic in the most critical region, so observational data contains no comparable untreated cases there.

Clarification / Fix: Narrow the claim to a population with overlap, redesign the rollout to create variation safely, or use a different identification strategy. If none of those are possible, Harbor City should state that the desired effect is not identified from the available data.

Advanced Connections

Connection 1: Sensitivity Analysis ↔ Causal Inference

Sensitivity analysis in 08.md showed Harbor City which assumptions most strongly move tunnel closure and route loss inside the model. Causal inference asks a different production question: which real interventions can reliably move those same outcomes in the city itself? Sensitivity tells Harbor City where the decision is exposed; causal inference tells it which levers are defensible to pull.

Connection 2: Causal Inference ↔ Uncertainty Communication

Even a well-identified estimate is not a single magical number. Harbor City still has uncertainty from sampling, model choice, and untestable assumptions about hidden confounding. That is why the next lesson, 10.md, moves from "is this an intervention effect?" to "how should we communicate its range, conditions, and residual uncertainty so decision-makers do not overread it?"

Resources

Optional Deepening Resources

• [BOOK] Causal Inference: What If
  • Focus: A rigorous but practical treatment of interventions, exchangeability, positivity, and causal estimands from a production decision perspective.
• [PAPER] Causal Diagrams for Empirical Research
  • Focus: Judea Pearl's foundational explanation of how directed acyclic graphs expose confounding, mediation, and admissible adjustment sets.
• [DOC] DoWhy Documentation
  • Focus: A concrete workflow for modeling assumptions, identifying estimands, estimating effects, and checking robustness in code.
• [BOOK] Causal Inference: The Mixtape
  • Focus: Applied causal-design patterns such as matching, difference-in-differences, and instrumental variables, with emphasis on interpretation rather than formula memorization.

Key Insights

1. A sensitive variable is not automatically a causal lever - Harbor City still has to ask whether a real intervention can move the outcome, or whether the variable only travels with deeper conditions.
2. Counterfactuals define the claim - The target is always a comparison between alternative worlds created by a deliberate intervention, even though the evidence comes from incomplete observed data.
3. Confounding lives in the assignment process - If treatment choice depends on the same forces that drive the outcome, no amount of naive correlation analysis will recover the policy effect.