Calibration - Fitting Models to Real Data

Learning / Complexity Dynamics And Control / System Dynamics and Causal Modeling / 006

LESSON

System Dynamics and Causal Modeling

006 30 min intermediate

Day 374: Calibration - Fitting Models to Real Data

The core idea: Calibration turns a model from a plausible story into a constrained hypothesis by adjusting uncertain parameters until the model reproduces observed behavior, while keeping those parameters physically and operationally believable.

Today's "Aha!" Moment

In 05.md, Harbor City used parameter sweeps to discover that Seawall District's resilience plan becomes fragile when pump uptime falls, drain blockage rises, or residents start evacuating later than expected. That was a useful map of uncertainty, but it still left the city with an uncomfortable question: which parts of that map describe the real district, and which parts are only mathematically possible?

Calibration is how the city answers that question. The planning team pulls together tide-gauge data from the October king-tide flood, pump telemetry from the January rain-on-surge event, tunnel closure logs from the same storm, and shelter arrival timestamps from last year's district evacuation drill. Those observations are not decoration. They are the evidence the model must now explain. If the model can match the flood peak only by assuming near-perfect pumps that maintenance records contradict, or if it can match evacuation timing only by using implausible household delay distributions, the model is not "close enough." It is exposing a mismatch between mechanism and reality.

The key realization is that calibration is not about forcing every curve to line up as tightly as possible. A model can overfit historical traces and still be useless for the next storm. The real job is narrower and more demanding: identify which parameters are uncertain, decide which observations can constrain them, and search for parameter values that fit the data without smuggling in impossible assumptions. That is why calibration sits between parameter sweeps and validation. Sweeps show where the decision is sensitive. Calibration tells you which sensitive regions are actually supported by evidence.

Why This Matters

Harbor City is about to recommend one of three capital programs for Seawall District: a taller fixed seawall, a lower wall backed by pump upgrades, or a staged adaptation plan with selective buyouts and a future barrier expansion. The difference between those options now depends less on abstract model structure and more on a handful of uncertain quantities: how fast drains clog when debris accumulates, how often backup generators fail under flood conditions, when the West Tunnel becomes operationally unusable, and how long residents delay after an evacuation order.

If the model leaves those parameters at guessed values, the recommendation is easy to manipulate without anyone meaning to. One analyst assumes generous pump reliability and the lower wall looks prudent. Another assumes aggressive blockage and suddenly only the expensive seawall survives. Calibration changes that workflow by tying the parameter discussion to observed behavior. The debate shifts from "which numbers feel reasonable?" to "which numbers let the model reproduce what the district has actually done?"

That matters in production because model outputs here drive budgets, emergency planning, and public promises. A calibrated model is still uncertain, but it is uncertain in a disciplined way. The city can say, "these parameters were fitted against observed flood depths, pump outages, and evacuation timing, and here is the remaining uncertainty," instead of presenting one polished scenario as if it were truth.

Learning Objectives

By the end of this session, you will be able to:

  1. Explain what calibration does and does not accomplish - Distinguish fitting uncertain parameters to observations from proving that the whole model is correct.
  2. Design a calibration workflow for a real system model - Choose calibration targets, free parameters, and fit criteria that match the mechanism you are trying to constrain.
  3. Diagnose when calibration is revealing structural problems - Recognize equifinality, implausible fitted values, and residual patterns that mean the model needs redesign rather than more optimization.

Core Concepts Explained

Concept 1: Calibration starts by linking uncertain parameters to observable behavior

Harbor City's hybrid model contains many numbers, but not all of them should be calibrated. Street elevations come from surveys. Pump nameplate capacity comes from engineering documents. Shelter locations are fixed by policy. Calibration begins by isolating the smaller set of parameters that are both uncertain and behaviorally important: pump derating during flood conditions, drain blockage accumulation, road-closure depth thresholds, and the distribution of household departure delays after an alert.

The next step is to ask what observations can actually constrain each parameter. Tide gauges and water-depth sensors can constrain hydraulic behavior. Pump telemetry constrains reliability and degradation. Tunnel closure timestamps constrain how flood depth becomes mobility loss. Shelter check-ins and bus boarding logs constrain evacuation timing. If a parameter has no observational handle, it is not really calibrated; it is only chosen.

That mapping can be made explicit:

uncertain parameter              observation that constrains it
------------------------------  ---------------------------------------
drain blockage coefficient      water-depth time series by intersection
pump derating under flooding    pump uptime and discharge logs
tunnel closure threshold        incident timeline for West Tunnel access
departure delay distribution    shelter arrivals and drill departure logs

Once those links are clear, Harbor City can define a fit objective that compares simulated outputs with observed data. In practice this is often multi-objective because the city cares about more than one trace:

score = (
    0.45 * rmse(sim.water_level_cm, obs.water_level_cm)
    + 0.30 * abs(sim.west_tunnel_close_min - obs.west_tunnel_close_min)
    + 0.25 * rmse(sim.shelter_arrivals, obs.shelter_arrivals)
)

The weights are not arbitrary decoration. They encode which observations are more trustworthy, which scales need normalization, and which operational failures matter most. The trade-off is clear: the richer the objective, the better the model reflects the real decision surface, but the harder it becomes to understand why a candidate parameter set is winning.

Concept 2: Good calibration manages identifiability instead of pretending the optimizer knows the truth

Suppose Harbor City lowers pump efficiency and also lowers drain blockage in the same candidate run. Peak water level at Harbor Avenue might still match the observed storm almost perfectly because the two changes partially cancel each other. This is the calibration problem called equifinality: different parameter combinations generate similar output, so the optimizer can report a good fit without actually identifying the real mechanism.

That is why calibration needs constraints before it needs clever search. Harbor City bounds each free parameter using engineering judgment and operational data. Pump derating might be allowed between 5% and 25% because that range matches maintenance history. The tunnel closure threshold cannot be set below ankle depth for emergency vehicles if incident reports show fire crews still passed at that level. Departure-delay distributions should respect what the drill data actually measured. These constraints keep the fitted model inside the world the city could plausibly inhabit.

Multiple data sources help break these ties. Water-level traces alone may not distinguish between blockage and pump degradation, but adding pump telemetry and closure timing often does. A staged workflow helps too: calibrate the hydraulic submodel first against depth and pump data, then calibrate evacuation behavior against observed mobility once the road-availability inputs are credible. Joint calibration across all submodels can capture interactions, but it also increases the search space and makes it easier to hide one bad assumption behind another.

This is where the parameter sweeps from 05.md become operationally useful. The sweep already told Harbor City which parameters move the policy choice. Calibration now spends effort on those high-leverage parameters first. The city does not need to fit every knob in the model; it needs to constrain the knobs that change whether West Tunnel stays open and whether the cheaper plan remains safe.

Concept 3: Calibration is also a structural test, not just a parameter-fitting exercise

Imagine Harbor City finds a parameter set that matches flood-depth traces for two storms but still predicts that West Tunnel remains open twelve minutes longer than the incident log shows. The immediate temptation is to keep tuning. Maybe the blockage coefficient should go higher. Maybe the departure model should start earlier. But if the same mismatch persists across several reasonable parameter sets, calibration is signaling something more important: the model structure may be wrong.

Perhaps the tunnel does not fail because of average segment depth, but because water pools first at one entrance ramp that the hydraulic grid smoothed away. Perhaps the evacuation model assumes residents react only to the official alert, while camera footage shows that people started leaving once they saw buses rerouted. Calibration is valuable precisely because it can surface these structural omissions. A low residual is not the only result worth learning from; a patterned residual can tell you which mechanism is missing.

A practical calibration loop therefore looks like this:

choose free parameters
-> fit against observed events
-> inspect residual patterns and fitted values
-> reject implausible fits
-> revise structure or interfaces if errors stay systematic
-> keep parameter uncertainty, not just one best-fit vector

The stopping rule is not "the optimizer converged." Harbor City can stop when the fitted parameters remain physically defensible, the residuals no longer show obvious structural bias on the fit events, and the resulting policy recommendation is stable across the calibrated uncertainty band. After that, the model is ready for the next question in 07.md: does it predict unseen events well enough to earn trust outside the calibration set?

Troubleshooting

Issue: The best-scoring parameter set uses values that engineers or operators immediately reject as impossible.

Why it happens / is confusing: The optimizer is only minimizing error. If the parameter bounds are loose or the objective ignores operational realism, it will happily exploit unphysical combinations.

Clarification / Fix: Tighten parameter bounds, add priors or penalties for implausible values, and include observations from the subsystem that the fitted parameter is meant to represent.

Issue: The model matches peak flood depth but still gets tunnel closure timing badly wrong.

Why it happens / is confusing: A good fit on one aggregated metric can hide a wrong interface rule, such as translating depth to road closure with the wrong threshold or wrong location.

Clarification / Fix: Add calibration targets for the interface behavior itself, not just the upstream state. In Harbor City, closure timing and route availability need to be fitted alongside water depth.

Issue: Calibration quality improves on the historical storms used for fitting, but the fitted model becomes extremely sensitive or brittle.

Why it happens / is confusing: The model may be overfitting to a few events or compensating for missing structure by pushing parameters into narrow regions.

Clarification / Fix: Prefer parameter ranges and posterior bands over one exact optimum, inspect residual patterns across multiple events, and keep a clean separation between the events used for calibration and the events reserved for validation.

Advanced Connections

Connection 1: Parameter Sweeps ↔ Calibration

Parameter sweeps in 05.md mapped the parts of Harbor City's model space where the capital decision flips from robust to fragile. Calibration uses observed flood, pump, and evacuation data to narrow that space. Together they answer two different questions: "which parameters matter?" and "which values for those parameters are actually credible?"

Connection 2: Calibration ↔ Validation

Calibration fits the model to known events; validation asks whether the fitted model can explain events it did not see during fitting. That handoff is the point of 07.md. A model that calibrates well but fails on held-out storms has learned the past too specifically. A model that calibrates within plausible ranges and still predicts new events is earning the right to influence expensive decisions.

Resources

Key Insights

  1. Calibration constrains a model; it does not certify it - A fitted model is only as trustworthy as its structure, data quality, and parameter realism.
  2. Observations must map cleanly to uncertain parameters - If you cannot say which data constrains which parameter, you are not calibrating so much as tuning by taste.
  3. Residual patterns are part of the result - Systematic mismatches often reveal missing mechanisms or broken interfaces that parameter search alone cannot repair.
PREVIOUS Parameter Sweeps - Systematic Exploration of Model Space NEXT Validation - Testing Model Predictions

← Back to System Dynamics and Causal Modeling

← Back to Learning Hub