Experiment, Control, and Confounding

By the end of this lesson, you will be able to...

• Distinguish treatment, control, comparison unit, outcome, and confounder in a small study.

• Explain why a before-and-after difference does not automatically identify a cause.

• Choose a comparison design and state the trade-off it introduces.

Idea in one sentence: A causal experiment makes the treated case comparable with a control case, while looking for hidden differences that could explain the outcome instead.

Core Insight

In the previous lesson, a team wrote this hypothesis:

Retrieval gating will reduce unsupported factual claims in an AI support assistant, without making the assistant refuse too many questions.

Now the team deploys retrieval gating on Monday. On Friday, the dashboard says:

The numbers look like the prediction. The team wants to say, “The gate caused the improvement.”

But Monday also brought a new knowledge base, a prompt change, a different mix of questions, and a support team that knew the new system was being tested. Any of those changes could affect the result.

The problem is not that the dashboard is useless. The problem is that it compares two bundles of circumstances. The treatment changed at the same time as several other things.

An experiment is a design for making the causal comparison less ambiguous. It does not create certainty by naming one group “control.” It makes the relevant alternatives visible, keeps important conditions comparable, and records what could still bias the comparison.

The Before-and-After Trap

The team's first design is attractive because it is easy:

1. Measure the assistant for one week.
2. Turn on the retrieval gate.
3. Measure it for another week.
4. Compare the averages.

This design can detect a change. It cannot, by itself, tell us which change caused it.

Imagine four explanations for the 6-point drop in unsupported claims:

• the retrieval gate rejected weak passages;
• the new knowledge base removed outdated instructions;
• the new prompt made the model answer more cautiously;
• the second week contained mostly simple questions.

The before-and-after number is compatible with all four explanations. A causal claim needs a comparison that separates them.

Check: Which change would most directly threaten the team's causal conclusion?

Think first, then reveal.

Answer: Any change that differs between the two weeks and also affects the outcome. A prompt update, question mix, or knowledge-base change is a possible alternative cause. The gate is not the only thing that moved.

The Pieces of a Fair Comparison

Before choosing a design, name the pieces.

Plain meaning:

A control is a comparison story. It asks what would have happened to a similar unit without the treatment.

In this scenario:

The control assistant should use the same model, prompt, knowledge base, evaluator, and time window as the treated assistant. The intended difference is the retrieval gate.

Technical name:

The counterfactual is the outcome the same unit would have had under the other condition. We cannot observe both versions of the same request at the same moment in a live service. A good design uses many comparable units to approximate that missing comparison.

The word “control” does not mean “perfect” or “inactive.” It means the condition against which the treatment is interpreted.

Three Designs, Three Kinds of Evidence

Design 1: Before and after

The team changes one system at a known time and compares time windows.

This is useful when:

• the intervention must apply to everyone;
• the outcome is monitored continuously;
• an abrupt operational change is itself informative.

Its main weakness is history. Anything that changes with time can be mistaken for a treatment effect. Seasonality, learning, incidents, new data, and policy changes all travel with the intervention.

Design 2: Parallel control and treatment

The team keeps two paths live during the same period:

• control: old retrieval behavior;
• treatment: retrieval gate enabled.

Both paths use the same model version, prompt, knowledge base, evaluator, and monitoring period. If the paths receive comparable requests, the difference is easier to attribute to the gate.

Its costs are operational. The team runs two systems, accepts that some users receive the old behavior, and must prevent one path from contaminating the other.

Design 3: Random assignment

For each eligible user or session, a rule such as a stable random hash assigns the user to control or treatment. The assignment is made before the outcome is known.

Randomization does not make every small sample identical. It makes treatment assignment independent of the measured and unmeasured characteristics in expectation. With enough units, difficult questions, languages, user types, and time periods should be mixed across both conditions instead of being selected into one side by a human decision.

Its costs include sample size, implementation discipline, interference between users, and the need to analyze people who were assigned even if they did not follow the assigned path.

Confounding: The Hidden Bridge

Suppose the team does not randomize. Instead, it sends difficult questions to the new gate because the team wants to help with the hardest cases.

The dashboard shows:

The treatment looks worse. But question difficulty affects both:

1. Assignment: difficult questions were more likely to receive the gate.
2. Outcome: difficult questions are more likely to produce unsupported claims.

Difficulty is a confounder in this comparison. It is a hidden bridge connecting treatment assignment to the outcome.

The causal story the team wants is:

retrieval gate -> unsupported-claim rate

The observed study also contains:

question difficulty -> treatment assignment
question difficulty -> unsupported-claim rate

The second pair of arrows can create a treatment difference even if the gate has no effect. The gate may even help, while the treated group still looks worse because it received harder questions.

This is why “we measured both variables” is not the same as “we controlled the confounder.” A variable becomes useful only when the design or analysis uses it to make the comparison more credible.

A Worked Experiment Path

Let us design a small parallel experiment for the support assistant.

Starting promise

The team wants to reduce unsupported factual claims without increasing unanswered questions by more than 3 percentage points.

Assignment

Assign each new user session to control or treatment using a stable random value. Keep the same assignment for that session so one user does not see the system switch while asking follow-up questions.

Common conditions

Freeze the model version, prompt, knowledge-base snapshot, evaluator rubric, logging format, and observation window. Record question category and language even though randomization should distribute them approximately.

One small trace

The input is a session and its question. The transition is the assignment rule. The intermediate state is the same prompt, model, knowledge base, and evaluator with only the retrieval gate differing. The output is the recorded answer and its coded outcomes.

In this tiny sample, treatment has one unsupported answer out of three and one unanswered answer. Control has two unsupported answers out of three and no unanswered answer. The direction looks promising, but six sessions cannot support a strong conclusion. The sample is a trace of the design, not proof of the effect.

Now imagine a hidden change: treatment sessions were routed to a new evaluator that labels cautious answers as “unanswered,” while control sessions used the old evaluator. The apparent treatment cost would be partly a measurement difference. The design failed to keep the evidence chain common.

So far, the path is:

causal hypothesis
-> treatment and control conditions
-> assignment rule
-> common conditions
-> outcome trace
-> confounder and contamination checks
-> cautious comparison

The naive path is:

turn on feature
-> see a better dashboard
-> attribute every difference to the feature

What Randomization Does and Does Not Buy

Random assignment is powerful because it prevents the investigator from choosing who receives the treatment based on the expected outcome. It also gives a clear story for why measured and unmeasured characteristics should balance over repeated assignments.

It does not solve everything.

• Small samples remain noisy. Randomization balances characteristics in expectation, not in every six-row table.
• Interference can break the unit choice. If one user shares treatment output with another user, their outcomes are no longer independent in the simple sense.
• Noncompliance matters. A user assigned to treatment may hit a fallback path. Analyze assignment and actual exposure carefully.
• Attrition can bias the result. If frustrated treatment users leave before answering the survey, the observed outcomes are selective.
• Measurement can still be wrong. Randomization does not make a vague rubric or biased sensor valid.
• A treatment can have heterogeneous effects. It may help setup questions and harm billing questions. An average can hide that boundary.

The central trade-off is between control and realism. More control makes the causal comparison cleaner, but it can make the setting less like normal use and more expensive to operate. A live randomized rollout may be realistic but exposed to interference, drift, and operational surprises. A tightly controlled benchmark may isolate one mechanism but fail to represent real users.

Common Confusions

Confusion: A control group receives nothing

Why it is tempting:

The word “control” sounds like a blank or inactive condition.

Better model:

The control receives the comparison condition. In a software experiment, that is often the current system. In a fertilizer study, it may be the standard fertilizer or no added fertilizer, depending on the question.

Confusion: Randomization makes groups identical

Why it is tempting:

Random assignment is described as if it removes all differences.

Better model:

Randomization prevents systematic selection in expectation. Any particular sample can still be unbalanced, and analysis must inspect important subgroups and uncertainty.

Confusion: A measured variable is automatically controlled

Why it is tempting:

The team can point to a column in the dataset and feel that the risk has been handled.

Better model:

Recording difficulty, language, or evaluator identity helps only if the design or analysis uses that information to make the comparison fairer. Measurement is not control by itself.

Confusion: A before-and-after improvement is useless

Why it is tempting:

The design has confounding risks, so it feels safer to reject it completely.

Better model:

A before-and-after series can reveal a change, generate a hypothesis, and detect operational problems. Its causal claim is weaker because history and treatment move together.

Check Your Understanding

Check: A team enables a safety filter only for users who have already reported harmful outputs. The filtered users still report more harmful outputs than other users. What is the first design concern?

Think first, then reveal.

Answer: Selection by prior risk is a confounding concern. The filter group started with a different risk profile, so the outcome difference cannot be attributed to the filter without a stronger comparison or an appropriate adjustment.

Check: A randomized experiment has a strong average improvement, but the treatment group includes many more mobile users by chance. What should the team do?

Think first, then reveal.

Answer: Report the imbalance, inspect the outcome by device, and consider whether the sample is large enough or whether a stratified design is needed. Randomization remains useful, but it does not erase chance imbalance in one run.

Practice: Design the Fertilizer Study

Return to the claim from the previous lesson:

Fertilizer F helps tomato seedlings grow better.

Design a small study. State:

1. The treatment condition.
2. The control condition.
3. The unit assigned to a condition.
4. One primary outcome and one possible side effect.
5. The assignment rule.
6. Two possible confounders and how the design reduces them.

A good answer might say:

• Treatment: Fertilizer F at the stated dose.
• Control: Standard fertilizer at the same watering and light conditions.
• Unit: One seedling, with seedlings assigned independently within the same variety and planting batch.
• Outcomes: Stem height after four weeks; leaf damage or plant death as a side effect.
• Assignment: Randomize seedlings after blocking by starting size so one condition does not receive all the larger seedlings.
• Confounders: Light position and watering amount. Rotate tray positions or randomize positions, and measure watering rather than trusting a visual estimate.

The exact design can vary. The key question is whether the proposed treatment is the main systematic difference between the groups.

Connection to the Next Lesson

An experiment can make treatment and control comparable, but it still relies on measurements. The next lesson asks how instruments, operational definitions, and proxies shape what the study can actually observe.

Resources

• [ARTICLE] Scientific Method - Stanford Encyclopedia of Philosophy - Focus: How experiments, controls, hypotheses, and evidence fit into scientific reasoning.
• [ARTICLE] Experimental Design - NIST/SEMATECH e-Handbook of Statistical Methods - Focus: Randomization, replication, blocking, and choosing factors and responses.
• [BOOK] Causal Inference: What If - Focus: Counterfactual questions, treatment assignment, confounding, and causal diagrams.

Key Takeaways

• A treatment-control comparison asks what would have happened without the treatment; it does not merely place two labels on a dashboard.
• Before-and-after designs can reveal change, but time carries many alternative causes with it.
• A confounder affects both assignment or exposure and the outcome, creating a misleading causal path.
• Randomization reduces systematic selection in expectation, but it does not fix small samples, bad measurements, interference, or attrition.
• The cleanest design is not always the most realistic. State what the design buys, what it costs, and what could still explain the result.