Capstone: Audit a Knowledge Claim

By the end of this lesson, you will be able to...

• Produce a concise audit that separates a claim from its evidence, sources, defeaters, and consequences.

• Explain how disagreement, motivation, institutional procedures, and remaining ignorance affect your confidence.

• State a proportionate conclusion and the next observation that would change it.

Idea in one sentence: A knowledge-claim audit makes judgment inspectable: it shows what supports a claim, where the support stops, and what would make you update.

Core Insight

Imagine that a town announces a wetland restoration project has worked:

Restoring the marsh increased the fish population by 20% and reduced flood risk for nearby homes.

The announcement includes a chart from the environmental agency. A local fisher says the river is healthier. A contractor involved in the project says the result proves the restoration method. Another researcher points out that rainfall was unusually high before the restoration and that fish moved between connected rivers.

The town must decide whether to extend the project, change its design, or wait for more evidence. You do not need to become a hydrologist to reason well about the claim. You need to make the reasoning path visible.

The capstone is a short audit memo. It is not a debate performance and not a demand for perfect certainty. It should let another reader see:

claim -> evidence -> source position -> defeaters
     -> disagreement and incentives -> institutional path
     -> remaining ignorance -> proportionate conclusion

The Capstone Contract

Choose one claim that matters enough to inspect but is small enough to describe in one sentence. Good candidates include:

• a claim in a paper, report, or public dashboard;
• an expert's recommendation;
• a statement made by an AI system;
• a technical benchmark or incident explanation;
• a historical or environmental claim;
• a policy promise with evidence attached.

Avoid claims that are only personal preferences, such as “this color is nicer.” Also avoid claims so broad that no reasonable audit could finish.

Your final artifact should fit on one or two pages. It can use tables. A reader should be able to identify your current confidence and the next check in less than a minute.

Choose a Claim You Can Actually Audit

Before writing, test the claim with three questions:

1. Can I name the evidence? “People say this tool is better” is too vague; “the benchmark reports lower latency on these requests” can be inspected.
2. Can I name the boundary? Include the population, time, location, version, or conditions that make the claim meaningful.
3. Can I imagine an update? If no possible observation would change your confidence, you have a slogan or a definition, not a useful audit target.

If the claim is still too broad, split it into two or three smaller claims. The wetland announcement became a fish-count claim and a flood-risk claim because one piece of evidence could not answer both. A smaller claim is not a weaker capstone. It is a better instrument for judgment.

A Worked Audit: The Wetland Claim

Here is a complete example before you write your own.

1. State the claim and the consequence

The public announcement contains two claims:

1. Fish population increased by 20% after restoration.
2. The restoration reduced flood risk for nearby homes.

The project team wants funding for another phase. A wrong conclusion could waste money, but delaying a useful flood intervention also has a cost. The audit should not merge the two claims just because they appear in one sentence.

2. Separate belief, evidence, and action threshold

The action threshold is not the same as the knowledge threshold. You may support a reversible monitoring step before you have enough evidence for a permanent expansion.

3. Map evidence and defeaters

The audit does not count every item as equal proof. It asks what question each item can answer and what would defeat that answer.

4. Evaluate source trust by domain

The agency is well placed to report its sampling procedure but has an incentive to show that its funded project succeeded. The fisher is well placed to notice local changes but not to estimate a percentage for the whole river. The independent researcher may be better placed to compare seasons, but may not have access to every raw record.

This is a source-trust map, not a ranking of people's worth. A source can be credible for one part of the claim and weak for another.

5. Use disagreement as information

The researcher disagrees with the announcement's causal interpretation. That disagreement does not prove the restoration failed. It reveals two possible gaps:

• the before-and-after comparison may not control for rainfall and fish movement;
• “increased by 20%” may describe a measured sample, not the whole ecosystem.

The next check should compare the same season, location, and method across years and include a nearby non-restored area if possible.

6. Inspect motivation and the institution

The agency wants continued funding. The contractor wants its method adopted. The fisher wants a healthier river. The researcher wants a defensible causal claim. None of these motives decides the truth, but they tell us where selective search or unequal standards could enter.

The institution controls the sampling schedule, the data archive, the report language, and the route for independent review. A strong correction path would publish raw counts, methods, uncertainty, and a way for outside researchers to submit contradictory measurements.

7. Name what remains unknown

These limits lower confidence in the causal and percentage claims. They do not erase the evidence that the restored area currently has different measurements.

8. Write the provisional conclusion

The available data support a local improvement in measured fish counts after restoration, but they do not yet establish a 20% ecosystem-wide increase or prove that restoration alone reduced flood risk. Continue monitoring with comparable sites and publish the raw measurements before funding a larger expansion. Reconsider the conclusion if the effect disappears across seasons or if an independent comparison shows the same change without restoration.

This conclusion is useful because it keeps the supported result, narrows the unsupported part, and names an update path.

The Audit Template

Use this structure for your own claim:

## Claim

[One sentence. Include scope, time, population, or system boundary.]

## Consequence

[Who might act on it? What is the action threshold?]

## Current Belief

[What do I believe now, and how strong is that confidence?]

## Evidence and Defeaters

| Evidence | Supports | Could be weakened by |
| --- | --- | --- |
| ... | ... | ... |

## Source Trust

[Who produced each item? What is each source well placed to judge? What incentives or limits matter?]

## Disagreement

[Who disagrees? Is the disagreement about the claim, definition, evidence, peer position, or action?]

## Motivation and Identity

[Who benefits if the claim is accepted? What conclusion would I prefer?]

## Institutional Path

[Who collects, filters, reviews, publishes, and corrects the information?]

## Remaining Ignorance

[Known unknown, blind spot, or current limit.]

## Provisional Conclusion

[What does the evidence justify now? What does it not justify?]

## Next Check

[What observation would change my confidence or action?]

The template is a thinking aid, not a form to complete mechanically. If a section does not apply, explain why. “No relevant disagreement found after checking” is better than inventing one.

Quality Checks Before You Submit

Run these checks on your draft:

Claim check

Could a reasonable reader tell what would count as evidence for or against the sentence? If not, narrow the scope.

Evidence check

Did you distinguish an observation, an interpretation, and a source's conclusion? Did you name at least one defeater?

Trust check

Did you evaluate domain fit and access to evidence rather than using status as a shortcut? Did you avoid rejecting a claim only because a source has an incentive?

Disagreement check

Did you investigate whether the disagreement concerns the same proposition? Did you treat a relevant peer as higher-order evidence without automatically splitting the difference?

Motivation check

Did you state what you want to be true, and did you apply the same evidential rule to your preferred and rival explanations?

Institution check

Can you trace how the claim was collected, filtered, authorized, communicated, and corrected? If there is no correction path, say so.

Ignorance check

Did you separate a named unknown from a blind spot? Did you state whether the next check could actually change the conclusion?

Decision check

Is the conclusion proportionate to the evidence and consequence? Do not turn an uncertain belief into a universal rule, but do not use uncertainty to avoid every reversible action.

Common Capstone Failures

A long summary with no decision

The memo repeats the source but never says what the reader should believe or do. Fix it with a provisional conclusion and an action threshold.

A verdict based on source reputation

“The university says it, so it is true” and “the company benefits, so it is false” use the same shortcut. Map what the source can actually know.

Criticism without an alternative check

Pointing out flaws is not yet an audit. Name the measurement, comparison, replication, or correction route that would reduce the uncertainty.

Endless qualification

Listing every possible caveat can make a true narrow result disappear. Keep the supported part, state the boundary, and stop when extra caveats no longer change the conclusion.

Treating disagreement as a vote

The number of people on each side does not replace evidence. Ask what each person saw, what claim they answered, and how well placed they are for that question.

Trade-offs and Limits

The central trade-off is that an audit makes judgment more defensible, but it costs time and can tempt the writer into endless suspicion.

• Fast verdict: easy to communicate, but hides evidence and limits.
• Complete investigation: richer support, but may arrive after the decision window.
• Bounded audit: states what is supported, what is open, and what to check next; it still depends on imperfect sources and institutions.

An audit cannot manufacture missing data or guarantee that a source is honest. It cannot remove values from decisions. It can make the reasoning visible enough for another person to challenge, improve, or responsibly accept.

You have reached the useful boundary when another reader can answer three questions: What exactly is the claim? Why should I believe it, and how strongly? What would make us change our minds?

Final Practice

Write your own audit using a claim you can cite or describe precisely. Keep it to one or two pages. Before finalizing, ask a second reader to identify:

1. one place where your claim is too broad;
2. one evidence item you treated too generously or too harshly;
3. one unknown that could change the conclusion;
4. one next check that would be worth its cost.

Then revise the memo once. Your final version should end with a conclusion in one of these forms:

Supported within [scope], with [limit].
Partially supported; [open question] remains.
Not supported by the available evidence; [next check] would test it.
Underdetermined for now; [decision] is still justified because [threshold/reversibility].

Rubric

Score each dimension from 0 to 2.

A score of 9 or more shows a usable audit. A lower score does not mean the claim is false; it shows which part of the reasoning needs another pass.

Resources

• [ARTICLE] Stanford Encyclopedia of Philosophy: Epistemology - Revisit belief, justification, evidence, and knowledge.
• [ARTICLE] Stanford Encyclopedia of Philosophy: Social Epistemology - Revisit testimony, disagreement, institutions, and collective knowledge.
• [ARTICLE] National Academies: Reproducibility and Replicability in Science - Revisit transparency, independent checks, and correction of published claims.

Key Takeaways

• A good audit starts with a narrow claim and separates it from belief, evidence, source trust, defeaters, and action.
• Disagreement, motivation, and institutional incentives are signals to inspect the reasoning path, not automatic refutations.
• Remaining ignorance should lower or bound confidence and identify a useful next check.
• A provisional conclusion can support action without pretending that uncertainty has disappeared.
• The capstone is successful when another reader can inspect your judgment, challenge it, and know what would change it.