Adversarial Thinking Without Paranoia

By the end of this lesson, you will be able to...

• Build a bounded adversary profile from a system promise and visible evidence.

• Separate an adversary's goal, starting access, capabilities, knowledge, constraints, and the assumptions behind the profile.

• Reject an unbounded attack story and identify the next useful security question.

Idea in one sentence: Adversarial thinking becomes useful when the imagined adversary is specific enough to challenge and limited enough to test.

Core Insight

Northstar is a fictional document-sharing service for one company. A document owner can create a link with this rule:

Anyone signed in with a company account who has the link may view the document.

The design team uses Northstar to share a confidential roadmap. The owner sends the link to six colleagues. Later, one colleague copies the link into an internal support ticket so that another team can diagnose a display problem.

The intended path looks reasonable: authenticated colleagues open a link that an owner deliberately shared. But the ticket is visible to more people than the six intended readers. The access rule protects against outsiders, not against every company account outside the design team.

Adversarial thinking asks a different question from ordinary feature testing:

Who could want an outcome that conflicts with the owner's intent, what could that person actually do from their starting position, and which assumption would have to fail?

The word actually matters. If we imagine an attacker who can already bypass authentication, read every database row, disable every log, and control every administrator, the answer is easy: Northstar loses. That story creates alarm, but it gives the team no useful next decision.

A useful adversary is bounded. The profile says not only what the adversary wants and can do, but also what the adversary cannot do, does not know, and must spend.

The Tempting Model: Think of the Worst Case

“Assume the attacker can do anything” can be useful for one narrow purpose: testing whether a claim depends on a secret or control remaining perfect. It is an upper-bound thought experiment.

It becomes a weak default model for three reasons.

First, it erases preconditions. If the attacker already has every privilege, we cannot see which identity, workflow, or trust boundary created the opportunity.

Second, it makes alternatives hard to compare. Every design fails against unlimited power, so the model cannot tell us whether one change removes a plausible path.

Third, it hides evidence. “A sophisticated attacker might do something” does not tell us which observation would strengthen, weaken, or falsify the concern.

The stronger model is not “attackers are weak.” It is:

Start with the strongest adversary that is relevant to the decision, then state the limits and evidence explicitly.

Six Fields That Bound an Adversary

NIST risk-assessment guidance characterizes adversarial threat sources using factors such as capability, intent, and targeting. For this lesson, we use a small teaching model that turns those ideas into six questions.

These fields are not a probability calculator. They create an inspectable model. Another reviewer can challenge a field, replace an assumption, or ask for evidence without discarding the whole analysis.

MITRE ATT&CK uses a related separation: a tactic represents why an adversary acts, while a technique represents how the adversary pursues that goal. ATT&CK is much more detailed than this lesson needs, but the distinction protects us from beginning with a technique list before we know the relevant objective.

Worked Profile: The Contractor Who Finds the Link

We will model one fictional person, Ari, who supports Northstar for the company. The name is only a stable label. It is not a demographic prediction.

The current evidence says:

• Contractors receive ordinary company accounts.
• The support ticket is readable by the product-support group.
• The ticket contains the Northstar link.
• Northstar checks for a company account but does not check membership in the design team.
• Northstar records a download event with an account identifier.

For this worked example, those are given facts about the fictional system. The following profile contains both facts and assumptions:

Notice what the profile does not claim. It does not say that Ari has exploited software, stolen a credential, bypassed authentication, or accessed the document. It identifies a starting position from which one security question becomes worth testing:

Does possession of the link plus any company account authorize access to the roadmap?

That question is specific. The team can inspect the configuration or run an authorized test with a test account. Either result updates the model.

Trace the Reasoning, Not an Attack Recipe

Here is the complete reasoning path:

System promise:
Only intended readers should view the confidential roadmap.

Observed starting state:
A support ticket exposes the link to a wider internal group.

Bounded adversary:
Ari has a normal company account and ticket access, but no elevated privilege.

Relevant capability:
Ari can make the same document request as an intended reader.

Control being tested:
Northstar checks company membership, not intended-reader membership.

Potential consequence:
The system may return the roadmap to an authenticated but unintended reader.

Evidence still needed:
Confirm the effective authorization rule with configuration or an authorized test.

The reasoning moves from promise to evidence to profile to question. It does not move from “attacker” to “breach.”

This distinction prevents two opposite mistakes:

• Underreaction: “Ari is authenticated, so the request is legitimate.”
• Overreaction: “Ari can see a ticket, so Ari can probably compromise the company.”

The bounded model supports the narrower conclusion: the link may cross an authorization boundary that the feature's intended-use story did not make visible.

Invert One Assumption at a Time

An assumption inversion asks:

What does this design expect to remain true, and what changes if one relevant actor does not cooperate with that expectation?

For Northstar, record the assumptions in a small ledger:

The ledger keeps facts, assumptions, and unknowns in different places. It also stops the discussion from expanding without limit. Revocation is relevant because it changes this adversary's window of opportunity. The security of the office building is not relevant to this decision unless new evidence connects it to the document path.

Compare Three Profiles

A bounded profile does not need to be realistic in every detail before discussion begins. It needs to be relevant, internally consistent, and revisable.

What Boundedness Buys—and Costs

Bounding improves focus. It lets a team compare controls against named preconditions. Removing broad ticket access, changing link authorization, shortening link lifetime, and improving download signals affect different parts of Ari's profile.

Bounding also costs work. The team must inspect access paths, distinguish facts from assumptions, and sometimes maintain more than one profile. A single profile may miss an outsider, privileged insider, automated abuse process, or accidental actor.

This is the central trade-off:

A narrow profile makes reasoning testable, but a profile that is too narrow creates blind spots.

Use several bounded profiles when the system promise faces meaningfully different starting positions. Do not repair a narrow model by quietly giving one adversary every possible power.

This lesson also has a boundary. An adversary profile does not establish likelihood, prove a vulnerability, or select a mitigation. It creates a disciplined input for those later decisions. You can see the boundary when the conversation moves from “Could this actor reach the control?” to “How often will this happen?” or “Which fix is proportionate?” Those questions require evidence and risk judgment beyond the profile itself.

Common Confusions

Confusion: A persona describes the kind of person who attacks.

Why it is tempting: stories are easier to remember when they have a character.

Better model: the profile describes a decision-relevant position—goal, access, capability, knowledge, constraints, and evidence. Demographic guesses usually add bias without improving the analysis.

Confusion: If an adversary has a capability, the action is likely.

Why it is tempting: capability makes the path vivid.

Better model: capability establishes possibility under stated conditions. Likelihood also depends on intent, targeting, opportunity, cost, and evidence.

Confusion: “Out of scope” means safe.

Why it is tempting: removing a case from the review makes the document shorter.

Better model: an exclusion limits what the current analysis claims. It does not remove the risk from the world.

Confusion: More powerful adversaries always produce better security.

Why it is tempting: stronger tests sound more conservative.

Better model: power helps only while the scenario still distinguishes designs and controls. Unlimited power can make analysis non-actionable.

Check Your Understanding

Check: A profile says, “The attacker is advanced and can probably get admin access.” Which fields are missing?

Think first, then reveal.

Answer: Almost all of them. “Advanced” does not name a goal, starting access, concrete capability, system knowledge, constraint, or evidence. Admin access is being assumed as an outcome rather than traced from a precondition.

Check: Northstar confirms that only design-team members can open the roadmap, even when another employee has the link. Is the adversary profile useless?

Think first, then reveal.

Answer: No. The test has falsified the suspected path, which is useful evidence. Record that the authorization control breaks this path, then decide whether another bounded starting position is relevant. Do not keep the original claim alive by silently granting Ari a stronger privilege.

Practice: Bound a Nearby Adversary

A fictional finance application lets authenticated support staff view customer account summaries. Exporting a full transaction report requires a finance role. Export events are logged. Temporary support accounts expire after seven days.

Write one bounded profile for a temporary support contractor who might seek transaction data. Then write one question the team can answer without performing an unauthorized test.

A good answer should include:

• a goal more precise than “steal data”;
• the contractor's starting account and legitimate summary access;
• actions possible from that access without assuming the finance role;
• system-specific knowledge the contractor has or lacks;
• constraints such as account expiry, role checks, and audit evidence;
• a clear label for facts, assumptions, and unknowns; and
• a next question such as whether the summary endpoint exposes fields that violate the product's promise, or whether role checks cover every export path.

A weak answer grants the contractor a finance role, database access, and log deletion without explaining how any of those conditions arise.

Connections

The next lesson uses bounded profiles to challenge an existing threat model with misuse and abuse cases. Later lessons will trace how small weaknesses compose, include human workflows, and map meaningful paths to defenses.

Resources

• [STANDARD] NIST SP 800-30 Rev. 1: Guide for Conducting Risk Assessments — Focus: threat-source capability, intent, targeting, relevance, and scope.
• [KNOWLEDGE BASE] MITRE ATT&CK: Get Started — Focus: the separation between adversary goals, techniques, and lower-level behavior.
• [GUIDE] OWASP Threat Modeling Cheat Sheet — Focus: structured, repeatable analysis from a system model to actionable responses.
• [MANIFESTO] Threat Modeling Manifesto — Focus: informed creativity, varied viewpoints, manageable scope, and avoiding admiration for the problem.

Key Takeaways

• A useful adversary profile states a goal, starting access, capabilities, knowledge, constraints, and the evidence behind each claim.
• Capability shows what may be possible under stated conditions; it does not establish intent, targeting, or likelihood.
• Constraints make an adversary model testable. They also create a risk of blind spots when the profile is too narrow.
• Assumption inversion should produce a concrete question or decision, not a dramatic story.
• When evidence breaks an attack path, update or discard the path instead of silently making the adversary more powerful.