Network Partitions and Failure Models

By the end of this lesson, you will be able to...

• Explain why a timeout is a local observation, not global truth.

• Distinguish crash, delay, overload, packet loss, asymmetric reachability, and partition interpretations.

• Compare quorum refusal with writable divergence under a partition.

Idea in one sentence: A failure model says what the system cannot know reliably, and that uncertainty decides which actions are safe.

Core Insight

The previous lesson ended with a careful warning: a timeout does not prove what happened on the server. Now widen that idea from one caller to a whole distributed system.

Imagine the learning platform runs across three zones:

zone A: API gateway and progress replica A
zone B: progress replica B and metadata service
zone C: progress replica C and certificate service

During an incident, A can still talk to B. B can still talk to C. A cannot talk to C. The certificate service in C still answers local reads. The load balancer sees some health checks pass and some fail.

No single machine has necessarily crashed. The harder problem is that different parts of the system now see different communication maps.

The naive idea is:

If one zone cannot reach another zone, the unreachable zone is down.

That may be true. It may also be false. The unreachable zone may be alive but slow. Replies may be dropped in one direction. A firewall rule may affect A but not B. The path may be congested. A few packets may be lost. Or the system may be partitioned into groups that can still do local work but cannot coordinate globally.

A failure model names the uncertainty the system is designed to survive. It does not say "bad things happen." It says which bad observations are possible and which assumptions the design is allowed to make.

The Small Situation

Start with one learner completing lesson 044.

The gateway in zone A receives:

POST /complete-lesson
learner_id=7
lesson_id=044
idempotency_key=req-44-a

The progress service is replicated across A, B, and C. A completion write is authoritative only after a quorum of replicas accepts it. With three replicas, quorum is two.

For this request, A can reach B but not C:

A -> B works
A -> C times out
B -> C works sometimes
C -> A replies are dropped

From A's point of view, C looks bad. From B's point of view, C sometimes looks fine. From C's point of view, it may be healthy and confused about why A is not receiving replies.

Plain meaning:

A network partition is a communication split where some parts of the system can still run but cannot all talk to each other reliably.

In this scenario:

Zone A and zone C do not share the same view of reachability. They can each be locally alive while disagreeing about what the system can coordinate.

Technical name:

This is a partial failure with split reachability. It is not the same as total outage or simple process crash.

Local Observation Is Not Global Truth

The API server in A calls the certificate service in C and times out. The safe statement is narrow:

A did not receive a useful response from C before its deadline.

That observation has several possible explanations:

This is why failure models are models of knowledge, not just lists of broken components. The system must decide what it can safely do when observations are incomplete.

Plain meaning:

A failure model is the set of failure shapes the design expects and handles.

In this scenario:

If the system only expects crashes, it may treat C as dead when A cannot reach it. If the system expects partitions and one-way loss, it will be more careful: A's failed call is evidence about one path, not proof about C's global state.

Technical name:

Crash failure, omission failure, delay, packet loss, asymmetric reachability, and partition are different failure assumptions. They support different safe designs.

Check: A times out while calling C. Which conclusion is safe?

Think first, then reveal.

Answer: A can conclude only that A did not get a useful response from C before its deadline. It cannot conclude that C is dead, that C did no work, or that B cannot reach C.

A Worked Partition Trace

Now trace the progress write.

Input:
  A receives POST /complete-lesson for learner 7
  quorum size = 2 of 3 progress replicas

Transition:
  A writes locally
  A sends write to B
  A sends write to C

Intermediate state:
  B acknowledges
  C does not answer A before deadline
  C may still be alive and reachable from B

Output or decision:
  A has acknowledgements from A and B
  quorum = 2, so the write can commit

Naive failure contrast:
  if A required every replica, one unreachable path would block all writes
  if A ignored quorum rules, isolated sides could create conflicting truth

Here is the same moment as a state table:

A quorum rule gives A a safe decision without requiring perfect knowledge. A does not need every replica. It needs enough overlap that future committed decisions cannot form a separate incompatible history.

Now change the split:

partition side 1: A only
partition side 2: B and C
quorum = 2

If A is alone, A is alive but not authoritative. It should reject new authoritative completion writes or return a clear unavailable result. B and C can still form a quorum.

This is not because A is bad. It is because local health is not the same as coordinated authority.

Quorum Refuses Some Work To Preserve Authority

Quorum systems buy one kind of safety: they prevent two isolated groups from both claiming to be the only authoritative writer.

With five replicas and quorum three:

5-node group
quorum = 3

partition left: 2 nodes
partition right: 3 nodes

left side: alive, but cannot commit authoritative writes
right side: alive and can commit authoritative writes

The two-node side may have healthy processes, disks, and local network links. It still lacks enough overlapping agreement. If it accepted writes as authoritative, and the three-node side also accepted writes, the system could heal into two different histories.

The rule is simple:

def can_commit(reachable_replicas, quorum_size):
    return len(reachable_replicas) >= quorum_size

The design idea behind it is not simple:

availability is reduced so authority stays coherent

The trade-off is availability versus coordination safety. A quorum system intentionally rejects some work during communication uncertainty. That rejection is not a bug when the alternative is split authority.

Check: A two-node minority side has healthy machines but cannot reach the other three replicas in a five-node quorum system. Should it accept authoritative writes?

Think first, then reveal.

Answer: No. It is locally alive, but it does not have enough overlapping agreement to make new writes authoritative without risking conflicting histories.

Staying Writable Means Owning Reconciliation

Not every subsystem needs the same answer.

The certificate issuer should probably fail closed during partition uncertainty. A certificate is a user-visible promise. It should not be issued from a side that cannot verify authoritative progress.

Recommendation hints are different. If a learner opens a lesson, dismisses a prompt, or clicks an interest tag, the platform may accept those hints locally and merge them later. The user experience stays smoother, and a temporary duplicate hint is not catastrophic.

That choice moves the cost:

partition starts
  -> side A accepts local recommendation hints
  -> side C accepts local recommendation hints
  -> histories diverge
partition heals
  -> system merges, deduplicates, or resolves hints

The merge rule must match the data.

Remaining writable is not free. It means accepting bounded staleness, divergent writes, conflict resolution, weaker immediate guarantees, or a narrower definition of what remains available.

The useful question is not "CP or AP?" as a slogan. The useful question is:

Which operations must preserve authority, and which operations can tolerate temporary divergence with a clear repair rule?

Common Confusions

Confusion: "Partition means everything is down"

Why it is tempting:

From one caller's view, a timeout can make a remote zone feel gone.

Better model:

A partition is split communication, not necessarily total failure. Each side may keep doing local work, which is exactly why authority and reconciliation become hard.

Confusion: "Healthy means authoritative"

Why it is tempting:

Health checks often report whether a process is alive and dependencies respond locally.

Better model:

A replica can be alive and still not know enough to accept a globally authoritative write. Coordination requires reachability to enough peers, not just process liveness.

Confusion: "Availability through partition is free"

Why it is tempting:

Keeping writes open sounds better than rejecting user work.

Better model:

Writable partitions move cost into staleness, divergent histories, merge rules, audit trails, and user-facing boundaries about what was really guaranteed.

Practice

Review this incident note:

During a zone incident, gateway A timed out calling certificate service C.
Progress replicas A and B could communicate.
Progress replica C could communicate with B, but replies from C to A were dropped.
Recommendation hints continued to be accepted in A and C.
Certificates were temporarily disabled.

Answer these questions:

1. What can A safely conclude from the timeout to C?
2. Can A and B commit a progress write if quorum is 2 of 3?
3. Why is disabling certificates a reasonable fail-closed choice?
4. What repair rule do recommendation hints need after the partition heals?

Model answer:

A can conclude only that it did not receive a useful response from C before its deadline. It cannot prove that C is dead or that B cannot reach C. If A and B both acknowledge and quorum is 2 of 3, they can commit the progress write, although C must catch up later. Disabling certificates is reasonable because certificate issuance is an authoritative user-visible promise. Recommendation hints need a merge or deduplication rule, such as union by stable hint id or bounded latest-event merge, so divergent local histories can be repaired.

Trade-offs and Limits

Failure models improve design clarity. They force the system to say what kind of uncertainty it can survive: crash, delay, loss, overload, asymmetric communication, minority partition, or total isolation.

They cost complexity. A richer failure model needs more careful protocols, clearer incident language, better telemetry, and more explicit user-facing behavior. A simpler model is easier to reason about, but it may make the system overconfident when real networks fail in messier ways.

This lesson does not teach consensus algorithms in depth. It uses quorum as a visible example of one design response to partitions. Full leader election, log replication, read leases, CRDTs, and formal proofs belong in later tracks.

You can see the boundary when a team says "C was down" but the evidence only says "A did not hear from C." That small language correction can prevent a large design mistake.

Resources

• [BOOK] Designing Data-Intensive Applications - Revisit network partitions, replication trade-offs, and why coordination under uncertainty forces real choices.
• [PAPER] Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services - Read CAP as a statement about behavior under partition, not as a product taxonomy.
• [PAPER] In Search of an Understandable Consensus Algorithm - See how leader terms and quorum rules respond to uncertain communication.

Key Takeaways

• A failure model names what the system can no longer know reliably, not just which component broke.
• A timeout is a local observation; it does not prove crash, loss, overload, or partition by itself.
• Quorum systems preserve authoritative state by refusing some work when reachability is uncertain.
• Staying writable during a partition usually means accepting divergence and designing explicit reconciliation rules.
• Good partition design chooses behavior per operation: fail closed for authority, degrade for optional reads, and reconcile only when the domain can tolerate it.