Network Failure Design Review
LESSON
Network Failure Design Review
By the end of this lesson, you will be able to...
Review a networked service design for unsafe timeout, retry, routing, and partition behavior.
Write a short failure-design memo that connects each operation class to policy and evidence.
Explain which remaining concerns belong in deeper protocol, HTTP, platform, reliability, or consensus tracks.
Idea in one sentence: A good network failure design says what each boundary can know, what it is allowed to do, and what evidence will prove the behavior during an incident.
Core Insight
Imagine the learning platform is preparing launch week for a large cohort.
The product must:
- serve lesson pages
- record progress
- issue certificates
- send notifications
- show recommendations
Traffic will rise sharply. One zone is known to have intermittent packet loss. The team asks for a "resilient network design."
The naive answer is a diagram with more boxes:
add a load balancer
add retries
add caches
add health checks
add tracing
Those pieces may help. But by themselves they do not describe safe behavior.
A convincing design review starts with promises:
Which requests may use stale data?
Which writes must not duplicate?
Which operations can fail open?
Which operations must fail closed?
Which retries are safe?
Which health signal removes a replica from traffic?
Which telemetry proves what happened after an incident?
The hard part is that these promises cross boundaries. A client deadline affects a gateway retry. A schema choice affects whether the receiver can deduplicate a write. A readiness check affects routing, but only if it checks the dependency needed by that request class. A trace can explain a timeout, but only if the request identity survives the retry and the async follow-up work.
The network does not become reliable by pretending ambiguity is gone. It becomes manageable when each boundary says what it knows, what it cannot know, and what it is allowed to do next.
This capstone asks you to produce that kind of design review.
Capstone Scenario
You are reviewing the network-facing behavior for the learning platform.
The platform has these services:
frontend
-> gateway
-> catalog-service
-> progress-service
-> certificate-service
-> recommendation-service
-> notification queue
It runs in three zones:
zone A: normal
zone B: normal
zone C: intermittent packet loss and occasional partition from A/B
The product promises:
- Lesson catalog reads should stay fast. Slightly stale data is acceptable.
- Progress writes must not be duplicated, even if the client or gateway retries.
- Certificate issuance must fail closed if authoritative progress state is uncertain.
- Recommendations are optional. They may disappear from the page.
- Notifications may be delayed, but should not imply the progress write failed.
- Operators need enough telemetry to explain whether an incident came from routing, retries, dependency latency, stale discovery, partition, or application semantics.
Your output is a short design review memo.
It should explain behavior under failure. It should not only list technologies.
The Design Review Frame
Use this frame:
1. Operation classes
2. Deadline and retry policy
3. Routing, health, discovery, and identity policy
4. Partition and dependency-failure behavior
5. Observability evidence
6. Tests or game days that prove the design
Plain meaning:
A design review is a structured way to ask, "What will this system do when the network gives each component only partial information?"
In this scenario:
The frontend, gateway, services, replicas, and operators all see different facts. The memo connects those facts into safe behavior.
Technical name:
This is a failure-mode design review: a review of expected behavior under delay, loss, retry, stale discovery, degraded dependencies, and partition.
Step 1: Classify Operations By Meaning
Start at the application boundary.
Do not assign one network policy to every request. First, separate operations by what they mean.
catalog read:
may be stale
safe to retry
may use cache
progress write:
changes user state
needs idempotency key
retry only through deduplication
certificate issue:
authoritative side effect
fail closed under uncertainty
no blind duplicate issuance
recommendation read:
optional
tiny deadline
fail open by omission
notification send:
async side effect
may be delayed
must be linked to original progress write
This step is where many weak designs fail. They talk about "requests" as if every request has the same risk.
The same timeout has different meaning for each operation:
| Operation | If it times out | Safe default |
|---|---|---|
| Catalog read | user may see older metadata | serve bounded stale cache |
| Progress write | caller does not know whether write committed | retry only with idempotency key |
| Certificate issue | authority is uncertain | fail closed |
| Recommendation read | optional feature missing | omit recommendations |
| Notification send | async work delayed | queue and expose delay |
Check: The gateway times out while issuing a certificate. Should it blindly retry certificate issuance because retry helped catalog reads?
Think first, then reveal.
Answer: No. Catalog reads and certificate issuance have different semantics. Certificate issuance is authoritative and user-visible. The design should fail closed or retry only through an explicitly deduplicated issuance path.
Step 2: Attach Deadline And Retry Policy
Now turn operation meaning into communication policy.
The policy should name:
- total deadline
- retry count
- retry condition
- backoff or jitter
- idempotency requirement
- what to do when the deadline is nearly gone
A useful starting matrix:
| Operation | Deadline | Retry policy | Stop condition | User behavior |
|---|---|---|---|---|
| Catalog read | short | one quick retry or cache fallback | cache too old or deadline nearly gone | show page or read error |
| Progress write | moderate | retry only with idempotency key and remaining deadline | no safe write target or ambiguous result | show retryable/ambiguous state |
| Certificate issue | strict authority check | no blind retry of side effect | cannot confirm authoritative progress | fail closed |
| Recommendation read | tiny | usually no retry | optional budget spent | omit block |
| Notification send | async budget | retry from queue with idempotent job key | poison or repeated failure | delay and surface operational alert |
Notice that "retry" is not one setting. It is attached to operation meaning.
The design should also explain where retries happen. A client retry, gateway retry, service retry, queue retry, and worker retry can multiply each other. If all layers retry without a shared budget, recovery logic can become the outage amplifier from lesson 3.
Capstone rule:
Every retry needs a reason, a budget, and a way to avoid duplicate side effects.
Step 3: Review Routing, Health, Discovery, And Identity
The next question is where traffic is allowed to go.
For each operation class, ask:
Which endpoint is a responsible destination for this request?
Do not let a shallow 200 OK health endpoint answer that question alone.
A progress replica may be alive but unable to reach authoritative storage. A catalog replica may be unable to reach origin but still able to serve bounded stale data. A certificate service may be reachable but running in a minority partition. A recommendation service may be slow, but optional.
Review these routing rules:
| Request class | Required routing evidence | Unsafe shortcut |
|---|---|---|
| Catalog read | local cache health, cache age, stale limit | treating stale data as always acceptable |
| Progress write | write dependency reachable, idempotency accepted, identity valid | routing to any alive replica |
| Certificate issue | authoritative progress state confirmed, no minority partition | issuing from uncertain state |
| Recommendation read | optional dependency under tiny budget | waiting long enough to consume page deadline |
| Notification job | queue health, dedupe key, worker identity | assuming send failure means progress failed |
Discovery and identity belong here too.
The service name progress-service is stable. The endpoint list behind it is temporary. A stale discovery cache may still include a draining endpoint. The router should record the service name, candidate endpoints, selected endpoint, rejected endpoints, route reason, and identity result.
Useful routing evidence:
service_name = progress-service
selected_endpoint = P1
endpoint_version = v2
zone = A
route_reason = write_ready and identity_valid
rejected_endpoint = P3 draining
discovery_cache_age = 4s
identity_result = success
This is the bridge from lessons 5 and 6. Health supplies evidence. Discovery supplies candidates. Routing makes the request-specific choice. Identity prevents blind trust.
Step 4: Design For Partition And Packet Loss
Now handle the known launch risk: zone C has intermittent packet loss and occasional partition from zones A and B.
During a partition, components can be alive and still disagree about the world.
That means the design should separate local observation from global truth:
local observation:
zone C can receive some traffic
unsafe conclusion:
zone C is safe for all writes and certificate issuance
better conclusion:
zone C may serve only operations whose semantics are safe under this uncertainty
A possible policy:
| Condition | Catalog reads | Progress writes | Certificate issuance | Recommendations |
|---|---|---|---|---|
| Zone C has packet loss but can reach authority | allow with bounded latency and fallback | allow if write dependency and idempotency path are healthy | allow only with authority confirmed | allow or omit |
| Zone C is minority partition | serve bounded stale catalog only | reject or route to majority if reachable | fail closed | omit |
| Discovery is stale in zone C | refresh or suppress unsafe writes | reject writes to draining/unknown endpoints | fail closed | omit if uncertain |
The design does not need full consensus depth. That belongs in downstream coordination tracks. But it must not confuse "the process is up" with "the operation is authoritative."
Check: Zone C can answer HTTP health checks but cannot reach the authoritative progress store. Which operations should it still serve?
Think first, then reveal.
Answer: It may serve bounded stale catalog reads and optional recommendation behavior if those policies allow it. It should not accept authoritative progress writes or issue certificates that depend on confirmed progress state.
Step 5: Make Evidence Part Of The Design
A failure design is weak if it says only what should happen.
It also needs to say how operators will know what happened.
For each important behavior, name the evidence:
| Behavior | Evidence |
|---|---|
| Catalog served stale data | trace field catalog_cache_age, log fallback=stale_cache |
| Progress write retried safely | shared request ID, idempotency key, attempt count, dedupe result |
| Certificate failed closed | log reason progress_authority_uncertain, metric count |
| Recommendation omitted | trace span recommendations_skipped, degraded-success metric |
| Zone C partition detected | zone reachability metric, packet loss metric, route suppression event |
| Draining endpoint avoided | route log with rejected endpoint and draining reason |
| Stale discovery caused first attempt | discovery cache age, selected endpoint, refresh event |
This is the lesson 7 habit: observe decisions, not only symptoms.
For launch week, the most useful trace might show:
trace_id = tr-launch-42
operation = complete_lesson
deadline_ms = 10000
attempt_1:
endpoint = P3
zone = C
route_reason = cached_discovery_candidate
result = timeout
attempt_2:
endpoint = P1
zone = A
route_reason = write_ready
idempotency_key = req-42
dedupe_result = first_commit
final:
user_status = 200
notification_deferred = true
degraded_success = true
That trace lets the team explain a slow success without pretending that success was clean.
A Worked Review: From Weak Design To Testable Design
Here is a weak design memo:
We will make the platform resilient by adding retries, load balancing,
health checks, service discovery, and observability. The gateway will retry
failed requests. Services will be monitored. Traffic will go to healthy nodes.
It sounds reasonable, but it hides the hard decisions.
Review it using the track vocabulary:
Input:
launch-week learning platform design
known packet loss in zone C
mixed operations: reads, writes, certificates, optional recommendations
Transition:
classify operations
attach timeout/retry rules
inspect routing and health evidence
define partition behavior
require telemetry for each policy decision
Intermediate state:
catalog reads are stale-tolerant
progress writes need idempotency
certificates need confirmed authority
recommendations are optional
zone C cannot be trusted for every operation during partition
Output or decision:
approve only if policies and evidence are operation-specific
request changes if retry, health, routing, or observability are generic
Naive failure contrast:
"retry failed requests" duplicates writes
"route to healthy nodes" sends writes to a replica without write dependency
"monitor services" misses degraded success and stale discovery
Now rewrite the weak memo into a testable one:
Catalog reads use a 300 ms budget and may fall back to local cache if cache age
is under 60 seconds. The trace records `catalog_cache_age` and `fallback=stale`.
Progress writes require an idempotency key. The gateway may retry once with
jitter only if at least 500 ms of deadline remains. The receiver stores the
idempotency key and logs `first_commit` or `duplicate_suppressed`.
Certificate issuance requires confirmed authoritative progress state. If the
service is in a minority partition or cannot confirm authority, it fails closed
and records `progress_authority_uncertain`.
Recommendations use a 150 ms optional budget. If they time out, the page omits
them and records degraded success, but the page request continues.
Routing uses request-class readiness. A replica that cannot reach the progress
store is not write-ready, even if it is alive. Discovery logs candidate
endpoints, selected endpoint, cache age, and rejected draining endpoints.
The revised memo is not perfect. But it is reviewable. Each statement can be tested.
Capstone Task
Write your own design review memo for the launch-week platform.
Use this structure:
1. Operation classes
2. Timeout and retry policy
3. Routing, health, discovery, and identity policy
4. Partition behavior
5. Observability evidence
6. Validation plan
Your memo should fit on one or two pages.
It should include:
- at least four operation classes
- one policy matrix
- one partition behavior rule
- one stale discovery behavior rule
- one degraded-success signal
- one game-day or fault-injection test
- one explicit boundary saying what this track does not cover
The boundary matters. This track does not ask you to design TCP congestion control, a full consensus protocol, or a CDN strategy. It asks you to review application-level behavior at network boundaries.
Rubric
Use this rubric to evaluate the memo.
| Criterion | Strong answer | Weak answer |
|---|---|---|
| Operation semantics | Separates stale reads, idempotent writes, authoritative side effects, and optional work | Uses one generic request policy |
| Retry safety | Names budget, deadline, idempotency, and duplicate behavior | Says "retry failures" |
| Routing and health | Uses request-class readiness and route evidence | Routes to any alive endpoint |
| Discovery and identity | Separates service name, endpoint, freshness, and trust | Treats lookup result as automatically safe |
| Partition behavior | Fails closed for authoritative operations under uncertainty | Lets minority partition continue all writes |
| Observability | Records decision evidence and degraded success | Records only final errors |
| Scope control | Defers protocol, HTTP/CDN, consensus, and reliability-program depth to later tracks | Tries to solve every networking topic here |
| Testability | Includes game-day or fault-injection checks | Cannot be tested before launch |
Minimum passing memo:
The reader can point to each operation and answer:
What can fail?
What should the system do?
What evidence proves it did that?
Validation Plan
A design review is stronger when it names how to test the behavior before launch.
Choose a small set of game days:
| Test | Expected behavior | Evidence |
|---|---|---|
| Add packet loss in zone C | optional dependencies degrade, authoritative work avoids unsafe paths | zone metrics, route logs, degraded-success counters |
| Drain progress endpoint P3 | new writes stop routing to P3, in-flight work finishes or rejects safely | draining event, selected endpoint trace |
| Delay progress database | write readiness drops, retries remain within budget | readiness metric, retry count, deadline remaining |
| Expire discovery cache slowly | stale endpoint is rejected or refreshed before unsafe write | discovery cache age, rejected endpoint reason |
| Duplicate progress write request | receiver suppresses duplicate side effect | idempotency log duplicate_suppressed |
| Partition certificate service from progress authority | certificate issuance fails closed | fail-closed metric and reason log |
You do not need dozens of tests. You need enough targeted tests to prove the design's promises.
Trade-off And Boundary
The trade-off is control versus complexity.
More explicit network failure policy gives the team safer retries, clearer routing, better degraded behavior, and stronger incident evidence. It also creates more configuration, more telemetry fields, more tests, and more ownership. A route rule nobody understands can become a new failure mode. A retry budget that is not measured can drift into a hidden overload source. A trace field that is not used during incidents can become noise.
That is why the capstone asks for a small memo, not a giant architecture catalog. The design should cover the few decisions that change safety:
Can this operation be stale?
Can this operation repeat?
Can this endpoint answer responsibly?
Can this zone prove authority?
Can operators prove what happened?
It does not need to solve every network problem. Packet-level transport depth belongs in network-protocols-and-transport-systems. HTTP caching and CDN strategy belong in http-protocol-and-content-delivery. Formal agreement and leader election belong in consensus-and-coordination. Reliability-program ownership belongs in reliability-engineering-foundations.
Common Design Mistakes
Mistake: One Policy For Every Operation
Why it is tempting:
It is simpler to configure one timeout, one retry count, and one health rule.
Better model:
Different operations carry different risk. Optional reads, side-effecting writes, and authoritative decisions need different behavior.
Mistake: Health Without Request Meaning
Why it is tempting:
A single health endpoint is easy to wire into load balancing.
Better model:
Health means "responsible destination for this request." A replica can be alive and still unsafe for writes, certificates, or new work during draining.
Mistake: Observability Only After Something Fails
Why it is tempting:
Teams often instrument errors first.
Better model:
Partial failure often appears as slow success, stale cache, retry, fallback, or skipped optional work. Those paths need evidence too.
Resources
- [BOOK] Designing Data-Intensive Applications - Revisit partitions, consistency, replication, and operational trade-offs as the deeper downstream context.
- [ARTICLE] Timeouts, Retries, and Backoff with Jitter - Review retry behavior as a design surface rather than a library default.
- [ARTICLE] OpenTelemetry Concepts - Connect request context, traces, metrics, and logs to failure diagnosis.
- [TUTORIAL] Kubernetes Services - Use service names, endpoints, and probes as concrete routing inputs.
Key Takeaways
- Network resilience starts from operation semantics, not from adding generic infrastructure.
- Timeouts, retries, routing, health, discovery, identity, and observability form one failure policy.
- A good design review names both safe behavior and the evidence that proves it happened.
- Authoritative operations should fail closed when partitions or stale discovery make authority uncertain.
- The capstone skill is transfer: review a small networked system and explain what each boundary can know, decide, and prove.