Degraded Modes, Playbooks, and Incident Evidence
LESSON
Degraded Modes, Playbooks, and Incident Evidence
By the end of this lesson, you will be able to...
define a degraded mode as a smaller tested promise, not accidental half-failure.
trace the evidence needed to enter, operate, repair, and exit a degraded workflow.
review a playbook for missing controls, user states, repair data, and recovery signals.
Idea in one sentence: A degraded mode keeps the system honest by replacing an unsafe normal promise with a smaller promise that the system can still prove and repair.
Core Insight
A live learning platform runs a timed quiz during an online class.
Normally it promises:
students submit answers
the grading service scores them immediately
the teacher sees a live leaderboard
students see their score
At 10:03, the grading service becomes slow and starts returning timeouts. Some submissions may have reached the grading service. Some may not. The quiz API cannot prove which ones were scored.
The naive response is to keep pretending the normal promise is available:
"Submit your answer. Your score will appear now."
That promise is no longer safe. It may lose submissions, duplicate grading work, show unfair leaderboard results, or tell students that they failed to submit when the platform simply cannot prove the outcome yet.
A better response is smaller:
"Submission received. Grading is delayed."
The platform keeps accepting answer submissions if it can store them durably. It disables the live leaderboard. It shows a receipt. It grades later when evidence is available.
Plain meaning:
A degraded mode is planned behavior for a bad condition. The system stops making the full normal promise and switches to a smaller promise it can keep honestly.
In this scenario:
The platform cannot promise immediate scoring, but it can promise durable receipt of answers, visible pending status, bounded repair, and later grading.
Technical name:
That smaller planned behavior is a degraded mode. The operational instructions for entering, running, repairing, and exiting it are a playbook. The facts used to make those decisions are incident evidence.
The Naive Idea: Let Normal Mode Struggle
The first version of the system has one path:
student browser -> quiz API -> grading service -> result store -> leaderboard
When grading is healthy, this path is fine.
submit answer
-> grade immediately
-> store result
-> update leaderboard
-> show score
When grading is unhealthy, the same path becomes ambiguous.
submit answer
-> quiz API calls grading
-> grading times out
-> browser retries
-> grading may receive duplicates
-> leaderboard sees partial results
The timeout does not prove the answer was ungraded. It proves only that the caller stopped waiting. The grading service may have accepted the work and failed to respond in time.
If the UI says "submission failed," students retry. If the UI says "score saved," the platform may be inventing a result. If the leaderboard continues, it may rank students based on whichever requests happened to complete during the incident.
The normal mode is now lying by accident.
Check: If the grading call times out, what does the quiz API know for sure?
Think first, then reveal.
Answer: It knows that it did not receive a grading response before its deadline. It does not know whether the grading service accepted, scored, rejected, or ignored the submission unless there is durable evidence for that outcome.
A Smaller Promise
A degraded mode starts by naming the promise that remains true.
Normal quiz promise:
submit answer
receive immediate score
teacher sees live leaderboard
Grading-degraded promise:
submit answer
receive durable submission receipt
see grading_pending
score appears later after repair
leaderboard is paused or hidden
This smaller promise changes product behavior, storage behavior, and operational behavior.
The system needs a durable submission record:
submission_id: sub-91
quiz_id: quiz-7
student_id: student-14
answer_hash: h123
received_at: 10:03:12
grading_state: pending
grading_operation_id: grade:sub-91
That record is the anchor for repair. It lets the system say, "We received your answer," without pretending it knows the score.
The degraded mode also needs a user state. grading_pending is different from failed. It tells the student not to resubmit. It tells support what happened. It tells repair workers which submissions still need grading.
A Worked Trace: Enter, Operate, Repair, Exit
Follow the incident as a sequence of decisions.
1. Input: Evidence Of Unsafe Normal Mode
At 10:03, metrics show:
grading p99 latency: 9 seconds
quiz API deadline: 2 seconds
grading timeout rate: 28%
browser retry rate: rising
leaderboard gaps: visible
pending submissions: growing
Naive decision:
keep normal mode
let browsers retry
keep showing live leaderboard
Naive intermediate state:
some students see scores
some students see failures
some answers may be scored twice
leaderboard is incomplete
support cannot tell received from lost
The incident is not only "grading is slow." The incident is "the platform is making a promise it cannot prove."
2. Transition: Enter Degraded Mode
The playbook gives an entry rule:
enter grading-degraded mode when:
grading timeout rate > 10% for 3 minutes
OR pending submissions oldest age > 2 minutes
OR leaderboard completeness < 95%
Entering the mode changes the path:
student browser -> quiz API -> durable submission store
-> repair queue
leaderboard -> paused
score view -> grading_pending
Output:
Students receive a receipt. Teachers see that live scoring is delayed. The platform stops claiming immediate ranking.
This is a controlled promise change, not a random failure.
3. Transition: Stop Amplification
The playbook also stops behavior that makes the incident worse.
controls:
reject duplicate submissions with same submission_id
reuse grading_operation_id for repair
pause leaderboard refresh
rate-limit status polling
disable browser auto-retry for grading result
These controls connect to earlier lessons. Idempotency prevents duplicate grading for one answer. Backpressure keeps polling and repair from overwhelming the grading service. Observability joins the student's receipt to the repair job.
Without these controls, the degraded mode is cosmetic. It changes the text in the UI but not the pressure on the system.
Check: Why is hiding the leaderboard not enough by itself?
Think first, then reveal.
Answer: Hiding the leaderboard avoids showing unfair partial results, but it does not preserve submissions, stop duplicate grading, or create repair evidence. A degraded mode needs controls and durable state, not only a visual change.
4. Transition: Repair From Durable Evidence
Repair workers process pending submissions.
repair input:
submission_id=sub-91
grading_operation_id=grade:sub-91
answer_hash=h123
grading_state=pending
Before calling the grading service, the worker checks whether grade:sub-91 already has a result. If yes, it records that result once. If not, it calls grading with the same operation id and writes exactly one final state.
pending -> graded(score=8)
pending -> manual_review(reason=invalid_question)
pending -> failed_repair(reason=grading_unavailable)
The repair path should be boring and inspectable. A student, teacher, support person, and engineer should all be able to trace one submission from receipt to final outcome.
5. Output: Exit Gradually
Exit evidence is not "the alert stopped."
The playbook requires:
exit when:
grading p99 latency < 1 second for 15 minutes
pending queue oldest age < 30 seconds
browser retry rate is normal
leaderboard completeness is verified
sample submissions trace from receipt to score
Then the platform ramps back:
restore immediate scoring for 10% of quizzes
watch timeouts and duplicate-operation counts
restore live leaderboard
keep repair worker active until backlog is empty
Recovery is part of the mode. If the platform flips everything on at once, old browser retries and a repair backlog can recreate the incident.
Playbooks Are Design Requirements
A playbook is not only a document for an on-call person. It describes features the system must already have.
If the playbook says "show grading_pending," the product must have that state. If it says "pause leaderboard," the leaderboard must be controllable independently. If it says "repair later," submissions must be durable and uniquely identified. If it says "exit after completeness is verified," the system needs a way to measure completeness.
playbook step:
pause live leaderboard
software requirement:
leaderboard can be disabled without disabling answer submission
playbook step:
repair pending grading
software requirement:
submission_id, answer_hash, grading_operation_id, and grading_state are durable
playbook step:
prove recovery
software requirement:
trace from submission receipt to final grade exists
This is why degraded modes should be designed before the incident. During the incident, the team should be executing a tested smaller promise, not inventing one under pressure.
Incident Evidence
Incident evidence is the set of facts that lets responders act without pretending they know more than they do.
For this quiz incident, evidence falls into four groups.
Trigger evidence explains why normal mode became unsafe:
grading latency
timeout rate
leaderboard completeness
pending queue age
retry rate
User-impact evidence shows what learners and teachers experienced:
submissions received
submissions pending
scores delayed
classes affected
support contacts
Amplification evidence shows whether the system is worsening the incident:
browser retries
duplicate submission ids
status polling rate
repair queue growth
grading active concurrency
Recovery evidence proves the system can leave degraded mode:
healthy grading latency
pending queue drained
retry rate normal
sample traces complete
leaderboard complete
Good evidence narrows uncertainty. It does not need to be beautiful. It needs to be joinable, current, and tied to the promise.
Rehearsal Makes The Mode Real
A degraded mode should be tested before the real incident.
For the live quiz, a rehearsal can be small:
1. Inject grading timeouts for one test quiz.
2. Confirm the entry rule switches the quiz to grading-degraded mode.
3. Submit one answer and receive a durable receipt.
4. Verify the leaderboard pauses.
5. Run the repair worker and produce one final score.
6. Exit gradually after recovery evidence is present.
This rehearsal finds design gaps while they are still cheap. Maybe grading_pending exists in the API but not in the teacher dashboard. Maybe the repair worker needs answer_hash but the submission store does not keep it. Maybe the leaderboard cannot be paused for one quiz without pausing every class.
Those are not documentation problems. They are missing controls. The playbook has done its job by making them visible.
Common Confusions
Confusion: Degraded Mode Means Broken Mode
Why it is tempting:
The system is not doing everything it normally does.
Better model:
A degraded mode is a named smaller promise. Broken mode is accidental behavior. The difference is whether the system can explain what it still guarantees.
Confusion: A Playbook Is Only For Humans
Why it is tempting:
Playbooks are often written as checklists.
Better model:
A useful playbook reveals software requirements. If a step cannot be executed by an existing control, state, query, or repair path, the playbook is describing missing design work.
Confusion: Recovery Means Turning Normal Mode Back On
Why it is tempting:
Once the dependency looks healthy, normal behavior feels safe again.
Better model:
Recovery must include backlog, retries, repair state, and user-visible evidence. A dependency can be healthy while the system still has old obligations to drain.
Trade-offs And Limits
The central trade-off is completeness versus honesty.
Normal mode gives the richest experience: immediate scoring, live ranking, and instant feedback. During the incident, that rich promise becomes unsafe. It may reward whichever requests completed by chance.
Degraded mode gives a smaller experience: receipt now, score later, no live leaderboard. That is less satisfying, but it protects fairness and preserves the facts needed for repair.
The costs are real:
- more product states;
- more durable records;
- more operational controls;
- more rehearsal work;
- more careful recovery.
There is also an ownership cost. Someone must decide when the mode is still correct, when the entry threshold is too sensitive, and when the repair path no longer matches the product. A degraded mode without an owner quietly becomes stale. The next incident then starts with an old promise that nobody fully trusts.
It can still fail if the degraded state is vague, if repair data is missing, if users are encouraged to retry, or if the team exits before the backlog is safe.
You can see the boundary when the system says "pending" but has no way to finish the pending work. That is not a degraded mode. That is uncertainty with a nicer label.
Practice: Review A Degraded Mode
Pick one workflow: password reset, file upload, message send, seat booking, live quiz, or account deletion.
Fill in:
normal promise:
degraded promise:
entry evidence:
user-visible state:
controls to stop amplification:
durable repair data:
exit evidence:
rehearsal test:
Model answer for live quiz:
normal promise:
submit answer, receive immediate score, update leaderboard
degraded promise:
submit answer, receive durable receipt, score later, leaderboard paused
entry evidence:
grading timeout rate, pending queue age, leaderboard completeness, retry rate
user-visible state:
grading_pending with submission receipt
controls to stop amplification:
idempotent submission ids, polling limits, paused leaderboard, bounded repair
durable repair data:
submission_id, answer_hash, grading_operation_id, grading_state
exit evidence:
grading latency healthy, queue drained, retries normal, sample traces complete
rehearsal test:
inject grading timeout, enter mode, submit answer, repair it, exit gradually
If the degraded promise cannot be tested before an incident, it is not ready yet.
Resources
- [BOOK] Site Reliability Engineering: Managing Incidents - Focus: Roles, evidence, and structured response during incidents.
- [BOOK] Site Reliability Engineering: Emergency Response - Focus: Prepared response, operational discipline, and avoiding second-order failures.
- [ARTICLE] Static Stability Using Availability Zones - Focus: Designing systems that keep a smaller reliable promise during dependency or zone failures.
Key Takeaways
- A degraded mode is a smaller tested promise, not whatever happens after the normal path breaks.
- Playbooks should map to real controls, states, queries, and repair paths.
- Incident evidence should cover trigger, user impact, amplification, and recovery.
- Recovery should be gradual because old retries, backlogs, and pending work can recreate the incident.
- A pending state is honest only when the system has durable evidence and a repair path behind it.