Runbooks: Symptoms, Dashboards, Commands, and Rollback

By the end of this lesson, you will be able to...

• Turn an operational symptom into a runbook path with checks, bounded actions, verification, and escalation.

• Distinguish a dashboard signal from a diagnosis and a command from a safe decision.

• Add stop conditions and rollback notes that prevent a runbook from becoming an unsafe command list.

Idea in one sentence: A runbook helps an operator move from a visible symptom to the next safe action, while making the evidence, boundary, and exit conditions easy to find under pressure.

Core Insight

At 09:12, Imani receives an alert: Import queue age above 15 minutes. A customer has uploaded a file, but the dashboard still says queued. Support asks whether the team can promise completion before a meeting at 10:00.

The existing “runbook” says only this:

If imports are slow, restart the workers.

That sentence is fast to read. It is not safe to use. Which workers? Are they actually stuck? Is the database unavailable? Would a restart abandon a job that is writing rows? When should Imani stop trying this and ask another team for help?

A runbook is not an explanation of every possible system failure. It is a decision path for a repeated operational situation. Its job is to make the first signal, the next check, the allowed action, the success condition, and the escalation boundary visible before urgency turns guessing into action.

The Production Symptom

The small system has four moving parts:

customer upload -> import queue -> import workers -> database
                                      |
                                      v
                               import status page

Imani can see a queue-age alert and a customer-facing queued status. She cannot yet see the cause. Several different conditions can create the same symptom:

• workers have no database connections;
• the queue is receiving more jobs than workers finish;
• one large file is slow but normal; or
• a deployment stopped workers from claiming jobs.

The naive model is: alert equals cause. If the queue is old, restart workers.

It works only when workers are genuinely wedged and restarting them is known to be safe. It breaks when the queue is healthy but busy, or when the database is the limiting component. A restart can add churn, hide evidence, and delay the work that is already making progress.

Plain meaning:

An alert tells you where to look. It does not tell you what to change.

In Imani's situation:

The alert says that at least some jobs have waited too long. She needs a small investigation before she decides whether to pause intake, restart a worker, or escalate the database problem.

Technical name:

This is an operational decision path. A runbook connects symptom, evidence, mitigation, verification, rollback, and escalation in an order an operator can follow.

What a Runbook Must Promise

A tutorial teaches a newcomer how a system behaves. An endpoint reference helps a reader look up a contract. A runbook helps someone act while the system is behaving badly.

Its promise should be narrow and explicit:

For an import queue whose age is above 15 minutes, identify whether workers are processing jobs, take only the documented low-risk mitigation, verify recovery, and escalate when the evidence points outside this service.

Notice what this promise excludes. It does not guarantee that every import finishes. It does not authorize an operator to change database capacity. It does not teach the whole import architecture. Those limits make the instructions safer, not less useful.

The precise values are fictional. The writing pattern is the point: every instruction couples an observation to a bounded decision.

The Evidence Path

Dashboards are useful because they turn hidden state into signals. They can also create false certainty when the runbook names a chart without saying what comparison matters.

For this alert, Imani opens one small dashboard view:

The evidence path makes a causal question inspectable. The charts do not prove a root cause. They help Imani choose the next safe branch.

Check: Queue age is high, worker claim rate is still normal, and one import is much larger than the rest. Should the runbook immediately restart workers?

Think first, then reveal.

Answer: No. The evidence does not show stuck workers. Record the unusually large job, check whether its duration is within documented limits, and keep monitoring or escalate only if the defined threshold is crossed. Restarting would act on the alert while ignoring the more specific evidence.

A Worked Runbook Slice

Here is the decision path Imani needs. The command names are fictional; a real runbook must use commands that the team has tested and authorized.

Trigger and safety note

Start when queue_age_p95 is above 15 minutes for 5 continuous minutes.

Do not restart workers that report writing_rows. Interrupting that state can leave an import incomplete. Capture the incident time and current dashboard values before changing anything.

Investigation

1. Check queue age, worker claim rate, failed-job rate, and database connection saturation for the same 15-minute window.
2. Run importsctl workers status and record each worker as idle, claiming, writing_rows, or unhealthy.
3. Compare the current deployment time with the first rise in queue age.

Decision table

One trace through the table

At 09:12, queue age is 18 minutes. Claim rate is 0/min; database saturation is 42%; three workers show unhealthy; none show writing_rows.

The naive failure would have restarted every worker at 09:12. That could have interrupted active work if the alert had a different cause. The runbook made the intermediate evidence and the smallest reversible action visible.

So far, we have used the symptom to choose a check, the check to choose one mitigation, and a measured signal to decide whether the mitigation worked.

Rollback, Escalation, and Prevention

Rollback belongs in a runbook whenever an action changes live behavior. Before pausing intake, record the feature flag's previous value. The rollback instruction can be simple:

If database saturation remains above 90% after 10 minutes, restore import intake to its previous value unless database on-call directs otherwise. Add the observed values and incident link to the escalation note.

This prevents two common errors: leaving an emergency control enabled after the original condition changes, and letting a later operator guess what was changed.

Escalation is not an admission of defeat. It is a boundary in the document. The runbook should say who owns the next decision, what evidence to send, and what the current mitigation has already done.

For Imani, the escalation note is:

Database saturation remains above 90% after intake was paused for 10 minutes. Queue age is 27 minutes, claim rate is 1/min, and no workers are unhealthy. Database on-call: assess connection-pool capacity. Import on-call: keep intake paused until a shared decision is recorded.

The prevention section is smaller and later. After recovery, create a follow-up if the same alert lacked a useful branch, a dashboard signal was missing, or the allowed action was too broad. The runbook is a maintained operational document, not a one-time command transcript.

Check: Why does “restart one worker” include a 5-minute stop condition?

Think first, then reveal.

Answer: The stop condition limits the action and turns its outcome into evidence. If one restart does not restore claims, repeating the command adds risk without supporting the worker-stuck hypothesis. The runbook then directs the operator toward escalation or another branch.

Trade-offs and Limits

A runbook improves speed, consistency, and handoff quality under pressure. The trade-off is upkeep and local knowledge: every dashboard name, threshold, command, and rollback must stay accurate as the system changes.

It does not replace judgment. A page cannot predict every interaction, and a dashboard cannot establish a root cause by itself. The boundary appears when none of the documented branches match, a command's safety condition is unknown, or user impact grows beyond the runbook's stated scope. Stop, preserve evidence, and escalate rather than inventing a new risky procedure in the middle of an incident.

The strongest signal that a runbook needs revision is repeated hesitation: operators ask what a chart means, choose different actions for the same evidence, or leave an emergency setting on because rollback was unclear. Those are document defects that can be fixed before the next alert.

Practice

Write a one-page runbook slice for a familiar alert: failed background jobs, a full disk, a webhook backlog, or a slow search index. Begin with a concrete trigger and user impact. Then add three signals, one small decision table, one reversible action, a verification condition, and an escalation note.

A good answer should:

• distinguish the symptom from the suspected cause;
• state what each signal changes in the decision;
• limit every command with a safety or stop condition;
• say how to verify recovery and how to restore a changed setting; and
• name the owner and evidence for the next escalation.

Resources

• [BOOK/TUTORIAL] Google SRE Workbook: Incident Response — Focus: Notice the distinction between mitigating impact, managing the response, keeping a record, and escalating early.
• [ARTICLE] Google SRE: Anatomy of an incident management handbook — Focus: Use the preparedness, response, recovery, and mitigation lifecycle to check the boundary of an operational document.
• [TUTORIAL] Google Technical Writing: General error handling rules — Focus: Connect a specific error, its root condition, and a useful action instead of reporting a generic failure.
• [TUTORIAL] Google Technical Writing: Set the right tone — Focus: Write neutral, instructive recovery language that explains what to do without blaming the operator.

Key Takeaways

• An alert is a starting signal, not a diagnosis or an automatic command.
• A useful runbook connects symptom, evidence, bounded action, verification, rollback, and escalation.
• Dashboard signals become useful when the document states which comparison changes the next decision.
• Stop conditions make an action reversible and prevent repeated commands from becoming a substitute for diagnosis.
• A runbook needs maintenance whenever its signals, commands, thresholds, ownership, or rollback behavior changes.