How to Write a Runbook On-Call Engineers Will Follow

A runbook that works at 3 a.m. does three things: it opens with something to run, it says what the output means, and every path ends in a fix or a named person to escalate to. Most runbooks fail the first test. They open with a paragraph about the service.

We have written and rewritten these for client systems for years, and the pattern is stable. This post gives you the template, a worked example, and the review habit that keeps runbooks from rotting. It builds on the alerting advice in incident response for small teams, which says every alert needs a linked runbook but not what goes in it.

Write for the person you will be at 3 a.m.

Picture the reader. They have been awake for ninety seconds. They may not own this service. They have a laptop, a terminal and a phone buzzing with the alert. They cannot hold more than two ideas in their head.

That reader does not need a history of the system. They need the next action. So the first rule is that a runbook describes what to do, and design explanations live somewhere else. If the reasoning matters, link to the architecture decision record and keep moving.

The second rule is one runbook per alert. A page titled “Database troubleshooting” is a wiki article, not a runbook. A page titled “HighReplicationLag firing” is a runbook, because the reader arrives with exactly one question.

The template

Every runbook we write has these sections, in this order:

1. What this alert means. One or two sentences, in terms of user impact. “Checkout requests are failing more than 2% of the time.”
2. First check. One command or one dashboard link, with the expected healthy result.
3. Decision steps. A short branching list: if you see A, do this; if you see B, do that.
4. Escalate when. An explicit trigger and a name or rotation. Not “if needed.”
5. After the incident. What to capture so the runbook improves.

The “escalate when” section matters more than it looks. Junior engineers lose the most time on the question “am I allowed to wake someone up?” Answer it in writing, in advance, with a time limit: “If error rate is still above 2% after 15 minutes, page the database lead.”

A worked example

Here is a trimmed runbook for a queue backlog alert. The details are invented for illustration, but the shape is what we use.

# Alert: EmailQueueBacklogHigh

**Means:** Outbound email is delayed. Password resets and receipts
are late. Customers cannot sign in if reset emails take over 5 minutes.

## First check
    kubectl -n prod get pods -l app=email-worker
Expect: 3 pods, all Running. If fewer than 3 are Running, go to Step 1.
If all 3 are Running, go to Step 2.

## Steps
1. Workers are crash-looping.
   - `kubectl -n prod logs deploy/email-worker --tail=50`
   - If you see `SMTP auth failed`, the provider key was rotated.
     Update the secret (see "Rotate SMTP key" below) and restart.
   - Any other error: escalate to the backend on-call.
2. Workers are up but the queue is not draining.
   - Check provider status page: <link>
   - If the provider is degraded, do nothing. Post in #incidents
     and let the backlog drain when they recover.
   - If the provider is fine, restart workers:
     `kubectl -n prod rollout restart deploy/email-worker`
     Expect the backlog metric to fall within 3 minutes.

## Escalate when
Backlog still rising 15 minutes after a restart. Page: backend on-call.

## After
Paste the log lines you saw into the incident thread.

Look at what the example leaves out. There is no description of how the email system works. There is no list of every possible failure. It covers the two causes that account for most pages, and it sends everything else to a human.

Details that separate good runbooks from decorative ones

State the expected output. “Run this command” is half an instruction. “Run this command, and expect three Running pods” tells the reader whether to continue or branch. Without it, a tired person guesses.

Prefer copy-paste commands over prose. Each command should run as written, with the namespace, cluster and flags filled in. If a value changes per incident, mark it clearly, like <pod-name>, and say where to find it.

Name the safe and the risky steps. Put a plain warning before anything destructive: restarts that drop in-flight work, failovers, data deletion. Better still, mark which steps are safe to repeat.

Link logs and dashboards by exact URL. A runbook that says “check the dashboard” has failed. If you log with correlation IDs, say how to search by one. Our guide to structured logging with correlation IDs covers the setup that makes this one-click.

Write for the second language. Plenty of on-call engineers read English as a second language, and plenty of incidents happen while someone is stressed. Short sentences and verbs first.

Keeping runbooks alive

A runbook is the first thing to go stale, because nothing fails when it does. Three habits help.

First, keep runbooks in the service repository and require a runbook change in the same pull request as any change that alters the alert or the procedure. Reviewers catch drift while the context is still fresh.

Second, make the post-incident review ask a standing question: did the runbook help, and what would have saved time? Edit it that day. A fix postponed to “later” gets lost.

Third, run a drill. Give the runbook to someone who did not write it, break a staging environment on purpose, and watch where they hesitate. Every pause is a missing sentence. We usually find three or four in the first session, which is a better return than most documentation work.

Delete aggressively. If an alert has not fired in a year, either retire the alert and its runbook, or confirm the check still works. Dead runbooks teach people to distrust the live ones.

Who benefits most

Small teams get the largest return. When two people carry the pager, a good runbook lets a third person cover a night. It also makes handoffs to a client’s in-house team smoother, which is the same logic behind a proper client handoff documentation package. If you are building out an operations function and want help standing up alerting and runbooks, our engineering services include it.

The takeaway: pick your noisiest alert this week, write its runbook in the template above, and give it to a colleague who has never touched that service. Whatever they ask you becomes the next sentence.