Incident response for web apps: runbooks, on-call and blameless post-mortems

Good incident response comes down to four things written down before you need them: how you decide how serious something is, who does what in the first few minutes, an incident response runbook people can follow under stress, and a blameless process for making sure it's harder to happen twice. None of this requires an enterprise tool budget — it requires a shared document and the discipline to use it every time, not just when it's convenient.

This post gives you a starting severity table, roles, a first-15-minutes checklist, templates, and a post-mortem structure adapted from Google's Site Reliability Engineering book and PagerDuty's incident response documentation. It's part of the same operational picture as our guide to DevOps as a service, and pairs closely with monitoring vs observability for small teams — this is what happens after an alert fires.

Severity levels: not every incident is a P1#

A written scale stops two opposite failure modes: treating a typo as an emergency, and sitting on a real outage because nobody wants to be the one who "overreacts." PagerDuty's own documentation uses a five-level SEV scale; many teams use an equivalent four-level P1–P4 scale. The labels differ; the logic — match response effort to actual impact — doesn't.

Google's SRE book suggests declaring an incident (rather than just quietly fixing it) whenever you need a second team's help, the outage is customer-visible, or the problem is still unsolved after an hour of focused effort — useful tie-breakers for anything sitting near a P1/P2 boundary. If you're setting these thresholds against an actual SLA rather than a rule of thumb, see 99.9% vs 99.99% uptime for how much downtime each tier really allows before a P1 becomes a contractual problem as well as a technical one.

Roles: who does what#

Both Google's and PagerDuty's incident processes converge on the same small set of roles, which scale down fine for a small team — one person can hold more than one role, as long as everyone knows who holds what:

• Incident commander — holds the overall picture, makes the calls, and coordinates rather than personally fixing the problem.
• Operations/technical lead — the person (or people) actually making changes to systems during the incident.
• Communications lead / scribe — owns the live timeline and sends updates, internal and external, so the people fixing the problem aren't also the ones answering "any update?" every ten minutes.

On a two- or three-person team, the incident commander and communications role can be the same person for a P3, but keep them separate for a P1 — the person driving the fix should not also be the one deciding whether to tell customers.

The first 15 minutes#

1. Acknowledge the alert so the on-call system knows someone has it.
2. Assess impact in a sentence: who is affected, and how badly.
3. Assign severity using the table above, even provisionally — you can downgrade later.
4. Name an incident commander out loud (or in the incident channel), especially if more than one person has joined.
5. Start a live timeline — a shared document or channel, timestamped, starting now. This becomes the backbone of the post-mortem later.
6. Send the first communication (see template below) even if the only news is "we're looking into it."
7. Only then start deep technical investigation — with the commander tracking state, not just the first responder's head.

Communication templates#

A short template beats a blank page while people are stressed and the clock is running.

Initial update (aim for within 15 minutes of detection):

• Status: Investigating
• What we know: [the symptom in plain language, e.g. "some users are seeing errors when checking out"]
• Impact: [who/what is affected, and roughly how many]
• Next update by: [specific time, not "soon"]

Follow-up updates (on the cadence you promised):

• Status: Investigating / Identified / Monitoring / Resolved
• What's changed since the last update
• Next update by: [specific time]

Resolution notice:

• What happened, in plain language
• When it started and when it was resolved
• What you're doing to reduce the chance of a repeat (a one-line preview of the post-mortem's action items)

Incident response runbook template#

A runbook is only useful if someone under pressure can follow it without thinking hard. Write it as steps, and link it directly from the alert that fires:

• Alert name and what it means in plain language
• Likely causes, ordered by probability, not by how interesting they are
• Checks to run first — exact commands, not "check the logs"
• Known fixes for each likely cause, with the exact command or action
• Escalation path — who to call and how, if the checks above don't resolve it
• Blast radius — what this service affects downstream, so a responder knows what else to watch

Keep runbooks next to the code or infrastructure they describe, and treat a runbook that turned out to be wrong during an incident as a bug to fix immediately afterwards, not a one-off.

On-call hygiene for small teams#

Google's SRE book recommends at least eight engineers for a sustainable single-site 24/7 rotation, with on-call capped at around a quarter of anyone's time and a target of zero-to-few incidents per shift. Most small teams can't staff that, and pretending otherwise leads to burnout. Be explicit instead:

• Agree realistic hours of coverage (business hours plus best-effort out of hours, for example) and say so in your own SLA rather than implying 24/7 coverage you can't actually deliver.
• Keep rotations short and predictable — even two or three people can rotate weekly if the expectations are clear.
• Write things down so on-call doesn't depend on one person's memory — a documented runbook is what lets a rotation of two survive one person being unreachable.
• Track how often anyone actually gets paged — a component paging daily is a signal to fix the underlying problem, not evidence that on-call is working.

Blameless post-mortem template#

Google's SRE book frames the goal plainly: understand the contributing causes well enough to prevent a repeat, without indicting an individual or team. Their example postmortem uses this structure, which works well as a starting template:

• Summary — a few sentences, what happened and the outcome
• Impact — who was affected, for how long, and how
• Root causes — the contributing factors, plural; incidents rarely have exactly one cause
• Trigger — what specifically set it off
• Detection — how you found out (an alert, a customer report — the gap between those two is itself worth discussing)
• Resolution — what actually fixed it
• Timeline — timestamped, pulled from the live document you started in the first 15 minutes
• Action items — specific, owned, dated
• Lessons learned — what went well, what didn't, and what was down to luck

Decide your triggers for writing one in advance — for example, any customer-visible downtime, any data loss, or any incident needing manual intervention — so the decision isn't made in the moment by whoever would rather skip it.

Follow-up actions: the part most teams skip#

A post-mortem without completed action items is just a well-formatted description of a problem you're going to have again. Give every action item an owner and a date, track them the same way you track other work (not in a document nobody reopens), and review at the next post-mortem whether the last one's actions actually landed. If the same root cause shows up twice, that's a stronger signal than any dashboard that the follow-through step is the one being skipped.

What to do next#

If you don't yet have a written severity scale or a runbook for your most common alert, start there — both take under an hour and immediately make the next incident less chaotic than the last one. For monitoring that feeds a good incident response process, see monitoring vs observability for small teams, or see how we run monitoring and incident response as an ongoing service.