Postmortem
The quiet part of an incident review
Everyone agrees on the timeline. Nobody agrees on what it meant.
OperationsMarch 20266 min
Reconstructing what happened takes an afternoon. Reconstructing what people thought was happening takes a week, and it is the only part that changes anything.
Timelines are cheap
Logs, deploy records and chat history give you a minute-by-minute account almost for free. That is why every review has one, and why every review that stops there produces the same three action items: add an alert, add a runbook, add a dashboard.
Belief is the missing column
At 02:14 the on-call engineer saw error rates climb and restarted the ingest workers. The timeline records a restart. What it does not record is that she had seen the same shape three weeks earlier, when a restart had in fact fixed it.
The question is never why someone did the wrong thing. It is what made it the right thing at the time.
What the numbers hid
- 11m
- To first action
- 94m
- To correct diagnosis
- 6
- Restarts before escalation
Eleven minutes to act is good. Ninety-four minutes to understand is not, and no alert would have shortened it, because the alert fired correctly at minute nine.
The interview that works
-
Ask what they expected to see
Before asking what they did. Expectation explains action; action alone does not.
-
Ask when they stopped believing it
The moment a mental model breaks is the moment the incident actually turns.
-
Ask who they would have called
And why they did not. This finds the escalation friction that no dashboard shows.
What changed afterwards
Not an alert. The team added one line to the restart runbook: if a restart has been tried twice, the hypothesis is wrong. Mean time to correct diagnosis fell by half over the next quarter.
What to take away
- A timeline is necessary and almost never sufficient.
- Record what people believed, not only what they did.
- Repeated identical actions are the signal that a model has failed.
- Fixes aimed at belief beat fixes aimed at detection.