Operations·4 min read
Four pages that should be read before every post-incident review
Cook was writing about medicine. Anyone who has run a production system will recognise all eighteen points immediately.
Source
How Complex Systems Fail
Richard I. Cook, MD · Cognitive Technologies Laboratory, University of Chicago · 1998
Read the originalAslanWay is not the author of this work. The summary below describes the source; the commentary that follows is ours and is not endorsed by the author.
What it argues
Richard Cook, a physician working on patient safety, set out eighteen short observations about how complex systems fail. The document is four pages and has become required reading in software operations for reasons that become obvious on first read.
The observations build on each other. Complex systems are intrinsically hazardous and are therefore heavily defended. Because those defences work, catastrophe requires several failures at once — single-point failures are not sufficient. Every complex system is running with multiple latent flaws at any moment, and does so successfully.
Why root cause analysis misleads
Cook argues there is no isolable single cause of a failure, because it takes multiple contributors to get past the defences. Naming one root cause is a decision about where to stop looking, and it is usually a socially convenient place to stop.
He also makes the point that hindsight distorts post-incident review. Once you know the outcome, the signals that pointed to it look obvious, and the operators look negligent for missing them among all the other signals that did not.
What we do with it
- Change the question in the review from "what went wrong" to "what made this seem reasonable at the time". You get more useful answers and fewer defensive ones.
- Cook's point that operators actively create safety, continuously, is the argument against treating on-call as an overhead. The people absorbing the variance are the reason it usually holds.
- This is why we rehearse failure before peak rather than after. Defences that have never been exercised are assumptions, not defences.