Quoted from Richard Bookstaber, in A Demon of Our Own Design
“Systems with high levels of interactive complexity are subject to failures that seem to come out of nowhere or that appear unfathomably improbably. With many moving parts and points of interaction, and with interactions that can bound out of control in any direction, the unavoidable missteps that in a normal system would be addressed in the course of each day’s work may take a freakish leap into disaster. The start of the crisis can be something as trivial as a tag covering a warning light in the case of Three Mile Island [pdf] or a failure to properly label an oxygen generator in the case of ValuJet [592] [pdf]. And these two cases illustrate the shortcomings in attempting to deal with interactive complexity by adding backups and safety features; in the cases of both Three Mile Island and ValuJet Flight 592, the safety features themselves contributed to the disaster.”
