AT&T 1990: how a misplaced break statement took down a network

Here is an uncomfortable observation from two decades on quality floors: the code that hurts you worst is the code that almost never runs. AT&T proved it on a national scale on 15 January 1990. Every switch was healthy. The trigger was trivial – a minor fault at one machine in New York. The catastrophe was manufactured afterwards, by the recovery routine: the mechanism built to absorb faults became the mechanism that spread them.

The situation

AT&T in 1990 was American long-distance, full stop. The bulk of the country's calls ran through roughly 114 4ESS electronic switching centres – distributed, redundant, self-monitoring. One assumption sat underneath all of it: switches fail independently, so the network routes around any single one. A switch in trouble detected the fault, took itself down, recovered, rejoined. Self-healing by design.

Through late 1989, AT&T had been rolling out new switching software across the fleet, meant to speed up that recovery. The rollout was gradual. It was also total: every switch ended up on the same new code. Hold that detail; it is the whole case.

How it unfolded

At about 14:25 Eastern on Martin Luther King Day, a New York switch hit a minor fault and did exactly what it was built to do: drop into recovery and restart. During the restart, the misplaced break threw execution out of the recovery routine one step early – before a value used to build the outgoing status message was set. The switch told its neighbours nonsense.

Each neighbour that took the malformed congestion message treated it as a genuine fault of its own, crashed into recovery, restarted, and emitted the same poison message onward. Every cure became a new infection. Within seconds, reset waves were propagating and most of the fleet was cycling through crash-and-recover. Airlines could not confirm flights. Bank lines stalled. Emergency services lost connectivity. Tens of millions of calls went nowhere.

To their credit, AT&T's engineers replayed the sequence on a lab replica, isolated the offending statement and brought the network back over roughly nine hours. The hardware performed flawlessly throughout. That is the most damning fact in the public record: the machines were fine, the code guarding them was not.

Root-cause anatomy

Technically, the chain is brutally short. One stray break, one unset variable, one malformed inter-switch message, neighbours that acted on corrupt input instead of quarantining it. No validation, no backoff, no throttle on reset propagation – a positive-feedback loop with a nationwide blast radius.

Organisationally, the root cause was not the line of C. It was the monoculture. Redundancy had been validated as hardware redundancy – multiple switches, multiple paths – with each failure assumed independent of the rest. Nobody asked whether the redundant units shared one brain. They did. Every backup was a copy, so one defect owned all of them.

A safety net cut from the same rope as the tightrope tears in exactly the same place.

Where the quality system failed

Set against the disciplines I have run for twenty years, this file fails four specific gates. The PFMEA covered call processing – the revenue path – in depth, while the recovery path carried no failure-mode analysis at all. Occurrence scores reflected normal operation; exception-handling code was assumed inert precisely because it rarely executes. Change control fared no better. The new software altered fault-handling behaviour, and a gate that reads "only error code" as low risk, then regresses on the happy path alone, waves that through every time.

Design reviews proved the network survived the loss of a switch. Not one asked the question that mattered: what single defect sits behind every "independent" channel? That is common-mode analysis, and it was absent – as was any external feedback loop. Mandatory outage reporting to the FCC did not yet exist; this case is why it exists. Systemic learning leaned entirely on the carrier's own rigour, and rigour has bad days.

What would have caught it

Nothing exotic. Any one of four moves ends the story before it starts. Fault injection in the lab: crash a switch mid-traffic and watch its neighbours – the cascade reproduces in minutes, and it did once engineers thought to look. Negative testing of the protocol: corrupt one field of an inter-switch message and see what the receiver does. A node that crashes on malformed input is a finding, not a footnote. Staggered deployment: roll the patch to a handful of switches first, so the first faulty recovery takes five neighbours dark instead of a nation, with a rehearsed rollback behind it.

And build the handling fail-safe in the first place. Validate and quarantine malformed messages. Rate-limit resets. Demand real diversity of implementation across nodes, so redundancy means something the day you need it.

My take

Here I leave the case file: I have never lost a phone network, but I have met this failure wearing cheaper clothes. In automotive plants I have watched "independent" stations running identical programs, one fault synchronising them like metronomes. I have reviewed PFMEAs with dozens of failure modes on the process and not one on the alarm logic that fires when the process stops. Building a QA department from scratch for a 900-person greenfield plant fixed that habit in me permanently: detection controls are designed, never assumed, and the rarest path needs the loudest test.

The security angle is not decoration. In my ethical-hacking work, "a node that crashes its neighbours on receipt of a malformed packet" is not a quality escape – it is a denial-of-service primitive. AT&T built one by accident and pointed it at itself. It is why I keep saying quality management and security are the same discipline: the study of how complex systems fail, and the craft of building them so they don't.

Nine hours. One line of C. Tens of millions of silent phones. Redundancy is not diversity – a safety system inherits every defect of the thing it protects unless you deliberately break that inheritance. The recovery path is not a footnote to your process. On the worst day of your system's life, it is the process.