FAA NOTAM: how one corrupted file grounded a nation and defeated its backup

Here is an uncomfortable observation from two decades of signing off maintenance windows in 24/7 plants: nobody ever asked me whether the backup contained good data. They asked whether a backup existed. Whether the failover test had passed. Who signed the protocol. On 11 January 2023 the FAA had all three answers, and the answers were worth nothing. A backup is a copying operation, and copying replicates corruption with perfect fidelity. The failover that morning worked exactly as designed – it failed over to the same poison.

The situation

The NOTAM system is unglamorous infrastructure that pilots depend on for safety-critical notices – closed runways, navigation outages, hazards along a route. It had run in some form for decades, and the architecture showed its age. Not a secret, either: the DOT Inspector General and outside investigators had flagged it for years as ageing and overdue for modernisation. It never reached the top of a funding list. It ran 24/7, it had not failed catastrophically, and in every budget review "still running" quietly matured into "acceptable risk". Then came the maintenance window of 10–11 January 2023.

How it unfolded

Scheduled maintenance introduced a corrupted database file into the primary system overnight. The damage was in the data, not the process – and a reboot does not clean a data set. By morning the system could not be trusted to deliver current safety information, and the FAA made the right call: at about 07:30 Eastern it halted all US departures, the first national ground stop since September 2001. Engineers threw the switch to the backup. Nothing changed. The corruption had already replicated, so the failover served the same unusable data through a different machine. The stop lifted within roughly two hours. The cascade ran all day – more than 10,000 flights delayed, spilling into the next morning's departure banks.

Root-cause anatomy

Technically, three failures stacked. First, a corrupted file crossed from maintenance into production, and no integrity gate stood between the work and the live data set. Second, replication did its job faithfully and copied the corruption into the backup; nothing on the receiving end questioned the content. Third, failover was validated for function, never for content – it proved you could switch, not that what you switched to was worth switching to. Textbook common-cause failure. Primary and backup shared one poisoned data set. The safety net and the trapeze failed together.

The organisational root is older. Auditors had documented the ageing architecture years earlier; the finding was known, filed and unfunded. Deferred modernisation is a risk-acceptance decision that almost never carries a signature or a date. Every budget cycle that left the system unrenewed was a quiet bet that the failure would land on someone else's watch.

Redundancy is not integrity – a faithful copy of bad data is just corruption with a witness.

Where the quality system failed

Run this through the disciplines and every gate stands open.

• PFMEA: scheduled maintenance on a live, safety-critical system was treated as routine activity, not as a failure mode. Severity: a national ground stop. Occurrence: low. Detection: none – no control sat between the maintenance action and production data. Catastrophic severity with no detection control should stop a review, not survive it.
• Change control: maintenance was exempted from the discipline applied to changes. No validation of the data payload, no proven rollback point, no go/no-go criteria signed before the window opened.
• Backup validation: the failover test proved switch-over, never content. The audit question was never asked: show me the last restore test with integrity verification, dated.
• CAPA gate: audit findings on an ageing architecture were never converted into a funded corrective action with a due date. A finding without a CAPA is filed risk, not managed risk.

What would have caught it

• An integrity gate on maintenance: any file promoted into a production data set is validated – checksums, schema checks, control totals – against written acceptance criteria, and rejected before replication.
• A replication firewall: the backup independently verifies incoming data and quarantines a mismatch instead of ingesting it.
• Restore drills instead of failover drills: quarterly restore into a sandbox with data assertions. When I ask suppliers this in audits, most show me a failover test. The two are not the same evidence.
• Maintenance under full change control, with a tested rollback point and an independent go/no-go before live promotion.
• A CAPA with budget and a due date attached to the Inspector General's findings. Modernisation with a date is a plan; without one, it is a hope.

My take

I was not in the room for this one – the FAA's systems are public record, not my case to claim. But I have signed the paperwork on the other side of this failure mode. In 24/7 automotive plants I approved maintenance windows where the Monday 05:00 restart depended on a backup, and the verification was almost always functional: the system boots, the line runs, we ship. Whether the data inside was clean was assumed, not proven. I have also stood where the FAA stood that morning, at plant scale – when my teams could not trust the information feeding a safety-critical decision, we stopped the line, took the hit and investigated afterwards. Containment first, root cause second. That is D3 discipline in an 8D, applied to a nation instead of a shift. And I have watched audit findings on ageing systems age politely alongside them. "Still running" is not a control state. It is luck with paperwork.

The FAA's backup did exactly what it was built to do. It copied – faithfully, completely, including the defect. Redundancy without data-integrity controls is theatre: it performs availability while the data rots underneath. And an ageing system that auditors have already flagged is not a legacy asset. It is a scheduled failure waiting for a calendar slot. On 11 January 2023 the slot opened, and 10,000 delayed flights were the invoice for decades of deferral.