FAA outage triggered by “damaged database file” -- after agency warned on old hardware
A serious IT outage at the Federal Aviation Authority (FAA) which forced it to halt all US departing flights on Wednesday 11 has been attributed by the transportation agency to a “damaged database file”
“At this time, there is no evidence of a cyber attack. The FAA is working diligently to further pinpoint the causes of this issue and take all needed steps to prevent this kind of disruption from happening again” the FAA said, as it continued to review the root cause of the Notice to Air Missions (NOTAM) system outage.
The NOTAM system flags anomalies in the National Airspace System (NAS) including short-notice warnings on “the establishment, condition, or change of any facility, service, procedure or hazard in the NAS.”
“The preliminary indications are that two people working for a contractor introduced errors into the core data used on the system… according to a person familiar with the FAA review” according to Bloomberg.
BT’s ‘reverse flywheel’ pain as CIO targets £600m legacy IT spend
The FAA itself admitted in a February 2022 report that "Many components within the FNS/USNS [Federal NOTAM System] are running on old hardware and improvements in the system architecture are needed."
CNN reports that the FAA IT outage saw an IT team force a hard reboot of the primary NOTAM database -- with plans to swap to a backup system finding that the error had also propagated to the backups.
IFR, a magazine for pilots, in 2021 described the US NOTAM system as featuring "cryptic abbreviations, abstract keywords and shorthand, and irrelevant, illogically ordered lines" and "the ancient Aeronautical Fixed Telecommunications Network [as] limited to upper case and free text, neither of which works once we seek to share data digitally across systems. All told, it’s a dusty back office full of legacy hardware and software."
The findings confirm the early analysis Mandiant’s John Hultquist, who had suggested on January 11 as unfounded speculation about the incident mounted that “I really doubt you'll find some sinister cyber plot at the root of this FAA thing, but if you're looking for cybersecurity angles I think it's this: we live in an increasingly complex, interdependent system that is prone to unforeseen consequences and cascading failures.”
FAA outage cause hints at complex risks
In October 2022 the FAA warned that legacy technology was a growing risk across the civil aviation space.
In its strategic plan for 2022-2026 it noted that across the aviation industry “legacy systems continue to be maintained at great expense despite, in some cases, a significant reduction in their operational use.”
The federal authority added that “building a business case to ensure the right systems and services are being used in the right places requires a coordinated and collaborative effort, involving data-sharing and external stakeholder engagement, as well as investment by stakeholders in newer technology” – further emphasising that “the relatively static NAS service levels provided today are unable to respond to both the rapid evolution and business dynamics of traditional aviation operations”, even as an “increased breadth of data has been a catalyst moving the aviation industry as a whole towards more predictive and prescriptive analytics.”
One of its biggest priorities in coming years is to “identify and develop strategies to address data gaps to support safety, equity, and other priorities; develop a Data Governance Structure [and] develop near real-time data reporting and single operational metric database” – ideally without changes triggering outages.
