On the 11th of January 2023, the Federal Aviation Administration (FAA) in the US stopped all departing commercial flights, grounding aircraft all over the US. Hot on the heels of what’s now being referred to as the “Southwest meltdown,” outdated IT systems are once again centre stage. Wednesday’s outage caused the most extensive ground stop in the US since 9/11.

The system actually failed on Tuesday night, however initially flights were allowed to continue. A NOTAM OUTAGE HOTLINE was quickly put into place so that new data could be distributed verbally for the remaining traffic, which was not expected to be high. When the outage still wasn’t resolved by daybreak the following morning, it became clear that they could not continue to attempt to deal with the situation manually.

The first notice came as an urgent advisory from the FAA Air Traffic Control System Command Center released on Wednesday at 4am.

THE UNITED STATES NOTAM SYSTEM FAILED AT 2028Z. SINCE THEN NO NEW NOTAMS OR AMENDMENTS HAVE BEEN PROCESSED. TECHNICIANS ARE CURRENTLY WORKING TO RESTORE THE SYSTEM AND THERE IS NO ESTIMATE FOR RESTORATION OF SERVICE AT THIS TIME.

The FAA released a public statement at 12:15 UTC (7:15 am EST) that the Notice to Air Missions system had suffered an outage and airlines had been ordered to pause all domestic departures until 14:00 UTC (9:00 am EST).

Flights already in the air continued. They should already have the NOTAMs relevant to their flight and the truth is, if they didn’t, it wasn’t going to help to divert them elsewhere. In addition, most (but not all) transatlantic flights to the US continued.

At 08:15 EST, the FAA updated their site with the news that departures at Newark Liberty and Atlanta-Jackson airports had been resumed to alleviate congestion at those airports and reiterated that departures would resume at other airports at 09:00. In fact, the ground stop was lifted at 08:50.

That evening, the FAA released a further statement that they were continuing to review the situation as the root cause of the outage was not yet known. “Our preliminary work has traced the outage to a damaged database file. At this time, there is no evidence of a cyber attack.”

Update 6: We are continuing a thorough review to determine the root cause of the Notice to Air Missions (NOTAM) system outage. Our preliminary work has traced the outage to a damaged database file. At this time, there is no evidence of a cyber attack. (1/2)

— The FAA ✈️ (@FAANews) January 11, 2023

Over 1,300 flights were cancelled while the system was down, with over 10,000 flights delayed., according to NBC News.

This situation is now resolved in terms of flights, and an investigation is in progress to discover how it happened in the first place. However, the situation has left many people wondering, what the hell is a NOTAM?

The NOTAM system dates from 1947 as a means of sharing important aviation information to pilots. It was based on Notice to Mariners which were used to notify ship captains of hazards at sea, which was developed by the US Navy in 1869. NOTAM stands for Notice to Air Missions in the United States and Notice to Air Men in the rest of the world. I’m generally in favour of gender-inclusive language but as pilot licences are still referred to as Airmen Certification in the US, this change seems questionable. At this stage, the aviation industry uses NOTAM as a set term, without much thought as to it being an acronym.

The NOTAM system is run by aviation authority in any given country, in this case the FAA. The point of the NOTAM is to alert airmen pilots to potential hazards which are either temporary or were not known far enough in advance to be included in charts and other published systems. When you plan a flight, whether you are flying an Airbus A380 or a Cessna 172, you check the NOTAMs on your route so that you know of any issues that may affect your flight.

NOTAMs are critical for announcing hazards, such as air shows, parachute jumps or rocket launches. They alert pilots to closed runways and faulty radio navigational aids. NOTAMs also will list the location of obstructions near airfields, such as construction sites with high cranes or even a problematic flock of migratory birds (also known as a BIRDTAM).

I’ve previously posted a collection of weird and interesting NOTAMs, including this example that ccbcc814 posted to Reddit, which illustrates what can happen if someone doesn’t do their homework.

… one of the local Airports became a Special Forces Training site (I guess simulating Airfield seizure or something like that) so its was NOTAMed that the field was closed from 2300 Local to 0500 Local for Military Exercises.

Most of the time this wouldn’t have been a issue as the particular field was rarely used for night training (IIRC it only had Runway lighting, but no PAPI/VASI or Taxiway lighting) and out in the middle of BFN. Except for a few pilots who decided that night to camp out without reading the NOTAM.

So cut to them getting woken up at 11pm to fucking Ospreys and shit hitting the field, and guys with guns running around outside. It took a few moments for the Military to realize something was up, (I imagine the conversation went something along the lines of “Dafuq is that 182 doing out here?”) and they just told the guys to stay inside the Pilots lounge and they would be finished shortly.

In the comments, Rudy told us how he learned to always ask for NOTAMS when he was towing banners in Germany. Three pilots were flying Piper Super Cubs in formation, with Rudy at the fore. As they passed over a normally dormant grass strip, they were surprised to see it full of people and parked cars. None of the pilots had seen the NOTAM closing the airspace for an airshow, with the result that three banners were towed directly through the airshow at 1,000 feet.

The manager and ATC controller at Meschede, Ewald Schuessler, (it is now so long ago, I am sure that I can now mention him by name) had been in a habit of dropping NOTAMs that he received in a drawer of his desk without ever bothering to post them on a notice board. Pilots had to ask for them. And from that moment on, I always did !

However, there is the problem that NOTAMs are often issued for trivial concerns in what seems to be an ass-covering exercise. In addition, although people are quick to issue NOTAMs, they often neglect to cancel them. A NOTAM should not last longer than 90 days but they do not automatically expire. The result is that there can be hundreds of irrelevant or out of date NOTAMs that need to be reviewed for a single long-haul flight, which can make it difficult to take in the truly important alerts. In 2018, the NTSB chairman described NOTAMS as “a bunch of garbage that nobody pays attention to.”

This Reuters article from last year shows how bad the situation currently is.

‘Bunch of garbage’: Campaign to ease pilot overload from antiquated safety warnings

In Albania, for example, there is an active NOTAM issued in 2000 that provides pilots with a telephone number to call should they have a Y2K-related communications problem.

“So I’m basically reading about Y2K thinking they can’t possibly mean that thing from 20 years ago that never happened,” Zee [the founder of a flight operations advisory firm] said.

“And right at the bottom there’s one that says, for example, no jet fuel available for two weeks. I’ve missed it. Now I’ve flown in and I can’t get fuel to get back out again.”

The current situation has led to an ICAO initiative to overhaul NOTAMs format, with a goal of reducing the number of NOTAMs and making them easier to read. A modern NOTAM includes the following components:

• Accountability, which includes a geographic tag used for logging

• NOTAM number in the format MM/NNN where MM is the two-digit month reference

• Affected location

• Details of the restrictions, which are described using a set of contractions of keywords which is part of what is currently being revised, as these abbreviations were not consistent

• Start of activity and end of validity

For example, on the 24th of February 2022, Ukraine issued a NOTAM closing its airspace to civil traffic in response to the Russian invasion.

A0576/22 NOTAMN

Q) UKXX/QAFXX/IV/NBO/E /000/999/4748N03112E999

A) UKBV UKDV UKFV UKLV UKOV B) 2202240156 C) 2202242359

E) FLIGHTS OF CIVIL ACFT WITHIN FIRS KYIV (UKBV) DNIPRO (UKDV) LVIV  (UKLV) ODESA (UKOV)  SIMFEROPOL FIR (UKFV) AND THE KYIV UIR (UKBU) ARE RESTRICTED DUE TO POTENTIAL HAZARD FOR CIVIL AVIATION.

I’m not sure how to breakdown the references for Q) but starting from point A), we can see the affected flight information regions: UKBV (Kyiv) UKDV (Dnipro) UKLV (Lviv), UKOV (Odessa) and UKFV (Simferopol). Then B) and C) offer the timing in Year-Month-Date format, so the restriction starts in 2022 in February on the 24th at 01:56 and then ends at the end of that day at 23:59, which in retrospect seems a bit overly optimistic!

Finally, E) describes the restriction: all civil flights are restricted in those areas.

In this example, many other countries also issued NOTAMs to not overfly Ukrainian and Russian airspace in the affected areas, which would hold even if Ukraine and Russia cancelled their restrictions. Here is an excerpt from the NOTAM that the FAA issued to US air carriers.

THOSE PERSONS DESCRIBED IN PARAGRAPH A (APPLICABILITY) BELOW ARE PROHIBITED FROM OPERATING AT ALL ALTITUDES IN THE MOSCOW FLIGHT INFORMATION REGION (FIR) (UUWV), THE SAMARA FIR (UWWW), AND THE ROSTOV-NA DONU FIR (URRV) WITHIN 160NM OF THE BOUNDARIES WITH THE DNIPRO FIR (UKDV), THE SIMFEROPOL FIR (UKFV), AND THE KYIV FIR

(UKBV) (INCLUDES THAT PORTION OF THE KYIV UPPER INFORMATION REGION (UIR) (UKBU) AIRSPACE WITHIN THE LATERAL LIMITS OF THE UKDV, UKFV, AND UKBV FIRS) DUE TO SAFETY-OF FLIGHT RISKS ASSOCIATED WITH ONGOING HOSTILITIES.

You can see the effect of these NOTAMS by looking at the live data on flightradar24 — it takes a minute to load, but it is then easy to see the completely clear area over Ukraine.

At the same time, the website Death to NOTAMs has a long list of NOTAMS posted which are of little relevance to pilots flying in and out of the airports.

Grass Cutting. How would you amend your flght?
FBFT/FRANCISTOWN B0086/17 27MAR 0456Z (NOTAMR B0016/17) – GRASS CUTTING IN PROGRESS. PILOTS TO EXER CTN ON LDG AND TKOF. 27 MAR 14:30 2017 UNTIL 27 MAY 14:30 2017.

Boar Slaughtering (by Night)
GMMB/Casablanca
MAY 2017:06-13-20-27 -JUN 2017:03 SLAUGHTER BOARS OPERATION BY NIGHT WITHIN THE AIRPORT(GMMB)

There’s a wall. It’s small. It may collapse on May 13th.
GCLA/LA PALMA A1681/17 13MAR 1309Z – RETAINING WALL ALONG IMMEDIATE END OF RWY01 STRIP. DOES NOT EXCEED HEIGHT OF RWY. REF SUP 48/16. 13 MAR 22:00 2017 UNTIL 13 MAY 22:00 2017 ESTIMATED.

There is no Hobart. Or is there?
D0506/17 NOTAMN
Q) LTAA/QFAXX/IV/NBO/A /000/999/3845N04139E005
A) LTCK B) 1703200805 C) 1706210900 EST
E) THERE IS NO HOBART.

That said, NOTAMs are still critical to keeping aviation safe, which is why the FAA grounded domestic flights on Wednesday.

ABC news wrote that the cause of the outage was an engineer error during scheduled maintenance.

An engineer “”replaced one file with another”,” the official said, not realizing the mistake was being made. As the systems began showing problems and ultimately failed, FAA staff feverishly tried to figure out what had gone wrong. The engineer who made the error did not realize what had happened.

And Yahoo!news added the juicy little detail that the engineer was an external contractor.

Investigators are working to determine if human error or malice is to blame for taking down the system, which eight contract employees had access to. At least one, perhaps two, of those contractors made the edit that corrupted the system, two government sources said Thursday.

To be honest, none of this is all that interesting unless you are deadset on human factors as the sole cause. Pointing at a single engineer is the IT equivalent of blaming the pilot because it is easy to fix: get rid of the contractor(s) and we’ve solved the problem.

A human error (or malicious action, if that’s what it was) should not have been capable of downing the system. The files and databases should all have been tested before getting anywhere near the live system. It shouldn’t be possible to apply an upgrade which cannot be rolled back. And even if every safeguard failed, there should be some form of fallback option. Those of us in aviation and in IT know that a simple mistake by an engineer in the middle of the night should not have the power to halt US aviation for even a minute, let alone hours.

NBC News reports that the software that failed was thirty years old and was scheduled to be updated, but not for another six years. Rumour has it that there was a second system in place for redundancy, however it was out-of-date and not considered reliable.

The FAA claim that they are dedicated to modernising the system and incorporating the ICAO format. However, this is likely hampered by leadership issues. The previous FAA Administrator resigned in March 2022, two years into his five-year term, saying that he wished to spend more time with his family. An Acting Administrator was put into place for an “interim period” and continues to serve in this role, almost a year since he was first put in place as a temporary solution. The Biden administration selected a prospective new FAA administrator in June 2022, but this candidate has received some criticism, with his detractors arguing that he does not have sufficient aviation experience. In order to move forward, a candidate needs to be officially nominated again and then confirmed by the US Senate to take on the role, something that might hopefully actually happen this year.

Procedures and processes must take human factors into account. A minor mistake during routine maintenance should not cause the entire system to fall over. It should not be necessary to decode the language of aviation safety. This outage and the Southwest meltdown should serve as a wake-up call for the aviation industry as a whole to prioritise robust systems and proper management. The current NOTAM system, designed in 1947, is clearly in need of an overhaul. While such changes may take time, it’s essential that aviation authorities prioritise the upgrading of these systems at the heart of the aircraft operations.