Air Astana Out Of Control (KC1833 in 2018)
On the 11th of November 2018, an Air Astana aircraft departed from Alverca airbase in Portugal as Air Astana flight KC1833, after undergoing heavy maintenance.
The aircraft was an Embraer E190-100LR registered in Aruba as P4-KCJ. This was a positioning flight from Alverca (LPAR), where the maintenance had been done, to Minsk (UUMS).
There were three crew on board, the captain and two co-pilots. The captain was the Pilot Flying. The other two pilots would cover the position of Pilot Monitoring at different times during the flight, acting under the command of the captain.
In addition, three Air Astana technicians were on board as passengers.
The weather was rainy, with a layer of cloud at 500 feet and a second at 1,500 feet. The horizontal visibility was about 2,000 metres (6,500 feet).
After the jet took off, the captain attempted to engage the autopilot, which failed. It soon became clear that something was very very wrong as the aircraft began oscillating side-to-side. The aircraft was not responding correctly and appeared to have a mind of its own. The flight crew immediately declared an emergency to Alverca tower, explaining that the aircraft was exhibiting “abnormal roll attitudes”. There was no clear cause for this. The only alert on the cockpit was the AP FAIL from the autopilot.
The flight crew attempted to counter the movements of the aircraft on all three axes as they climbed through cloud, losing all visual references.
The Pilot Monitoring struggled with nausea as the aircraft bucked and pitched violently. They attempted to climb higher, however the aircraft repeatedly plunged towards the ground at high speed. They knew that their attempts to recover were causing high structural loads on the aircraft, but they had no choice.
Still, there were no malfunction indications to explain what had gone wrong, only continuous alerts for abnormal flight attitudes.
Flight KC1833: We cannot control the airplane. Do you have an area where the weather is better? We need visual conditions to land.
ATC: I believe south, the weather is better.
ATC: When able, report people on board and fuel.
Flight KC1833: Six people on board. Airplane is completely uncontrollable. Flight control problems. We have manual control now. We want to land on the sea.
ATC: The sea and not the river?
Flight KC1833: Maybe the sea is better, more space to decelerate. Need better weather.
The flight crew repeatedly requested headings from air traffic control to take them out to sea; they could not control the aircraft for a landing but they could at least avoid causing harm to people on the ground.
However, they couldn’t follow the headings and ended up flying northeast rather than west towards the coast. Worse, rather than finding clearer weather, they flew directly into a thunderstorm.
The three flight crew discussed the symptoms, also calling upon the expertise of the technicians on board, desperate to work out what was going on.
They concluded that longitudinal control was worse when they attempted to retract the flaps, so they left the FLAPS configured to setting 1. Looking out of the windows of the cabin, they confirmed that the position of the ailerons did not match the command given by the captain in the cockpit, although it was clearly not a simple reversal. They attempted to switch between direct and normal mode but, although there alerts for failures showed in normal mode, they agreed to maintain direct control of the elevators, rudders and spoilers, removing the flight control module from the equation.
They finally managed to turn westwards.
F-16: Kilo Zulu Romeo, this is Portuguese Air Defence. How do you read?
Flight KC1833: Hi, I read you fine now.
F-16: Copied. We are 60 miles north of your position. We will try to intercept you and give you any assistance required. How can we assist you?
Flight KC1833: Flight control problem, maintaining flight level 140 westbound, we will make ditching [unintelligible].
As they continued to test the different configurations, they gained some control of the aircraft. This brought them into better weather, making it easier to see what the aircraft was doing and how the flight controls were affecting it.
ATC: You are flying away from the sea, you are flying into Spanish territory.
Flight KC1833: Our aircraft is [unintelligible]controllable.
ATC: Roger, confirm intentions.
Flight KC1833: We need an airport with good weather [unintelligible]
The air traffic controller quickly came up with a plan for an emergency landing at an airport in clear visual conditions with an emergency team standing by. Beja Airport was initially a German Air Force training facility with two large parallel runways. It became a dual-use military-civilian airport in 2011 but has not seen commercial success. The airfield is quiet and, on that day, the weather was better. She decided that this was their best option for an uncontrolled landing.
ATC: You have an aerodrome with good weather south of your position, 135 miles south of you, can you fly that?
Flight KC1833: Copied, can you say the heading?
ATC: right heading 1-9-0
F-16: Kilo Zulu Romeo, are you now able to maintain heading and altitude?
Flight KC1833: Affirm.
F-16: If you wish, we can intercept and guide you to the aerodrome for you to land in a safe way.
The crew agreed as they turned south to attempt the landing. Two F-16 fighters from the Portuguese Air Force caught up to the flight to help guide them to the airport.
F-16: Maintain 20,000 [Unintelligible] on your left wing.
Flight KC1833: OK. Could you please give us some space because we have TCAS.
The Traffic Alert and Collision Avoidance System must have been sounding non-stop with the F-16s flying alongside them.
The E190 followed the F-16s for a direct approach to runway 19R, the right-hand of the parallel runways at Beja airport. After significant control difficulties, the crew broke off the approach to try again.
The pilot in the right seat, who had been Pilot Monitoring despite his air sickness, was becoming fatigued. They agreed that the pilot in the jump-seat would take on the role of Pilot Monitoring to support the second approach.
Again, the captain struggled to maintain control and again, he had to break off the approach. Now he, too, was becoming fatigued. He brought the aircraft around for a third attempt.
This time, he managed to keep the aircraft steady enough to continue but in the final moments, the E190 drifted to the left. Seizing the moment, he continued his approach, landing on runway 19L, the left-hand parallel runway.
Here is a video of the successful approach, which was recorded by one of the F-16s:
They were on the ground. Tyre Number 4 struck a runway edge lamp but they managed to stay on the runway and come to a halt.
As the pilots slowed the aircraft down, the Engine Indication and Crew Alerting System (EICAS) lit up with an error message: FLT CTR NO DISPATCH
All of the aircraft occupants were described as “physically and emotionally shaken.” Only one passenger had been injured, when he stood up to try to communicate with the pilots in the cockpit and was knocked off balance by the aircraft’s movements. He was treated for a sprained ankle and released the same day. A second passenger was treated for stress-related concerns.
The aircraft’s skin was deformed from the stress of the uncommanded manoeuvres and the leading edges of the wings were wrinkled.
But what had happened?
The Embraer E190 had flown to the Portuguese military airport Alverca do Ribattejo on the 2nd of October, five weeks before that terrible flat. It was scheduled to undergo a “heavy maintenance work package” at there as a part of its maintenance programme. This package included a basic inspection (C2-check), corrections of any issues found during the inspection, and any service bulletin modifications which would need the modular avionics unit to be removed and installed.
While the aircraft was parked there, on the 9th of October, a service bulletin came into force which changed how the aileron control cables were routed. In order to reduce cable friction, the pulleys and their structural supports were to be replaced with non-contact support.
On the 11th of October, the maintenance team disconnected the aileron control cables completely.
One of the the modifications in the Embraer Service Bulletins changed a cable routing support near rib 21 of the aircraft with complex instructions and an inaccurate depiction of the aileron cable routing. These instructions were hard to understand and made it hard to understand how the cables were to be atttached. It also never mentioned that it was critical that the cables never crossed.
At some point during this work, the aileron control cables were inverted, that is, the up cable terminated in the down position and the down cable terminated in the up position.
Admiral Cloudberg offers a clear explanation of what went wrong in his write-up of the event: Flying the Unflyable Plane: The near crash of Air Astana flight 1388
It was here that the critical error occurred: during reinstallation on October 11th, the technicians inadvertently crossed the cables in both wings, connecting the “up” cables to the “down” fittings on the hydraulic power control units, and vice versa. There were no markings on the cables which would have distinguished them, nor was there any design difference that would have prevented them from being fitted the wrong way around. According to Embraer, the main assurance against swapping the cables was the fact that no procedure ever called for them to be removed at the same time, ensuring that there was only ever one way to put a removed cable back in. But Embraer’s own service bulletin SB-190–57–0038R2 could not be accomplished without removing both cables simultaneously, so this basic design principle was violated, with the predictable result that the OGMA technicians installed the cables the wrong way around.
To be clear: it should not be possible to invert the aileron cables. The E190s certification rested on the idea that the engineers could not attach them wrongly as only one cable would be changed at a time, so there was only one space in which it could be installed. Once this protection was removed, there was no easy way to see where the cables were meant to attach.
Complying with a further service bulletin, the engineers replaced the installed stainless-steel control cables with carbon-steel cables. They lubricated and adjusted the aileron control cable system, ensuring that the new cables were fitted exactly as the old ones had been. There was no way for them to see that the cables they were replacing had been crossed, and so, new cables were crossed as well.
There were some inconsistencies in the records, including wrong dates on the sign-off cards and inspection procedures which were not performed.
This was partially caused by the pressure to deliver quickly. The maintenance was due to be completed before the 24th of October but the maintenance organisation were unable to make the date, so they asked for another week, promising Air Astana that the aircraft would be ready for the 31st of October. To ask for another extension would be a catastrophe.
On the 26th of October, the maintenance team powered up the aircraft for operational tests.
The Engine Indication and Crew Alerting System (EICAS) lit up with an error message: FLT CTR NO DISPATCH.
This message shows the location of the issue (Flight Control) and the severity (aircraft should not be dispatched for flight).
The error meant that the engineers were unable to complete further operational tests, including the aileron control system operational test. Everything was focused on resolving this error.
Maintenance continued. The aircraft underwent further testing. This included the flight control tests, but the E190 passed them, as the ailerons and spoilerons were only tested for full and free movement, not the direction in which they were moving. It seems like it should be obvious that the ailerons were moving the wrong way but not a single engineer noticed that the control cables for the ailerons were inverted. It’s important to note that the spoilerons went in the correct direction, which is why the effect was not a simple reversal. Instead, the correlation of the flight controls to the aircraft angles appeared to be random but of course, that was only obvious after they were in the air.
On the 31st of October, the aircraft was delivered to the area manager, who allocated a small team to investigate the FLT CTR NO DISPATCH fault. By now, there was significant pressure to resolve the problem and deliver the aircraft as quickly as possible.
They attempted to clear every possible fault that could cause the FLT CTR NO DISPATCH alert. The list of possible faults did not include the aileron cables: it was a further test that triggered a further test that resulted in the fault making its way back to the Flight Control Modules. That is to say, the wrongly-installed aileron cables were at the heart of the matter, but this was not obvious to anyone as the results were two steps removed.
Air Astana, wanting their aircraft back in operation, brought in their own experts. Two Air Astana staff came to the site in hopes of resolving the fault: the director of technical asset management and a material supply technician, joining an Air Astana maintenance technician who had been shadowing the C-Check. Embraer also sent an engineer to help troubleshoot. Everyone involved concluded that they had a software problem: one of the four Flight Control Modules must be faulty. They began replacing the modules one by one. When they replaced the final module, FCM 4, the FLT CTR error message finally went away.
This meant that they could finally hand-over the aircraft as the fault had been cleared. They did not rerun the tests which had caused the fault to appear in the first place, as the aircraft was now almost three weeks overdue and clearly showing as ready for dispatch.
On the 11th of November, the aircraft was prepared and positioned for the Air Astana crew to accept delivery. During the final operational checks, the Air Astana crew detected more issues: there was a problem with the ventilation in an avionics department and one of the hydraulic systems was low on fluid.
The maintenance organisation went back to work and resolved the issues. Once again, the E190 was prepared and positioned for handover to Air Astana.
The Air Astana crew found no further faults and they finally took possession of the overdue E190. On the flight deck were the captain and two co-pilots. The three Air Astana technical staff were in the cabin as the only passengers.
The flight crew did not have any guidance on accepting an aircraft after heavy maintenance. Following the standard pre-flight checks, they confirmed that the ailerons moved freely in all directions but again, they didn’t spot that the ailerons were misconfigured.
All were happy that the aircraft was in good shape, never guessing the nightmare that would start once the aircraft left the ground.
FIGURE 2 SHOWING ABNORMAL AIRCRAFT ATTITUDE AS BASED ON THE FDR
Once they had landed safely at Beja, it did not take the investigators long to confirm that both aileron surfaces were acting in the opposite direction of the control yoke command.
From the report
The incorrect installation of the aircraft’s flight control cables system during heavy maintenance activities, resulted in the aircraft’s aileron system reversal operation and consequent loss of control during flight, was identified as the cause of the accident.
Contributing to the event, the existing gaps in the maintenance service provider quality assurance system and failure to implement an effective internal and external maintenance activities oversight, enhanced by the aircraft design and the corresponding maintenance instructions regarding the ailerons control system configuration.
The rush to get the aircraft back in service resulted in gaps in quality control at the maintenance organisation. The fact that everyone involved, including Air Astana technical management and an Embraer engineer, believed that they had found the fault and fixed it did the rest.
The GPIAA made four safety recommendations targeted at the maintenance service provider and the aircraft manufacturer. Embraer did not change the design of the cables but rewrote the maintenance manuals and revised the service bulletin so that only one cable would be detached at a time. Air Astana wrote their own in-depth report on the flight and instituted training for maintenance acceptance flights. In addition, the pre-flight control check now includes a call-out of the direction of the movement. The maintenance organisation
The aircraft was written off and the three pilots were offered a leave of absence at their discretion. All three reported back to duty after only a few weeks.