Air Astana Out Of Control (KC1833 in 2018)

25 Nov 22 18 Comments

On the 11th of November 2018, an Air Astana aircraft departed from Alverca airbase in Portugal as Air Astana flight KC1833, after undergoing heavy maintenance.

The aircraft was an Embraer E190-100LR registered in Aruba as P4-KCJ. This was a positioning flight from Alverca (LPAR), where the maintenance had been done, to Minsk (UUMS).

Air Astana P4-KCJ

There were three crew on board, the captain and two co-pilots. The captain was the Pilot Flying. The other two pilots would cover the position of Pilot Monitoring at different times during the flight, acting under the command of the captain.

In addition, three Air Astana technicians were on board as passengers.

The weather was rainy, with a layer of cloud at 500 feet and a second at 1,500 feet. The horizontal visibility was about 2,000 metres (6,500 feet).

After the jet took off, the captain attempted to engage the autopilot, which failed. It soon became clear that something was very very wrong as the aircraft began oscillating side-to-side. The aircraft was not responding correctly and appeared to have a mind of its own. The flight crew immediately declared an emergency to Alverca tower, explaining that the aircraft was exhibiting “abnormal roll attitudes”. There was no clear cause for this. The only alert on the cockpit was the AP FAIL from the autopilot.

The flight crew attempted to counter the movements of the aircraft on all three axes as they climbed through cloud, losing all visual references.

Abnormal aircraft attitude based on FDR data

The Pilot Monitoring struggled with nausea as the aircraft bucked and pitched violently. They attempted to climb higher, however the aircraft repeatedly plunged towards the ground at high speed. They knew that their attempts to recover were causing high structural loads on the aircraft, but they had no choice.

Still, there were no malfunction indications to explain what had gone wrong, only continuous alerts for abnormal flight attitudes.

Flight KC1833: We cannot control the airplane. Do you have an area where the weather is better? We need visual conditions to land.

ATC: I believe south, the weather is better.

ATC: When able, report people on board and fuel.

Flight KC1833: Six people on board. Airplane is completely uncontrollable. Flight control problems. We have manual control now. We want to land on the sea.

ATC: The sea and not the river?

Flight KC1833: Maybe the sea is better, more space to decelerate. Need better weather.

The flight crew repeatedly requested headings from air traffic control to take them out to sea; they could not control the aircraft for a landing but they could at least avoid causing harm to people on the ground.

However, they couldn’t follow the headings and ended up flying northeast rather than west towards the coast. Worse, rather than finding clearer weather, they flew directly into a thunderstorm.

The three flight crew discussed the symptoms, also calling upon the expertise of the technicians on board, desperate to work out what was going on.

They concluded that longitudinal control was worse when they attempted to retract the flaps, so they left the FLAPS configured to setting 1. Looking out of the windows of the cabin, they confirmed that the position of the ailerons did not match the command given by the captain in the cockpit, although it was clearly not a simple reversal. They attempted to switch between direct and normal mode but, although there alerts for failures showed in normal mode, they agreed to maintain direct control of the elevators, rudders and spoilers, removing the flight control module from the equation.

They finally managed to turn westwards.

F-16: Kilo Zulu Romeo, this is Portuguese Air Defence. How do you read?

Flight KC1833: Hi, I read you fine now.

F-16: Copied. We are 60 miles north of your position. We will try to intercept you and give you any assistance required. How can we assist you?

Flight KC1833: Flight control problem, maintaining flight level 140 westbound, we will make ditching [unintelligible].

As they continued to test the different configurations, they gained some control of the aircraft. This brought them into better weather, making it easier to see what the aircraft was doing and how the flight controls were affecting it.

ATC: You are flying away from the sea, you are flying into Spanish territory.

Flight KC1833: Our aircraft is [unintelligible]controllable.

ATC: Roger, confirm intentions.

Flight KC1833: We need an airport with good weather [unintelligible]

The air traffic controller quickly came up with a plan for an emergency landing at an airport in clear visual conditions with an emergency team standing by. Beja Airport was initially a German Air Force training facility with two large parallel runways. It became a dual-use military-civilian airport in 2011 but has not seen commercial success. The airfield is quiet and, on that day, the weather was better. She decided that this was their best option for an uncontrolled landing.

ATC: You have an aerodrome with good weather south of your position, 135 miles south of you, can you fly that?

Flight KC1833: Copied, can you say the heading?

ATC: right heading 1-9-0

F-16: Kilo Zulu Romeo, are you now able to maintain heading and altitude?

Flight KC1833: Affirm.

F-16: If you wish, we can intercept and guide you to the aerodrome for you to land in a safe way.

Flightradar24 data for Air Astana KC1388 E190 loss-of-control incident

The crew agreed as they turned south to attempt the landing. Two F-16 fighters from the Portuguese Air Force caught up to the flight to help guide them to the airport.

F-16: Maintain 20,000 [Unintelligible] on your left wing.

Flight KC1833: OK. Could you please give us some space because we have TCAS.

The Traffic Alert and Collision Avoidance System must have been sounding non-stop with the F-16s flying alongside them.

The E190 followed the F-16s for a direct approach to runway 19R, the right-hand of the parallel runways at Beja airport. After significant control difficulties, the crew broke off the approach to try again.

The pilot in the right seat, who had been Pilot Monitoring despite his air sickness, was becoming fatigued. They agreed that the pilot in the jump-seat would take on the role of Pilot Monitoring to support the second approach.

Again, the captain struggled to maintain control and again, he had to break off the approach. Now he, too, was becoming fatigued. He brought the aircraft around for a third attempt.

This time, he managed to keep the aircraft steady enough to continue but in the final moments, the E190 drifted to the left. Seizing the moment, he continued his approach, landing on runway 19L, the left-hand parallel runway.

Here is a video of the successful approach, which was recorded by one of the F-16s:

They were on the ground. Tyre Number 4 struck a runway edge lamp but they managed to stay on the runway and come to a halt.

As the pilots slowed the aircraft down, the Engine Indication and Crew Alerting System (EICAS) lit up with an error message: FLT CTR NO DISPATCH

All of the aircraft occupants were described as “physically and emotionally shaken.” Only one passenger had been injured, when he stood up to try to communicate with the pilots in the cockpit and was knocked off balance by the aircraft’s movements. He was treated for a sprained ankle and released the same day. A second passenger was treated for stress-related concerns.

The aircraft’s skin was deformed from the stress of the uncommanded manoeuvres and the leading edges of the wings were wrinkled.

LPBJ approach and landing trajectory

But what had happened?

The Embraer E190 had flown to the Portuguese military airport Alverca do Ribattejo on the 2nd of October, five weeks before that terrible flat. It was scheduled to undergo a “heavy maintenance work package” at there as a part of its maintenance programme. This package included a basic inspection (C2-check), corrections of any issues found during the inspection, and any service bulletin modifications which would need the modular avionics unit to be removed and installed.

While the aircraft was parked there, on the 9th of October, a service bulletin came into force which changed how the aileron control cables were routed. In order to reduce cable friction, the pulleys and their structural supports were to be replaced with non-contact support.

On the 11th of October, the maintenance team disconnected the aileron control cables completely.

One of the the modifications in the Embraer Service Bulletins changed a cable routing support near rib 21 of the aircraft with complex instructions and an inaccurate depiction of the aileron cable routing. These instructions were hard to understand and made it hard to understand how the cables were to be atttached. It also never mentioned that it was critical that the cables never crossed.

At some point during this work, the aileron control cables were inverted, that is, the up cable terminated in the down position and the down cable terminated in the up position.

Control cables routing view as found after the accident

Admiral Cloudberg offers a clear explanation of what went wrong in his write-up of the event: Flying the Unflyable Plane: The near crash of Air Astana flight 1388

It was here that the critical error occurred: during reinstallation on October 11th, the technicians inadvertently crossed the cables in both wings, connecting the “up” cables to the “down” fittings on the hydraulic power control units, and vice versa. There were no markings on the cables which would have distinguished them, nor was there any design difference that would have prevented them from being fitted the wrong way around. According to Embraer, the main assurance against swapping the cables was the fact that no procedure ever called for them to be removed at the same time, ensuring that there was only ever one way to put a removed cable back in. But Embraer’s own service bulletin SB-190–57–0038R2 could not be accomplished without removing both cables simultaneously, so this basic design principle was violated, with the predictable result that the OGMA technicians installed the cables the wrong way around.

To be clear: it should not be possible to invert the aileron cables. The E190s certification rested on the idea that the engineers could not attach them wrongly as only one cable would be changed at a time, so there was only one space in which it could be installed. Once this protection was removed, there was no easy way to see where the cables were meant to attach.

Complying with a further service bulletin, the engineers replaced the installed stainless-steel control cables with carbon-steel cables. They lubricated and adjusted the aileron control cable system, ensuring that the new cables were fitted exactly as the old ones had been. There was no way for them to see that the cables they were replacing had been crossed, and so, new cables were crossed as well.

There were some inconsistencies in the records, including wrong dates on the sign-off cards and inspection procedures which were not performed.

This was partially caused by the pressure to deliver quickly. The maintenance was due to be completed before the 24th of October but the maintenance organisation were unable to make the date, so they asked for another week, promising Air Astana that the aircraft would be ready for the 31st of October. To ask for another extension would be a catastrophe.

On the 26th of October, the maintenance team powered up the aircraft for operational tests.

The Engine Indication and Crew Alerting System (EICAS) lit up with an error message: FLT CTR NO DISPATCH.

This message shows the location of the issue (Flight Control) and the severity (aircraft should not be dispatched for flight).

The error meant that the engineers were unable to complete further operational tests, including the aileron control system operational test. Everything was focused on resolving this error.

Maintenance continued. The aircraft underwent further testing. This included the flight control tests, but the E190 passed them, as the ailerons and spoilerons were only tested for full and free movement, not the direction in which they were moving. It seems like it should be obvious that the ailerons were moving the wrong way but not a single engineer noticed that the control cables for the ailerons were inverted. It’s important to note that the spoilerons went in the correct direction, which is why the effect was not a simple reversal. Instead, the correlation of the flight controls to the aircraft angles appeared to be random but of course, that was only obvious after they were in the air.

On the 31st of October, the aircraft was delivered to the area manager, who allocated a small team to investigate the FLT CTR NO DISPATCH fault. By now, there was significant pressure to resolve the problem and deliver the aircraft as quickly as possible.

They attempted to clear every possible fault that could cause the FLT CTR NO DISPATCH alert. The list of possible faults did not include the aileron cables: it was a further test that triggered a further test that resulted in the fault making its way back to the Flight Control Modules. That is to say, the wrongly-installed aileron cables were at the heart of the matter, but this was not obvious to anyone as the results were two steps removed.

Air Astana, wanting their aircraft back in operation, brought in their own experts. Two Air Astana staff came to the site in hopes of resolving the fault: the director of technical asset management and a material supply technician, joining an Air Astana maintenance technician who had been shadowing the C-Check. Embraer also sent an engineer to help troubleshoot. Everyone involved concluded that they had a software problem: one of the four Flight Control Modules must be faulty. They began replacing the modules one by one. When they replaced the final module, FCM 4, the FLT CTR error message finally went away.

Test #4 event sequence and findings

This meant that they could finally hand-over the aircraft as the fault had been cleared. They did not rerun the tests which had caused the fault to appear in the first place, as the aircraft was now almost three weeks overdue and clearly showing as ready for dispatch.

On the 11th of November, the aircraft was prepared and positioned for the Air Astana crew to accept delivery. During the final operational checks, the Air Astana crew detected more issues: there was a problem with the ventilation in an avionics department and one of the hydraulic systems was low on fluid.

The maintenance organisation went back to work and resolved the issues. Once again, the E190 was prepared and positioned for handover to Air Astana.

The Air Astana crew found no further faults and they finally took possession of the overdue E190. On the flight deck were the captain and two co-pilots. The three Air Astana technical staff were in the cabin as the only passengers.

The flight crew did not have any guidance on accepting an aircraft after heavy maintenance. Following the standard pre-flight checks, they confirmed that the ailerons moved freely in all directions but again, they didn’t spot that the ailerons were misconfigured.

All were happy that the aircraft was in good shape, never guessing the nightmare that would start once the aircraft left the ground.

FIGURE 2 SHOWING ABNORMAL AIRCRAFT ATTITUDE AS BASED ON THE FDR

Once they had landed safely at Beja, it did not take the investigators long to confirm that both aileron surfaces were acting in the opposite direction of the control yoke command.

From the report

The incorrect installation of the aircraft’s flight control cables system during heavy maintenance activities, resulted in the aircraft’s aileron system reversal operation and consequent loss of control during flight, was identified as the cause of the accident.

Contributing to the event, the existing gaps in the maintenance service provider quality assurance system and failure to implement an effective internal and external maintenance activities oversight, enhanced by the aircraft design and the corresponding maintenance instructions regarding the ailerons control system configuration.

The rush to get the aircraft back in service resulted in gaps in quality control at the maintenance organisation. The fact that everyone involved, including Air Astana technical management and an Embraer engineer, believed that they had found the fault and fixed it did the rest.

Sequence of lost opportunities

The GPIAA made four safety recommendations targeted at the maintenance service provider and the aircraft manufacturer. Embraer did not change the design of the cables but rewrote the maintenance manuals and revised the service bulletin so that only one cable would be detached at a time. Air Astana wrote their own in-depth report on the flight and instituted training for maintenance acceptance flights. In addition, the pre-flight control check now includes a call-out of the direction of the movement. The maintenance organisation

The aircraft was written off and the three pilots were offered a leave of absence at their discretion. All three reported back to duty after only a few weeks.

Category: Accident Reports,

18 Comments

  • You would think that this incident would result in at least a service bulletin requiring that the “up” cable be marked with paint or shrink-tubing at both ends to distinguish it from the “down” cable.

    Of course they might have to disconnect the cable to mark it so who knows what else might happen.

  • In the electrical world, this kind of thing is commonly dealt with by pre-preparing the two cables with gender/form A connectors at each end of one cable, and gender/form B at each end of the other cable. You do this prep in the workshop, where you can clearly see that each cable has the correct connectors at each end. The fixings/coupling on the mechanisms/equipment are then manufactured with matching gender/form terminations permanently attached.

    This makes it physically impossible to cross-over the cables (or at least you can cross them over, but only if you do a whole 360deg twist).

    You see an equivalent setup in the handedness of fuel and oxygen threaded connectors, where the former is LH and the latter RH. Irrespective of colour codes/letter visibility, and irrespective of whether you aren’t paying attention, it is physically impossible to misconnect fuel to oxygen.

    Depending on just removing one cable at once feels a very “exciting” choice for a safety critical piece of kit…

  • Some very good comments here. Note the absence of remarks regarding the maintenance procedures. Anyone heard of the old saying “Haste will make waste”?
    Of course, the last word about the maintenance probably has not been spoken yet, so comments about that subject are in this stage probably premature and not helpful.
    In my days, when testing the controls, we devised: “Left turn, all up”. Meaning: aileron and spoilers up to reduce lift on the wing that is in the direction of the intended turn.
    Apart from this, the pilots, especially the captain, did an outstanding job.

  • There were several problems in play here. There is no record of them all being seriously addressed. That is concerning.

  • Sylvia: the next-to-last paragraph (listing actions taken to prevent this happening again) ends “. The maintenance organization” — looks like the description of what they did got left off.

    Like others, I am massively impressed at a landing that controlled after the plane had been cartwheeling around the airspace. I wonder whether the captain was at the end ignoring the instruments and reacting to “The aircraft does THIS when I do THAT.” (The San Francisco Exploratorium had a hands-on exhibit that let visitors show themselves how quickly the brain can adjust to discrepancies between action and sense; very impressive, but I don’t know whether it’s been kept over 24 years.) cf “What Goes Up…” describing helicopter piloting as observing how the copter moves when the pilot does something, and repeating that action when that movement is desired.

    • The instruments weren’t broken. Their first step was sensible: turn off the automation. It’s impossible to figure out what’s going on when you don’t know what the computer is doing, and it may be a source of error itself. Their final step must’ve been “leave the ailerons alone”? since the other controls did work; I think it explains why they weren’t able to correct for the crosswind. Amazing feat of airmanship for sure!

  • “Embraer did not change the design of the cables but rewrote the maintenance manuals and revised the service bulletin so that only one cable would be detached at a time. ” Jesus Christ all bloody mighty. And the next time it happens Embraer will say smugly “Had they but followed the Manual…” I can’t believe that re-engineering the cables to make cross connecting physically impossible wasn’t recommended, or carried out by Embraer on their own initiative. You can’t fix built in traps by changing the bloody book!

    • I presume there was no easy way to fix this but yes, it’s clearly a design flaw. I would like to think that a new plane would not get certified as it clearly sidesteps the actual requirement.

  • Ok, I’m the old, hard nosed Marine. When you do your control check, look out the window and watch the control surfaces go up and down. If you are turning right and the left aileron on the PIC’s side of the plane is going up instead of down, Captian, you got a problem.

    But those checks only happen on Carrier decks these days. Civil Aviation has come to rely on automation and repeated reliability. When that reliability fails, crashes happen.

    As Brian Shul said, “Any landing you can walk away from is a good landing.”

    These Pilots were some good Jocks to get the plane back on the ground without killing anybody. I would love to buy them a Pint and hear the stories of Butt Pucker Factor.

    We had some plumbing issues lately, in the back garden we had a Backhoe, 2 tractors and a DitchWitch, working away to make our black water go away. I went out to find the Forman with a shovel, digging away. He looked at me and said, “$175,000 worth of equipment and it all comes down to this, a Man with a Shovel”

    Piloting a plane is no different, you can have a Glass Dash, the best Autopilot, Radar, StormScope and bluetooth everything but when the shit hits the fan; you need a man with a shovel in the cockpit.

    • But can you actually see anything useful from the cockpit of the E190? I note that they went to the cabin to try to make sense of the control surface movements in response to the captain’s input during the flight.

    • I totally agree. Check full and free movement and in the correct sense. A cockpit indicator to show such movement direction required.

    • Military pilots get canopies because they need wider vision, but civil aircraft get front windows; that’s why they need people on foot to watch where the rest of the plane is. (I don’t know whether it’s still an issue, but when I was actively flying the Jeppesen plate for Dulles taxiiing warned of parts of the apron where two jumbos couldn’t pass without ~”wingwalkers”.) However, I’m not convinced that an indicator is a good idea as it’s one more thing to go wrong — who makes sure it’s not cross-connected if the ailerons are? Maybe the answer is groundchecks — pilot says “I’m doing something” (deliberately not saying what) and ground crew lead tells him what moved which way — but that would require training and time from crews that are probably overworked as it is.

      • That works. Not much training. Tailplane “trailing edge up- trailing edge down”. Etc etc. Been there. Done it. Easy.

  • Sylvia may have a point. Most readers don’t have experience with this type of aircraft and maybe the ailerons are not visible from the cockpits. But since the airplane came out of heavy maintenance, why not ask the engineers in the cabin to check?

  • Some facts and factors:
    Haste. Time pressure.
    Insufficient information in the manuals about how to execute this procedure and checks.
    Insufficient training and supervision of personnel involved in the maintenance procedure.
    Insufficient communication between the people involved.
    Flight crew not familiar nor trained with regards to the necessary steps to check the aircraft before acceptance after heavy maintenance.
    It would seem that virtually all parties involved must share a portion of the blame for this incident. The cockpit crew actually did a good job, all concerned and deserve praise. I would feel comfortable in an aircraft piloted by them.

Post a comment:

Your email address will not be published. Required fields are marked *

*
*
*

This site uses Akismet to reduce spam. Learn how your comment data is processed.