Lionair flight 610 : The Maintenance
One of the aspects of Lion Air flight 610 has been the quality of the maintenance. It is clear, now, that intermittent problems related to the air data system had been logged and clearly had not been resolved in a satisfactory manner.
I have been wanting to do a deeper analysis of the various aspects of Lion Air flight 610 but to be honest, I haven’t been sure where to begin. I think there’s a logic to following the aircraft around for the weeks previous in order to get an understanding of the fault(s) and the level of maintenance received.
For this post, I’m going to presume basic familiarity with crash. I have already covered the Preliminary Report on Lion Air flight 610. However, I’d like to invite you to highlight areas that are unclear or confusing, which will help me to know where further explanation would be useful and I’ll tackle those aspects separately.
There’s a lot of ground to cover, so my writing about the maintenance doesn’t mean that the MCAS system or the previous flight or the crash itself are not important, just that they aren’t the focus of this specific post. Instead I want to follow the events that affected aircraft serial number 43000, with the registration PK-LQP, in the weeks before the crash.
The aircraft was a Boeing 737-8 (MAX) manufactured in 2018 with a Certificate of Airworthiness issued on the 15th of August 2018. Before crashing, it had flown 895 hours and 21 minutes with a total of 443 cycles (a take-off and landing makes for one cycle). The first issue related to the crash came up when it was just six weeks old.
On the 9th of October 2018, twenty days before the crash, an error of Angle of Attack Signal is Out of Range was detected by the left-hand (captain’s side) Air Data Inertial Reference Unit shortly before arrival in Jakarta.
This is important as it shows what seems to be the start of an ongoing series of errors and flags to do with the air data and the angle of attack readings of the aircraft.
The angle of attack (AOA) is the angle between the direction of the relative wind and a reference line on the aircraft or the aircraft’s wing; a simple way to describe it is the difference between where a wing is pointing and where it is going. Increasing the angle of attack increases lift and induced drag up to a certain point, known as the critical angle of attack. If the critical angle of attack (usually around 17°) is exceeded, the air begins to flow less smoothly and starts to separate from the upper surface of the wing. As the angle of attack increases, the wing loses lift and the aircraft stalls.
The stall speed of an aircraft changes based on configuration and weight, so the angle of attack cannot easily be determined by the pilots while in flight. Modern commercial aircraft have an AOA indicator: the angle of attack is measured using an AOA sensor and displayed on the primary flight display. They also have warning systems to signal a high AOA, for example aural alarms and stick shaker.
While the Boeing 737 was being parked, the STBY PWR OFF (stand-by power off) light illuminated. Then a number of circuit breakers tripped: the DC battery, the Auxiliary Power Unit Generator Control Unit, the Generator Control Unit for both engines and the Generator Disconnect of both the left and the right generator. The engineer in Jakarta reset all of the circuit breakers and ran the engine to test before logging an entry to say that the problem was resolved. The aircraft was released back into service.
On the 26th of October, just three days before the crash, the flight crew of the aircraft that day arrived in Manado and logged a report that the SPD and ALT flags had appeared on the captain’s pilot flight display (PFD).
These flags appear over the speed and altitude information to show that there has been a data failure. The maintenance light on the overhead panel was also illuminated. The engineer in Manado checked the Onboard Maintenance Function to check the maintenance messages and found one highlighting a fault related to the Stall Management Yaw Damper (SMYD). After performing the associated task for the message, he did a self-test of the Stall Management Yaw Damper. The self-test passed and the engineer erased the maintenance message. The airspeed and altitude indicators now correctly showed on the captain’s display so the engineer released the aircraft for flight.
The Boeing 737-8 (MAX) flew to Denpasar next, arriving the following day. The SPD and ALT flags had appeared again during this flight along with the MAINT light on the overhead. The engineer in Denpasar checked the Onboard Maintenance Function and found another message to do with the Stall Management Yaw Damper: AD DATA INVALID. He performed the self-test which passed, and the MAINT light extinguished. He released the aircraft for flight.
The next flight for the 737 was a round trip between Denpasar and Lombok. Upon arrival back in Denpasar, the flight crew did not report any problems.
However, on the next flight that day, from Denpasar to Manado, the SPD and ALT flags again appeared on the captain’s display. This time the SPEED TRIM FAIL and MACH TRIM FAIL lights also illuminated during the flight. The speed trim and the mach trim both trim the stabiliser (as does the MCAS).
The stabiliser is a horizontal piece on the tail which tilts up and down in order to affect the pitch. The pilots can trim the angle of the stabiliser for nose-up or nose-down by using the electric trim switches or the manual trim wheel. The autopilot and other systems within the aircraft can also make trim inputs. For those interested, here’s more detail from The Boeing 737 Technical Site page about flight controls and stabiliser trim:
Speed trim is applied to the stabilizer automatically at low speed, low weight, aft C of G and high thrust. Sometimes you may notice that the speed trim is trimming in the opposite direction to you, this is because the speed trim is trying to trim the stabilizer in the direction calculated to provide the pilot with positive speed stability characteristics. The speed trim system adjusts stick force so the pilot must provide significant amount of pull force to reduce airspeed or a significant amount of push force to increase airspeed. Whereas, pilots are typically trying to trim the stick force to zero. Occasionally these may be in opposition.
Mach trim is automatically applied above M0.615 (Classics onwards), M0.715 (-1/200) to the elevators. This provides speed stability against Mach Tuck; i.e. as Mach increases, the centre of pressure moves aft and the nose of the aircraft tends to drop.
The important thing to understand in all this is that the speed trim and mach trim are automatically applied to the stabiliser in every 737, just as the MCAS trim inputs are in the 737 MAX.
The pilot can disengage the electric trim, and thus the automatic trim inputs, by flipping a cutout switch on the control stand. Generally, moving the control column (or yoke) in the opposite direction will stop the trimming action, so if a nose-up trim input is made then pushing the control column forward to pitch down will stop the trim input.
All that said, the really important thing to understand here is that the air data was already wrong and the speed trim and mach trim were failing as a result.
At this point, the 737 MAX had arrived in Manado for an overnight stay, with a flight to Denpasar scheduled for first thing the following morning.
The engineer used the Onboard Maintenance Function to find the error messages. This time when he performed the self-test of the Stall Management Yaw Damper, it failed. He found correlated maintenance messages and stepped through other equipment tests which led him to receive two more maintenance messages: AIR DATA INVALID and AOA SIGNAL FAIL.
Briefly: AOA stands for Angle of Attack. The performance of an aircraft is affected by the speed that the aircraft moves through the air; it is critical for the aircraft’s Angle of Attack. If your aircraft is about to stall and you aren’t able to reduce the angle of attack, you will crash. A common cause for a stall is that the airspeed is too low: the wings lose lift and it doesn’t matter one bit what your groundspeed is, your aircraft will stop flying. On the other hand, if your airspeed gets too high, it can cause fatal structural damage to the aircraft. The important thing to understand here is that Air Data is used to determine your angle of attack and the faulty angle of attack on the left (captain’s) side was a critical point in the crash on the 29th.
The engineer at Manado reset a number of circuit breakers to do with the left Air Data Inertial Reference Unit and then ran another self-test of the Stall Management Yaw Damper and the Digital Flight Control System. This time both tests passed.
As a part of the fault isolation procedure, he should have checked the wiring of the Air Data Module and the Air Data Inertial Reference Unit. But it was raining and there was a risk of lightning in the bad weather, so the engineer did not perform the wiring checks. He inspected the electrical connectors and did not find anything wrong. He recorded in the log that the problems were not active.
When the flight crew arrived the next morning (the 28th of October and the day before the crash), the engineer spoke to them about the actions he had taken. This particular flight crew had experienced one of the previous faults with the SPD and ALT flags and requested that more be done to fix the underlying problem.
The engineer suggested that they would be better off doing this in Denpasar.
The 737 MAX departed Manado for the flight to Denpasar and, as the flight crew expected, they experienced another data failure and the SPD and ALT flags again appeared where the speed and altitude information should have been. In addition, the SPEED TRIM FAIL and the MACH TRIM FAIL lights had illuminated again. The Onboard Maintenance Function showed AD DATA INVALID and STALL WARNING SYS L on the status message page.
The 737 MAX offers an advanced onboard network system (ONS) to connect airline operations and maintenance with key data. This is a network of aircraft systems collecting a high volume of aircraft data and consolidating that data with the Onboard Maintenance Function (OMF) which can be access via the flight deck or remotely, for example on a tablet. This data is used to create maintenance procedures, enabling the engineers to perform maintenance and fault isolation for each of the aircraft systems without having to access each one individually in the electronics equipment bay. It allows for focused troubleshooting and report.
The engineer at Denparsa once again performed the self-test of the Stall Management Yaw Damper, which failed. The Onboard Maintenance Function showed various maintenance messages again referring to the Inertial Air Data and the AOA signal. The engineer reset the circuit breakers of the left Air Data Inertial Reference Unit and conducted the various self tests again. This time the self-tests all passed.
However this time, probably at the request of the flight crew, the engineer took into account that the faults kept recurring and decided to replace the AOA sensors for trouble-shooting. There was no AOA sensor on site so he ordered an AOA sensor from Batam Aero Technique in Batam and grounded the aircraft until it arrived in Denpasar.
Once the AOA sensor arrived, he removed the existing left-hand sensor and installed the new one which had arrived from Batam Aero Technique. The next step was to perform an installation test.
The Aircraft Maintenance Manual gives two methods for doing this but one of them required test equipment which the engineer did not have. The second method involves deflecting the AOA vane up and down to the furthest and checking each position on the Stall Management Yaw Damper computer to ensure it gives the correct indication. There is no record of the results which means that it is now impossible to know whether the installation test was successful or not.
The engineer then performed the heater test, which involves dropping water onto the AOA vane. It passed the test. He used the Built-in Test Equipment on the control display unit of the flight management computer and it showed that there were no current faults.
Jumping ahead for a moment: The engineer claims to have completed all of these tests as normal and even supplied photographs, which he claimed he had taken after he had replaced the AOA sensor replacement.
However, the photograph of the Captain’s pilot flight display has a time-stamp that was taken before the AOA sensor part had arrived in Denpasar and the photographs of the Stall Management Yaw Damper were not the same aircraft.
So when I say that the aircraft passed all the standard tests after the new AOA sensor was installed, we should remember that this is based on the word of one man, an engineer who did not correctly log his results. He may have cut corners and certainly had high motivation to claim that he had run all the necessary checks but no evidence to back his claims. Or maybe he did everything correctly except for the log and the photographs.
He released the 737 MAX back into service on the evening of the 28th October (local time), the night before the crash. It was put into action for the next flight which was from Denpasar to Jakarta. This flight deserves looking at in more detail but within this context I’d like to note that when the flight crew landed in Jakarta, they reported that after they took off, they received IAS DISAGREE and ALT DISAGREE alterts, that is, the indicated air speeds and the altitude information as collected from multiple sources did not all match. The FEEL DIFF PRESS light had also illuminated which meant that there was a very specific failure.
A common reason for the illumination is that one of the hydraulic systems powering the elevators (which control the pitch) has failed. This might be because one of the elevator pitots has failed. The Elevator Feel Computer receives dynamic pressure from two pitot tubes mounted on either side of the vertical stabiliser.
Finally, it may be signalling a fault related to the Stall Management and Yaw Damper which uses a reduced system A pressure. If this reducer fails, the A system pressure related to the feel actuator is higher than normal, which triggers the FEEL DIFF PRESS light.
The engineer at Jakarta responded to this report by flushing the left pitot and the static Air Data Module and ran an operational test. He noted that the results were satisfactory. Then he cleaned the electrical connector to the computer which had illuminated the error and ran a self-test, which passed.
He released the 737 MAX back into service at 2:30 in the morning of the 29th, local time, ready for Lion Air flight 610 departing at 05:45.