Lion Air 610: The Faulty AOA Sensor
Previously, I stepped through the maintenance of the aircraft registered PK-LQP, which was destroyed in the crash of Lion Air flight 610 on the 29th of October 2018.
From that post, it is clear that the aircraft was released back into service with a faulty AOA sensor on the left (Captain’s) side. Now I’d like to look at that sensor and piece together what happened.
The Boeing 737-8 (MAX) has two independent angle of attack (AOA) sensors which are installed just behind the nose on either side of the aircraft.
In basic terms, the angle of attack allows us to understand the amount of lift the wing is generating. The key external component of the AOA sensor installed on the Boeing 737 MAX is a vane which rotates to align with the airflow. The vane is connected to two internal resolvers, each of which independently measures the rotation angle. The vanes then provide the measured angle of the direction of the airflow relative to the fuselage.
One resolver is connected to the Air Data Inertial Reference Unit (ADIRU) computer for its side (left or right) and the second resolver is connected to the respective Stall Management Yaw Damper (SMYD).
The Air Data Inertial Reference Unit combines and measures various information to provide inertial position and track data, as well as attitude, altitude and airspeed data, to the flight deck displays. So the data from the left-side AOA sensor is processed and passed on by the left side ADIRU to the captain’s flight displays.
The Stall Management Yaw Damper (SMYD) uses the information from the other resolver to calculate and send commands to the Stall Warning System. The left SMYD, with information provided from the left AOA sensor, will activate the captain’s stick shaker based on a number of factors, including an excessive angle of attack as measured by the AOA sensor.
In the post about the maintenance history of the aircraft, we saw that the left-side AOA sensor was replaced on the 28th of October 2018 after multiple reports of speed and altitude flags appearing on the Captain’s primary flight display which had been connected to maintenance messages of faults related to the SMYD.
The sensor which was removed was returned to Batam Aero Technik (part of the Lion Air Group), who handed it over to the investigation after the crash.
The Component Maintenance Manual required that the test be done using the output from Resolver 1. Resolver 1 was fine and the AOA sensor passed testing when its output was used. However, when Resolver 2 was tested, the recording instrument could not interpret the output.
A further resolver accuracy test was performed with the internal heaters operating, to determine if unintended electrical coupling was occurring between the heaters and the resolvers.
The first two measurements taken on Resolver 2 showed that the values were unstable similar to values observed in previous resolver accuracy testing. Once the unit warmed up with the heater operation the unit resolver 2 output stabilized and was within the CMM performance requirements. The remaining Resolver 2 values were found within limits. The first two measurements were re-taken and were found within limits. The vane and case heaters were turned off and the values for Resolver 2 went unstable after 12 minutes and 51 seconds. The sine and cosine signals (observed on an oscilloscope) were also being observed and went to zero when the API output went unstable.
That is, the output from Resolver 2 was correct when the internal heaters were operating but stopped working again when the temperature dropped.
The investigation discovered that a loose loop of the very fine magnet wire from the primary rotor coil was trapped in the epoxy which was meant to hold the end cap insulator in place. The trapped magnet wire thus adhered to both the end cap insulator and the rotor shaft insulator which had very different coefficients of thermal expansion (CTE). As the wire expanded and contracted to the two different environments, the wire became fatigued and showed multiple ridges and cracks before breaking. The wire failure created an intermittent open circuit, dependent on temperature. The sensor worked fine at temperatures above 60°C (140°F) but failed at temperatures below that.
This explains the ongoing issues in the weeks before the crash. However, a new AOA sensor was installed on the left side in Denpasar. This sensor could not be tested in the same way, as it was completely destroyed in the crash. The only course open to the investigators was to track the maintenance history of the sensor.
The sensor had previously been installed on the right side of a Boeing 737-900ER operated by Malindo Air. Malindo Air’s maintenance had removed the sensor on the 19th of August 2017 after maintenance reported that the SPD and ALT flags appeared on the first officer’s pilot flight display during a pre-flight check. Analysis of the aircraft data showed that fifteen flights had recorded “no computed data” for the right flight director parameter, so presumably the SPD and ALT flags had appeared fifteen times on the first officer’s display before the sensor was replaced.
The sensor was returned to Batam Teknik who sent it to Xtra Aerospace in Florida to be repaired.
The Work Order at Xtra Aerospace noted that the sensor had been removed as the speed and altitude flags had displayed and the speed and altitude indications had not appeared. The preliminary inspection showed that the unit was in “fair but dirty” condition. It failed the operational test which was recorded as caused by an eroded vane causing erroneous readings.
They disassembled the unit, replaced the vane, calibrated the unit and put it back into testing. The work order notes that the required tests were satisfactory. Xtra Aerospace approved the unit for return to service in November 2017 and returned it to Malindo Air who returned it to Batam Teknik in December. That unit was then sent to Denspasar on the 28th of October 2018.
The investigators went to Xtra Aerospace to review and document the maintenance records, the test equipment and the procedures used during the repair process. The interesting thing they discovered was the process of the vane-slinger-shaft assembly removal and replacement, as per the Component Maintenance Manual.
During this process, the resolver gears and damper gear are disengaged from the main gear. This allows them to rotate independently, as the main gear is fixed to the vane shaft. The main gear is then removed from the vane-slinger-shaft.
So there is nothing to keep the resolvers in their original position and they can independently rotate to a new position.
The angle is verified in an Alignment Accuracy test which is measured using an Angle Position Indicator. The Component Maintenance Manual specifies a specific Angle Position Indicator but notes that equivalent substitutes may be used. The Angle Position Indicator used by Xtra Aerospace offered a level accuracy equal or better than the recommended unit and the FAA had approved the equipment as in conformance with the Component Maintenance Manual.
However, that Angle Position Indicator had an extra function, which was a relative mode for testing. The equipment in the manual only allowed for absolute testing and so there was no reference to relative mode. Xtra Aerospace did not develop any specific instructions for the use of their Angle Position Indicator, which would have drawn attention to the existence of the two modes. The FAA should have required this to be documented but had overlooked the lack of a written procedure.
As a test, the technicians developed a procedure to test whether a 25° bias could inadvertently be introduced into both resolvers as a part of of the vane-slinger-shaft replacement.
Resolvers are calibrated to output 45° when the vane is at its zero position. If the Angle Position Indicator is set to absolute, then the resolver output reads as 45°. However, if the switch is then moved to the relative postion, the Angle Position Indicator reads as 0°, as the 45° offset has been established. If you move the AOA vane to a new position, the the Angle Position Indicator will display the actual angle -45°. The 45° offset is constant through the full range of the vane rotation as long as the REL/ABS switch remains in the relative position.
In the test in which the technicians introduced a 25° error, they found that if the REL/ABS toggle switch (relative/absolute) in the Angle Position Indicator was selected to the REL position, then the unit would pass testing, even though the angles recorded after the testing showed a 25° bias over the full range of vane travel.
It is impossible to say for certain whether this is what happened to the replacement AOA sensor. However, the investigators were able to demonstrate that in this case, an AOA sensor which was calibrated and tested could result in an equal bias introduced into both resolvers. This bias was not visible during AOA sensor calibration and was not detected in the return-to-service testing as required by the Component Maintenance Manual (CMM).
In February 2019, Collins Aerospace repeated the Peak API offset demonstration at its facility. The test was repeated for the benefit of NTSB and FAA personnel who did not witness the original demonstration at Xtra Aerospace. The procedures followed during the February 2019 demonstration were fundamentally the same as those performed at Xtra Aerospace in December 2018. The conclusions were identical. First, that an equal offset could inadvertently be introduced to both resolvers. Second, that the magnitude of the offset is essentially random. And third, that the offset could go undetected through the CMM return-to-service tests.
All it took was for someone to accidentally toggle (or forget to reset) a simple switch from absolute mode to relative.
When the sensor arrived in Denpasar, the Batam engineer replaced the left AOA sensor. He did not have the equipment for the recommended installation test and so had to use an alternative method, which involved deflecting the vane to the fully-up, centre and fully-down positions while checking the SMYD computer for each position. The engineer said that the values displayed for each position were correct, however, contrary to company policy, he did not record them.
After the crash, he submitted photographs of the SMYD display to prove that the AOA sensor passed the installation test. The timestamp on the photographs was earlier than the spare part had arrived at the premises and investigators were able to show that the images showed a different aircraft. Thus, there’s no record at all of the results of the installation test.
If the AOA sensor had been shipped with the revolvers offset, then this should have been obvious from the SMYD display, which would show the misalignment angle or, at its maximum lower stop, AOA SENS INVALID. Investigators tested the offset at 33°, 21° and 20°. In every case, the sensor failed the installation test.
The flight recorder recovered from the wreckage showed that the left-side AOA sensor showed a value approximately 21° higher than the value given by the AOA sensor on the right.
It’s hard to believe that the engineer conducted the installation test and checked the values from the SMYD. It seems much more likely that he never conducted the AOA sensor installation test and released the aircraft into service with the misaligned AOA sensor.