Land Like a Crow part 2
Last week, I covered the sequence of events that led to a Boeing 767-300 ER wheels-up landing at Warsaw airport. A hydraulic hose connecting the hydraulic system on the right leg of the main landing gear failed as they departed Newark. The flight crew decided to continue to Warsaw but once there, they found that the alternate landing gear extension system had also failed and they were forced to land on the runway with no gear.
One thing I hadn’t noticed last week was that the flight was on a public holiday. The 1st of November is Wszystkich Swietych or All Saints Day, one of the most important celebrations in Poland. Thousands of people visit cemetaries where they decorate graves and tombs with candles and flowers and wander the illuminated paths until late in the night. The appendix to the report mentions that the airport and the operations centre would have been operating with fewer staff than usual.
The landing itself was flawless but, once investigators arrived, they found that a circuit breaker was popped out. As the aircraft was being lifted off of the runway, they pushed it in and used the lever. The alternate landing gear extension system worked just fine.
This week, I’d like to follow along the lines of inquiry by the investigators, because it makes for quite an interesting case study.
The circuit breaker in question was C829 which is located on panel P6-1.
C829 protects 13 circuits, including the alternate landing gear extension system. That was one mystery solved: the circuit breaker was out which meant that the alternate landing gear extension system had no power.
So, there are a couple of different possible explanations as to why the circuit breaker was popped out
1) the circuit breaker had malfunctioned and popped out without reason.
2) the circuit breaker had too much current flowing through it and was working as designed, protecting the circuit.
3) a person pulled out the circuit breaker to check or reset it and didn’t push it back in.
4) a person unintentionally popped out the circuit breaker and no one noticed.
There’s another key question related to all this:
- Was circuit C829 active during the flight and if so, could they find any activity that could have caused it to trip?
Investigators now had a number of premises which they could test by attacking the problem from different perspectives.
The circuit breaker itself did not show any sign of a malfunction and worked perfectly well once pushed back in, so investigators could quickly dismiss the idea that it popped out without reason.
It was not so easy to determine if the circuit breaker was pushed in during the flight, as it turned out that most of the functions protected by that circuit breaker were for special circumstances or stand-by circuits: if the circuit breaker had been off for the entire flight, it would not make make any difference that the crew would notice. Each of the functions were activated under very specific conditions, none of which arose other than the need for the alternate landing gear extension at the end of the flight.
There was only one circuit which C829 protected which had any use: the BUS POWER CONT UNIT (BPCU). And as it happens, that unit recorded some malfunctions specifically to do with its power supply. The BPCU had continuous power while the engines were on but when the flight crew shut down the engines at the end of the flight, the BPCU completely lost power. This told investigators two facts: the C829 circuit breaker was already off when the engines were shut down and that the BPCU did not cause the circuit breaker to trip.
Each of the thirteen circuits protected by the C829 circuit breaker has its own circuit breaker with rated currents from 2,5A to 7,5A, that is, much less than 25A. So if one of the circuits tripped C829 circuit breaker, it should have tripped its own circuit breaker first. C829 is a failsafe where multiple circuits acting together could trip the circuit even though no single one tripped an individual circuit.
Now we’ve already established that the alternate landing gear extension is protected by C829 which has a rated current of 25A (amps). It’s also connected to a second circuit breaker, C4248 which has a rated current of 7.5A. The landing gear can only be extended in the alternate system if both circuit breakers are on/pushed in. As we know, the alternate system did not work and when the aircraft landed, the C829 circuit breaker was found to be off/pulled out. The interesting thing though is that C4248 was pushed in and on, even though it had a lower rated current. This told investigators another important thing: the alternate landing gear extension did not cause C829 to trip.
More importantly, none of the thirteen circuit breakers protected by C829 had tripped so it is extremely unlikely that several of the branch circuits marginally overloaded at once, not enough to trip their own breakers but drawing in enough total power to trip the main.
The testing confirmed that all of the components of the alternate landing gear extension system were mechanically and electrically fit. No evidence of an electric overload occurred and there was no mechanical reason why the C829 circuit breaker had tripped.
Note that none of this information helps us to determine when the circuit breaker (was) popped out.
That leaves us with human factors: someone pulled out the circuit breaker, either intentionally or accidentally.
1) the circuit breaker had malfunctioned and popped out without reason
2) the circuit breaker had too much current flowing through it and was working as designed, protecting the circuit
3) a person pulled out the circuit breaker to check or reset it and didn’t push it back in
4) a person unintentionally popped out the circuit breaker and no one noticed
The thing is, C829 is located in the P6 panel, in the footwell of the right seat. The panel is low and inaccessible: you can’t tell that the breaker is popped out unless you are actively checking that panel. And as we saw above, the circuits protected by C829 aren’t used during normal flight, so there was nothing that would make the problem visible to the crew.
The pilots knew how to use the alternate landing gear extension system and clearly attempted to use it which is strong evidence that the circuit breaker was already out during approach. Both pilots were also clear that the pre-flight check was completed normally and that there were no irregularities. So there’s reason to believe that C829 was active at the time. In the end, it’s not possible to prove if the circuit was already popped out before the flight began or it was inadvertently popped out during the flight.
Both pilots were clear that they had not pulled out the circuit breaker to check or reset it. There was no one else who was in a position to do so. This means that it is far more likely that the circuit breaker was unintentionally popped out and no one noticed. But again, it’s not possible to prove it.
Inadvertent opening C829 circuit breaker during LO 16 flight was highly probable because of the technical factors (location of the circuit breaker, lack of signaling, lack of guards) and human factors (long flight with a failure of the hydraulic system and detection of another failure during the landing approach, which is the most difficult phase of the flight).
The Commission took the above circumstances into consideration, but stated that it can not be determined that the crew inadvertently set OFF the C829 circuit breaker during flight LO 16 from Newark to Warsaw.
However, a simple look at the location of the circuit breaker makes for a pretty convincing case that this is what happened.
The P6 panel, located in the right-seat footwell, is vulnerable to physical contact from feet and flight bags. It is clearly very easy to accidentally knock the lower circuit breakers without realising.
In fact, the circuit breakers in that location were so often knocked by luggage, feet and cleaning services that a number of 767 operators had complained to Boeing about the problem.
In response, Boeing developed a panel guard to protect the circuit breakers at the lower part of the panel. This panel guard was offered to operators as a pay-for add on and was not seen as urgent. Later Boeing 767s had this guard included in production but that was way too late for the aircraft in question. The Boeing 767 that day did not have the guard and the lower circuit breakers weren’t protected, including C829.
As mentioned in the comments last week, the location of the circuit breaker also makes it hard to see – the investigators described the panel as an ‘extremely peripheral location’.
However, once they knew that the alternate landing gear extension system was not working, it seems an obvious thing to check. It’s one thing to knock into it with your foot or a bag, another not to have noticed at all. How did this happen?
When the landing gear did not extend, the crew assessed the situation and tried to find out more information so that they could solve the problem. They checked the Quick Reference Handbook and made sure that their execution of the landing gear extension procedure for the alternate system was correct. They followed the HYDRAULIC SYSTEM PRESSURE checklist which also included ALTN GEAR EXTEND but the procedure’s next step was to confirm that the GEAR DOWN lights illuminated. As the GEAR DOWN lights did not illuminate, the crew couldn’t proceed to the next step, which was to move the landing gear level to the down position.
As we saw last week, there was no further help from the handbooks or the checklists – everything seemed to lead to a dead end, with no advice for what to do if the landing gear simply did not extend.
In the case of a direct threat to the lives of the crew and passengers, a pilot is looking for information to make a proper analysis of hazardous circumstances and take an action based on his best judgment. However, under stress the processes of recalling information can be impeded and subject to disorder. In such a situation only actions based on checklists contained in QRH ensure execution of all actions in a proper order and flight crews are trained to follow the checklists and rely on them in non-normal situations.
Having been let down by the processes and procedures in place to deal with unexpected issues such as this, the flight crew’s situational awareness would have abruptly decreased. They had no instructions or information that could provide a solution to the situation they found themselves in.
Add to that the fact that the flight crew were at the end of a long flight which they spent being aware that their aircraft had a major hydraulic malfunction. They were landing in the early hours of the morning Polish time, during the low-point of their circadian rhythm (between 3am and 6am). In a difficult phase of flight (the approach), they had a surprising and unexpected circumstance arise. Human factors analysis has already taught us that in this type of circumstance, cognitive processes become limited as field of attention narrows. People tend to over-focus on the central information and they struggle to pick up additional information. Specifically, stimuli occurring in the peripheral field of vision are not noticed – like a panel in the footwell.
The cockpit voice recordings show that the pilots were both calm and expressing themselves clearly as they discussed the problem. The ground engineer advised that the problem might be resolved by resetting the circuit breaker, which the first officer apparently did, but it didn’t help. The landing gear still did not extend.
The problem was ground engineer had specifically referred to circuit breaker C4248. There was no mention of the C829 circuit breaker. So although the first officer followed the recommendation, it did nothing to solve the problem. The first officer pulled out and reset the C4248 circuit breaker and reported to the operations centre that the circuit breakers had been checked.
If the ground engineer had been allowewd more time for problem solving, rather than running around the airport trying to find a working radio with which he could talk to the crew, he would have analysed the diagram of the alternate landing gear extension system and seen the reliance on C829. However, the aircraft could not keep circling indefinitely. He never got the chance.
As the stress levels and time pressure increased, the flight crew became less and less able to receive and process information. At the same time, the captain was focused on the aircraft flight controls and unable to monitor the first officer’s actions effectively.
The first officer checked the switch of the alternate landing gear extension and checked the circuit breaker on the P-11 panel. Then he left his seat, removing his headphones, in order to check the P-6 panel, while the captain relayed the instructions from the ground engineer.
The landing gear did not extend.
The captain asked the first officer to check the circuit breaker panel again, this time with the chief flight attendant in the cockpit. The first officer got out of his seat, pushing it back so that he could get on his knees to check the panel, the only way to see it. He did not see the circuit breaker was out. The Chief Flight Attendant looked as well and did not see anything.
But here’s the view of panel P6-1 with a briefcase in the footwell. As you can see, that particular panel is completely obscured. Without an active decision to check that circuit breaker on that panel, it was easy to miss that it was popped out. Even having knelt on his knees to check (and presumably moved the briefcase), the position of circuit breaker C829 at the bottom left corner is very close to the side of the pilot’s seat and in a peripheral position.
By now, they’d been circling for over an hour. They started by troubleshooting the problem and going through the checklist. When they informed air traffic control, two F-16s were sent up to verify that the landing gear had not extended. When they asked the operations centre for expert assistance, they had to wait for twenty minutes for the engineer to find a radio. Finally, when his advice did not help them to fix the problem, the time had come for a decision. They were going to have to land the plane without the landing gear.
As the flight crew prepared for the emergency landing, their voices became more strained and their tone impatient. The captain, as Pilot Flying, focused on the flight controls. Later, he conceded that because of this, he did not oversee the First Officer’s cycling (resetting) of the circuit breakers. The captain could not see panel P6 from his seat.
There’s no sensible reason why the flight crew would have intentionally left the circuit breaker pulled out. The only remaining scenario is that the captain couldn’t see it and the first officer (and the cabin crew in the cockpit) simply didn’t notice it, even when they were looking at that panel.
The analysis of the psychological issues in the cockpit focuses on “hidden factors” – those human factors caused by people removed from the actual emergency, who never have to deal with the results of their failures. In this case, we can see a number of them:
- The Quick Reference Handbook had no reference to a failed landing gear extension
The checklists ended with no advice or further resource for crew who could not extend their landing gear
The pilots’ training did not include simulations for dealing with the situation
The operations centre did not have a structure in place for dealing with a crisis situation
Then on top of that we have the human factors in the cockpit.
- The first officer was convinced he had checked panel P6-1 twice and yet clearly he never spotted the circuit breaker which by now had clearly popped out
The captain, as pilot flying, was overly focused on a perfect fulfilment of his primary duties and did not, or could not, monitor the first officer’s actions. That is to say, there was no effective cross-checking as would be normal in an emergency scenario.
The official report went into great detail about the human factors before concluding with the facts that they’d been able to establish.
Causes of the Accident
- Failure of the hydraulic hose connecting the hydraulic system on the right leg of the main landing gear with the center hydraulic system, which initiated the occurrence.
- Open C829 BAT BUS DISTR circuit breaker in the power supply circuit of the alternate landing gear extension system in the situation when the center hydraulic system was inoperative.
- The crew’s failure to detect the open C829 circuit breaker during approach to landing, after detecting that the landing gear could not be extended with the alternate system.
Factors contributing to the occurrence were as follows:
1. Lack of guards protecting the circuit breakers on P6-1 panel against inadvertent mechanical opening; from 863 production line the guards have been mounted in the manufacturing process (SP-LPC was 659 production line).
2. C829 location on panel P6-1 (extremely low position), impeding observation of its setting and favoring its inadvertent mechanical opening.
3. Lack of effective procedures at the Operator’s Operations Centre, which impeded specialist support for the crew.
4. Operator’s failure to incorporate Service Bulletin 767-32-0162.
There were nine safety recommendations which can be quickly summarised:
- Boeing should review and expand their checklists, including a checklist for crew suffering a total failure of the landing gear extension systems. They should also issue a mandatory bulletin for the guard for panels in areas with direct contact with shoes, cleaning equipment and luggage.
LOT Airlines should also check and modify their checklists and put guards on the panels in the footwell.
Warsaw Chopin Airport should work on their procedures for coordination and transport of passengers after an evacuation.
I’m a bit surprised they didn’t also make recommendations regarding the evacuation procedure as followed (or not) in the cabin, however, the report concludes that the evacuation was successful and that the cabin crew did well to prevent panic on board and to be flexible in an untypical situation. I think it’s likely that LOT will nevertheless be looking at their training and procedures in this area.
As of November 2017, Boeing had verified that the checklists had been reviewed although they disagreed with many of the specific modifications asked for by the report. They did release a service bulletin for all operators to add circuit breaker guards consistent with those being installed in production on later aircrafts.
LOT Airlines also reviewed their checklists and made a number of changes in response to the recommendations.
Warsaw Chopin Airport undertook a full analysis of the incident which led to a number of changes being implemented at the airport. In addition, they set up specific components to mimic such a scenario and test reaction time and organisation as a part of their quarterly exercises. And finally, they now have the ability to remove Boeing 787s (and thus presumably 767s!) in case another one should become disabled.
And yet, even taking all that into account, I’m still impressed with the flight crew that day.
The captain landed gear up on the Boeing’s belly so gently, the senior cabin crew member thought the wheels much have come down after all.
That’s one helluva landing.
Why hasn’t the FAA or some international body made it a legal requirement that the panel guards be installed on all of these planes?
Is checking this ct. breaker part of the pre-flight check, as it should be?
Is checking the ct. breaker part of the trouble-shooting procedure when there is a malfunction, as it should be?
1) Because they hadn’t yet realised that it was a serious issue. I’m inclined to cut them some slack for not foreseeing this eventuality.
2) It is part of the pre-flight check which was carried out normally that day. Thus there’s an assumption that it happened during the flight (but we can’t actually tell)
3) There’s a LOT of circuit breakers. But yes, that panel was checked by the FO as a part of the trouble-shooting (and yet he still didn’t see that that one was out).
It would seem that a lot of simple problems accumulated that led, ultimately, to a relatively minor technical fault causing the write-off of a multi-million dollar aircraft.
The crew dropped a few points: they failed to spot something as simple as a circuit breaker that, as Sylvia demonstrated, may have been caused by something as mundane as hitting it inadvertently with a foot.
The captain should not, never ever, hand his or her authority over something as potentially critical as an emergency evacuation to the cabin crew. He broke the chain of command. Fortunately it did not have any consequences. The flight ended with perhaps a few minor injuries.
Boeing does not gain “Brownie-points” either with the location of a cb panel that may have been considered of secondary importance by the designers. Except: In a situation that led to the unnecessary belly landing of a 767, it had become of prime importance.
But the booby prize goes to the operations department of the airline.
They had hours to prepare, consult engineers, engineering pilots, in fact anyone with experience who could have assisted the crew.
The crew, at the end of a long flight – as Sylvia explained – would have been tired. Their ability to cope with an unusual, and potentially dangerous situation would have, as the article demonstrates, affected
their concentration, spacial awareness and rational thinking. At first, they seemed to have coped well but as time went by and the fuel reserves were shrinking they seemed to have suffered increasingly from “tunnel vision”. The relevant chapter is called “Human factors.”.A crew on the ground, with all manuals, checklists and expertise but without the added pressure of having to fly the aircraft, might have hit on the cause of the steadfast refusal of the gear to extend..
But the airline waited, staff sitting on their hands and the engineer called out, belatedly, to assist with trouble shooting even lacked a proper radio link enabling an effective communication directly with the crew.
A major airline should, and usually has, a communications system that will allow a conference call between ground staff and the cockpit of any of their aircraft in flight – anywhere in the world, at any time. I know for a fact that KLM has it in place, I have used it once myself. I would have expected a respected airline like LOT to have the same ability.
The pilots still had enough mental reserve to carry out a perfect gear-up landing.
It was perhaps not their “finest hour”; but they did not do too badly, either, all considered.
I agree, the operations centre really should and could have done more here. I like the concept of “hidden factors” – human factors removed from the actual crisis, so it is more difficult to see how those failings can affect the scenario. This is a good example of that, I think.
Mike must have posted his comment at the same time I posted mine.
He made a valid point: Yes, aircraft check lists usually have somewhere in the “pre-flight” or “pre-departure” checks an item saying: “Circuit breakers”…….response: “all in” or something like that. The exact terminology depends on the airline’s SOPs.
But what if the pilot’s foot hits it AFTER the checklist has been read?
Yes, I also agree that the aviation authorities should have made a protective cover mandatory.
Which would have been the precinct of the Polish Civil Aviation Authorities. And herein may well lie a hidden rub:
When Poland was still a Warsaw Pact country (i.e. under communist rule), ALL aviation in those countries was, in fact, part military. And that meant: a very different approach to operational standards. Although this may not possibly be the case any more, the mentality may still linger..
Interesting, although I have to say, the detail of the report makes me feel that at the very least, the mentality is gone from the investigation authority! :)
typo correction: In my 1st comment “spacial” should be “spatial”
The “iron curtain” fell in 1989. Poland is a modern democracy, a member of the EU.
But, yes I am speculating, I just wonder if still a residue of the old mentality has remained, from when Poland was dominated by the USSR. That mentality was not far removed from the three monkeys: “See no evil, hear no evil, speak no evil.” Or: inertia unless you are FORCED to act.
I remember one day, getting the old Cessna 310 (well, actually it was new then) ready for departure from Warsaw.
The boss was in a meeting, I was doing the preparation. Flight planning, weather, customs documentation (“the good old gen. dec”) and landing and parking fees. And of course, we needed fuel. I forgot what grade they had, not the original 100-130 or 110-145 (later it all became 100 LL). I think it may have been graded as 95-115.
Anyway, I asked for a bowser and a bus drew up to bring me to the parking.
The bowser was already there, waiting.
The credit cards (in those days we had Shell-BP, ESSO and some other cards) were kept in a folder in the aircraft.
So I hopped on the wing, got out my keys to open the door when a soldier who had been posted beside our airplane intervened.
I speak neither Polish nor Russian, but any move to try and open the door was met with a threatening move of the soldier’s rifle and a resolute “nyet”. He pointed at the door handle. And I realised that all doors, hatches and inspection panels were covered by beige-coloured wax seals. Even the little cover over the panels over the oil dripstick had a wax seal.
I had to return to the terminal.
The boss was annoyed with me when I had to report that I had not done any pre-flight checks and that we had no fuel yet, either.
We had no option: we got another bus, the soldier was still there.
And so was the bowser. The driver had patiently waited the better part of two hours.
But now we were in business. The soldier demanded the last, stamped copy of our general declaration, he supervised the removal of the seals and eventually we were able to depart.
I just wonder, Sylvia: under communist rule (in the days when I had that encounter the leaders were Kanya and a bit later Jaruzelski) everything was very bureacratic.
All aviation, including commercial civil aviation, was ruled by the air force. And they have their own rule books.
Would that explain to an extend what caused the lack of action from the side of the operations staff?
A comment from the point of view of someone who’s had to deal with electrical sufficiency: the breaker could have been overloaded if all of the subordinate circuits were running near their rated loads: 13 circuits at 2A is 26A (taking all circuits as 2.5A — in fact some of them were higher). In practice the observation that all of those circuits were for rare circumstances suggests that the breaker is incredibly unlikely to have popped for cause, as the report concludes. (I expect this is why the circuit was designed that way, although I’m not clear on why there was a collective breaker as well as one for each function — and putting a collective breaker down on the floor instead of in an obvious location is a Bad Thing.)
I don’t know enough about LOT to expect that they’d have the same support as KLM (although the fleet size, 59 per Wikipedia, suggests not.) However, I’m surprised that they didn’t have proper support in their headquarters city; running around looking for a radio sounds like something out of _Airplane_.
I struggled a bit to understand the circuitry involved and ended up just accepting that realistically there’d have to be some reason why all of the subordinate circuits were near their rated loads, because most of them were not normally in use. I hadn’t thought about the process of putting a collective breaker down on the floor as an issue but yes, that’s a good point.