Automated to Death

Photo: United Photos/Reuters

The crash in February of Turkish Airlines Flight 1951 just short of Amsterdam’s Schiphol International Airport, which killed 9 people and injured 86 others, raised this concern anew. As the aircraft passed through 1950 feet, the left radio altimeter failed and indicated an altitude of –8 feet, which it passed on to the autopilot, which in turn reduced engine power because it assumed the aircraft was in the final stages of approach. The pilots did not initially react to the warnings that something was wrong until it was too late to recover the aircraft.

”When we start removing active learning for the operator, the operator begins to overtrust the automation,” Rovira says. ”They’re not going back and gathering those data pieces that they need” to make an effective decision.

Another issue associated with overtrusting the automation is that it can encourage ”practical drift,” a term coined by Scott Snook in his book Friendly Fire: The Accidental Shootdown of U.S. Black Hawks over Northern Iraq (Princeton University Press, 2002). The phrase refers to the slow, steady uncoupling of practice from written procedures.

Illustration: Boswell/MCT/Landov

We see how that happened in the Royal Majesty incident, where the watch officers failed to follow established procedures. This was also the case in the October incident involving Northwest Airlines Flight 188 on its way to Minneapolis-St. Paul International Airport, which overshot the airport by 150 miles. The pilots claimed they were working on their laptops and lost track of the time and their location. The aircraft was on autopilot, which in normal circumstances leaves the pilots with little left to do other than monitor the controls.

 Again, when you are only a system’s monitor, especially for an automated system that rarely, if ever, fails, it is hard not to get fatigued or bored and start taking shortcuts.

The situation isn’t hopeless, however. For some time now, researchers have been working to address the ironies and paradoxes of automation. One new approach has been to address the issues from the human point of view instead of the point of view of the system.

”We draw a system’s boundary in the wrong place,” Thomas states. ”There is an assumption that the system boundary that the engineer should be interested in [sits] at the boundary of the sensors and actuators of the box that is being designed by the engineers. The humans who are interrelating with these systems are outside it. Whether they are operators, pilots, controllers, or clinicians, they are not part of the system.

”That is just wrong,” Thomas adds. ”The system’s designer, engineer, and overall architect all need to accept responsibility for the ways those people are going to act.”

Victor Riley, associate technical fellow in Boeing Flight Deck, Crew Operations, argues that there needs to be a two-way dialogue between the operator and the automated system.

”The operator-to-the-system part of the dialogue is more important than the system-to-the-operator part,” Riley says. ”People see what they expect to see, and what they expect to see is based on what they thought they told the system to do.”

Studies by Parasuraman, Rovira, and others have found that operators of highly reliable automated systems will often perform worse than if they were operating a lower-reliability system, which seems paradoxical.

Parasuraman explains that ”if you deliberately engineer anomalies into the automation, people rely less on it and will perform a little bit better in monitoring the system. For example, if the system is 90 percent reliable, operators will be better at picking up the 10 percent of the errors than if the system is 99 percent reliable.”

Rovira also says that operators need to be able to see how well the automation is working in a given context.

”The goal for us as designers is to provide an interface that allows a drill-down if the operator needs to query the system, in the event they have a different perspective of the decision than the automation has given them,” Rovira says. ”Or if not a drill-down, there should be some visibility or transparency right up front about what the underlying constraints or variables are that make this decision not totally reliable.”

Maybe one way to remind ourselves of the potential effects of the ironies and paradoxes of automation is to simply pull the plug.

”If we don’t want people to depend on automated systems, we need to turn them off sometimes,” Thomas observes. ”People, after all, are the backup systems, and they aren’t being exercised.”