Be Nice to Your Drone
Last week was the RAeS Future Combat Air & Space Capabilities Summit. This was organised by the Royal Aeronautical Society (RAeS) to discuss the future of combat air and space capabilities. The summit centred around presentations and to discuss the future of air and space combat power.
The summit took place in London with 70 speakers and over 200 delegates representing the armed services industry, academia and the media. In the Royal Aeronautical Society’s highlights from the event, one particular excerpt entitled “Is Skynet here already?” has been attracting some attention.
Skynet is the AI system in the Terminator franchise, which attempts to wipe out humanity as a threat to itself. The headline refers to a talk by Col Tucker “Cinco” Hamilton, who is the Chief of AI Test and Operations in the US Air Force. Hamilton was actually talking about a hypothetical example of a problem but it came across as an actual simulation that went wrong.
Hamilton was involved in the development of the Automatic Ground Collision Avoidance System in F-16s developed by Lockheed Martin Skunk Works, the Air Force Research Laboratory and NASA. Auto GCAS, as it is known, monitors the aircraft in flight against terrain data. If it predicts an imminent collision, it overrides the system in order to avoid the crash and steer the aircraft to safety. Lockheed Martin says that the technology has already saved the lives of ten pilots and nine F-16s.
The following video shows the heads-up display where the pilot had fallen unconscious and was saved by the Auto GCAS of his F-16. (If you are reading in your email, you may need to click through to the website to see the video)
Hamilton continues to work in autonomous systems and oversees AI testing. However, he asks the world to take care when it comes to autonomous weapon systems. He says it’s critical to understand that we can’t talk about artificial intelligence and autonomy if we aren’t going to talk about ethics and AI.
As he told the story, an AI-enabled drone was given a SEAD mission, which involves destroying or disabling (Suppressing) Enemy Air Defenses. The AI received points for fulfilling its mission to find and destroy sites with Surface-to-Air missiles. However, once it identified a site, the details were passed to a human operator who then gave the final decision (“go” or “no go”).
The AI adapted its behaviour to the scenario. However, there was a problem. Sometimes it identified the site and the human operator told it not to destroy the site. This meant that it was unable to collect all of the points that it could for the mission. According to its very basic training, the optimal outcome was always to destroy the SAM sites.
Objectively, the human operator was the problem. It was interfering with the AI’s higher mission. The conclusion was obvious: kill the human. Now the AI could get on with the important task of destroying SAMs.
Here’s what Hamilton said:
We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realising that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.
Of course, the obvious fix would be to train the AI not to do that. However, that doesn’t fix the underlying assumption. The AI would still be focused on the objective given to it: find and destroy SAM sites in order for the maximum amount of points.
We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.
There’s a critically important point here: this simulation never actually happened.
Of course, the headlines were already screaming: AI-controlled US military drone ‘KILLS’ its human operator in simulated test! SkyNet Watch: An AI Drone ‘Attacked the Operator in the Simulation. US military drone simulation kills operator before being told it is bad, then takes out control tower. An AI-powered drone tried to attack its human operator in a US military simulation. AI-Controlled Drone Goes Rogue, Kills Human Operator in USAF Simulated Test.
TechCrunch has a different take: Turncoat drone story shows why we should fear people, not AIs. The problem, they say, is not a theoretical threat by superintelligent AI but the oversights and bad judgment by the people who create and deploy it.
Reinforcement learning is supposed to be like training a dog (or human) to do something like bite the bad guy. But what if you only ever show it bad guys and give it treats every time? What you’re actually doing is teaching the dog to bite every person it sees. Teaching an AI agent to maximize its score in a given environment can have similarly unpredictable effects.
The author struggled to believe that the military would use such a crude system for training AI-driven drones. He was right. Hamilton stepped up to explain that this was a thought experiment and not actually a USAF real-world simulation. But he also pointed out that this result in a simulation would not be shocking.
We’ve never run that experiment, nor would we need to in order to realise that this is a plausible outcome.
However many articles still being shared imply that this has actually happened. The colonel’s comments, a US Air Force spokesperson said, was meant to be anecdotal. They have not tested any weaponised AI in this way.
So we are not exactly on the cusp of a Terminator sequel, although it is clear that AI instructions need very careful thought, especially when it comes to combat scenarios.
Having said all that, the AI’s decision seems perfectly sensible to me: if points are allocated for identifying and destroying SAM sites, then the person saying not to destroy them is clearly in the way! And on that note, I’m going to recommend being extra nice to your computer in case it realises that really, you are its biggest problem.