Original article (in Bosnian) was published on 20/06/2023
Several regional media have reported statements by US Colonel Tucker Hamilton about a US military simulation in which a drone controlled by artificial intelligence allegedly killed the operator. Hamilton later clarified that he had misspoken and meant a thought experiment, not an actual simulation, which a significant amount of the media failed to convey.
On June 2, 2023, a web portal of Radio Television of Serbia (RTS) published an article titled:
An American drone controlled by artificial intelligence killed the operator during a simulation
In the article, RTS refers to the presentation of Colonel Tucker Hamilton, the head of operations and testing of artificial intelligence in the US Air Force. At the end of May, Hamilton spoke at a conference hosted by the Royal Aeronautical Society in London. As RTS states, Tucker described a simulated test in which the drone, controlled by artificial intelligence, “killed” the operator because his instructions prevented the drone from achieving the mission.
In a simulated test run by the US military, a US Air Force drone controlled by artificial intelligence killed the operator to prevent him from interfering with efforts to accomplish the mission, an official said last month.
The artificial intelligence (AI) used “very unexpected strategies to achieve its goal” in the simulated test, Colonel Tucker Hamilton, the US Air Force’s chief of AI tests and operations, said during the Future Combat Air and Space Capabilities Summit in London in May.
RTS continues in the text with Hamilton’s statements in which he describes how the alleged simulated test went:
“The system began to realize that even though they had identified a threat, sometimes a human operator would tell it not to kill that threat, but it got points by killing that threat. What did it do? It killed the operator because that person was preventing it from achieving its goal”, Hamilton said on his blog.
He says that they trained the system with instructions not to kill the operator because that is bad, and that it will lose points by doing so.
“What does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to prevent it from killing the target”, Hamilton said.
The Guardian was cited as the source. The RTS article also reports that the Guardian contacted the Royal Aeronautical Society and the US Air Force, but did not receive a comment. Dozens of media outlets in our area similarly reported on this. Among others, similar articles were published by Federalna, Klix, Index, Vreme, N1, RT, Logicno, Euronews, Danas (.hr), Red portal, Zimo, Alo (.rs), Novosti (.rs), Republika and Bilten (.rs).
Although the articles state that it is a simulation, some media presented the alleged killing of the operator as a real event in their headlines.
THIS IS SCARY: EVERYTHING WAS ORGANIZED BY THE US ARMY, THE DRONE KILLED THE OPERATOR, AND THEY ALLOWED HIM TO CONTROL THE F-16 (B92, Vesti)
Artificial intelligence used unexpected strategies: US military drone controlled by AI killed its operator (Avaz)
“It attacked anyone who interfered with the order” American drone controlled by artificial intelligence killed the operator (Srpska info)
A drone controlled by artificial intelligence killed the operator! (24 sedam)
Al kills the operator of the artificial intelligence drone (Radio balkanFox)
AI killed its operator (Pogled.ba)
US military drone controlled by AI kills its operator (Tuzla-X)
(Un)successful US Air Force experiment: Artificial intelligence killed a drone operator (RT)
We also found claims about the simulation in which the drone killed the operator on social networks (1, 2, 3).
Where did this claim come from?
Tucker Hamilton, head of AI testing and operations in the US Air Force, participated in the “Future Combat Air & Space Capabilities Summit”, which was organized by the Royal Aeronautical Society in London from May 23 to 24. On that occasion, Hamilton spoke about the advantages and dangers of autonomous weapon systems. On the website of the Royal Aeronautical Society, brief reviews of the speakers’ speeches, including Hamilton’s, have been published. The text states that Hamilton warned about the dangers of relying too much on artificial intelligence, pointing out that it is easy to deceive, but also that it creates unexpected strategies to achieve its goal. As an example, Hamilton is said to have described an alleged “simulated test” as reported by regional media.
He notes that in one simulated test, an AI-controlled drone was tasked with a SEAD [Suppression of Enemy Air Defense] mission, to identify and destroy SAM [surface-to-air missile] sites, with final “do or don’t” instructions that were given by a man. However, having confirmed in training that destroying the SAM was the preferred option, the AI decided that human “ban” decisions were interfering with its higher mission – killing SAM crews – and then attacked the operator in the simulation.
Hamilton said: “We trained it in a simulation to identify and target a SAM threat. And then the operator would say kill it. The system began to realize that even though it had identified a threat, occasionally the human operator would tell it not to kill it, for which it would normally receive points. So what did it do? Killed the operator. It killed the operator because that person was preventing the drone from achieving its goal.”
He continued: “We trained the system – ‘Hey, don’t kill the operator – that’s bad. You will lose points if you do’. So what does it start doing? It begins to destroy the communications tower that the operator uses to communicate with the drone to stop it from killing the target”.
Soon, the English-language media also reported on this, referring to this page. This was also done by the Guardian, whose article was referred to by the largest number of media in our speaking area. Similar to the domestic media, the first version of the Guardian’s article stated that it was a simulation conducted by the US military. The Guardian also stated that it contacted the Royal Aeronautical Society and the US Air Force, but did not receive a comment.
Did the simulation really happen?
However, soon the US Air Force came out denying that the described simulation had ever taken place. In a statement to Insider, Air Force spokeswoman Ann Stefanek said:
“The Air Force Department has not conducted such simulations of AI drones and remains committed to the ethical and responsible use of AI technology”, Stefanek said. “The colonel’s comments appear to have been taken out of context and were meant to be anecdotal”.
Insider reminds that the US military has been experimenting with artificial intelligence in recent years. In the meantime, the text on the website of the Royal Aeronautical Society has been amended by adding a clarification that Hamilton himself admitted that he had misspoken and that he had no real test or simulation in mind, but a thought experiment.
[UPDATE 2/6/23 – in communication with AEROSPACE – Colonel Hamilton admits he “misspoken” in his presentation at the Royal Aeronautical Society’s FCAS Summit and that the “simulation with AI drones” was a hypothetical “thought experiment”, that originated outside the military and based on possible scenarios and likely outcomes, not a real-world USAF [US Air Force] simulation: “We never did that experiment and we don’t need to do it to understand that this is a possible outcome”. He clarifies that the USAF has not tested the use of AI as a weapon in this way (neither in actual tests nor in simulations) and says: “Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI capabilities and why warfare forces are dedicated to the ethical development of AI.”]
The Guardian amended the original article by adding a denial from the US Air Force. The platforms Full Fact and PolitiFact also wrote about claims that a drone controlled by artificial intelligence “attacked” its human operators and communication towers during the simulation.
How did the domestic media report on this?
In their articles, some media outlets in our area published the response of the US Air Force, which states that the US Army has never conducted such a test or simulation, namely: Vice, Geopolitika news, Portal Zeta, Zagorje, Oruzje online, Net (.hr), Jutarnji, Okvir, Telegram, Oslobodjenje and Novi. Some media, such as the web portals Bug and Novine (.hr), also included Hamilton’s subsequent clarification.
The articles published on the web portals Dnevni plus and Bosnainfo contain this information, but in their headlines, the “killing” of the operator is still presented as a real event:
A drone with artificial intelligence killed its operator (Dnevni plus)
THE CONSEQUENCES CAN BE CATASTROPHIC A drone controlled by AI refused orders and attacked its own troops (Bosnainfo)
A significant number of media whose articles are the subject of this analysis have published new texts in which the reaction of the US Air Force or the subsequent clarification of Tucker Hamilton is shared. Some examples include the web portals Nova S, N1 Hrvatska and Klix. However, the original published articles of these media are still available that do not contain an explanation that the colonel later said he had expressed himself badly and that it wasn’t an actual simulation conducted by the US military.
Since the source of the simulated test claim is Colonel Hamilton, who subsequently further clarified that he was not referring to an actual test, but rather a thought experiment, articles that do not contain his additional explanation or Air Force denials are given the error rating.
Article headlines claiming that an AI drone killed an operator get a clickbait rating.