A flawed intelligence report generated by artificial intelligence almost led the United States military to plan the interception of a Chinese ship in the Middle East, which they incorrectly believed was transporting components for a nuclear weapons program, according to four sources familiar with the incident, as reported by CNN.
The document circulated within the US military amidst ongoing tensions with Iran. American forces were preparing to board the vessel, and military aircraft were deployed in response, noted one source familiar with the events.
Just before the planned operation, US officials scrutinized the report, which was drafted by an analyst at the Special Operations Command. They discovered that the report had been created using AI technology, specifically a chatbot that erroneously identified the ship’s cargo.
The AI Report Was “Entirely False”
The report was described as “entirely false,” with one source stating it “almost sparked a war.” Any US operation against the Chinese vessel could have escalated into an armed conflict between the two nations.
The analyst had queried a chatbot regarding specific information about the ship’s manifest, which originated from the US Pacific Command. It remains unclear whether the chatbot was a commercial tool or a government product.
The chatbot had combined information from open sources with classified data obtained through signal interceptions, arriving at a decisive but incorrect conclusion regarding the ship’s cargo. Subsequently, the analyst employed AI once again to compile these findings into a standard intelligence report, which is typically trusted by military officials, and forwarded it onward.
Not an Isolated Incident
According to one source, the type of “hallucination” produced by the tool used by the analyst in this case is not an isolated incident within the intelligence community since these tools have begun to spread throughout government agencies.
The implementation of AI in the military is intended to assist in quicker decision-making on the battlefield, such as target selection or locating military resources. However, some seasoned intelligence officials believe these tools pressure analysts to produce and disseminate information more rapidly, leaving room for errors.
Particularly, younger analysts, according to multiple sources, are native users of these tools and tend to place greater trust in them without applying critical scrutiny.
