There is no battlefield exception to human-centered AI

Stanford has built one of the world’s most influential brands around human-centered AI. From the start, the Stanford Institute for Human-Centered AI (HAI) asserted that AI should augment people’s work, not replace them. At its “AI in the Loop” conference, HAI faculty argued that humans must remain at the center of decision-making. That governing principle of human-centered AI must be considered most carefully when it comes to war, where the stakes are lethal. 

Anthropic’s Claude became the first AI model approved to operate on classified military networks, and Anthropic says it is extensively deployed across the U.S. military for analysis and operations. Palantir’s Maven system, used for intelligence analysis and weapons targeting, included prompts and workflows built with Claude Code.

When the Trump Administration designated Anthropic a supply-chain risk after a dispute over safety guardrails, Pentagon users resisted the phaseout, estimating that recertification could take 12 to 18 months, demonstrating AI’s early entrenchment in our military. As that phaseout drags, the Pentagon is now solidifying Palantir’s Maven as their go-to AI system. 

According to Reuters, American investigators believe U.S. forces to be responsible for the strike on the school in Minab, Iran, which reportedly killed 168 school affiliates, including schoolchildren. The Washington Post reported that the school was on a U.S. target list and that Maven (powered in part by Claude) was used to suggest targets, issue precise coordinates and prioritize them by importance.

It is not clear whether Maven nominated the school specifically; a former senior defense official cautioned the Post against assuming it did, noting the site had likely been on U.S. target lists for years. A preliminary Pentagon review cited by the Post indicates the strike was probably the result of an intelligence error about the location, and a full investigation continues.

Nonetheless, the episode points to a larger problem: once AI systems are integrated into operational workflows, their outputs carry undue weight. Regardless of who or what first nominated the school, the chain that approved it did not catch that the underlying intelligence was stale in a military campaign whose pace and scale Maven is theoretically built to enable. People have begun to defer to AI systems even after machine errors should have raised alarms. 

This overreliance is documented among average AI users. University of Pennsylvania (UPenn) Wharton researchers recently found that when an AI assistant was wrong, users still followed it about 3/4 of the time and reported higher confidence in their answers even after it led them astray. In a productivity app, that is frustrating. In war, it becomes a tragedy.

Stanford has already studied a similar failure in cars. When a person resumes control after automation, they need a measurable period to readapt before their hands behave safely enough to navigate the road. The National Transportation Safety Board warns that drivers are susceptible to automation complacency, and the Insurance Institute for Highway Safety found that only one of 14 partial automation systems it tested earned an acceptable safeguard rating. In other words, partial automation can put a human to sleep at the wheel without taking the wheel away.

Military AI creates the same trap, only with lethal stakes. A human can remain “in the loop” while losing control over the reasoning that matters. When the UPenn researchers added a 30-second time limit to a person’s designated activity, it roughly tripled the odds that users would surrender to a faulty AI rather than override it.

Once a system has framed the picture and compressed the timeline, the person at the end of the chain may still approve the action but fail to actually pass judgment themselves. The head of U.S. Central Command has said that “humans will always make final decisions on what to shoot and what not to shoot.” But a human signature is not the same thing as human control.

In 2024, HAI researchers put five off-the-shelf large language models through military and diplomatic simulations. All five demonstrated forms of escalation that were difficult to predict.  That finding should concern anyone who thinks the answer is simply to keep a human somewhere nearby. Without enough time and context, the human does not challenge the machine. The human ratifies it.

None of this means AI can never be useful. The International Committee of the Red Cross (ICRC) argues that AI tools can help humans synthesize information, offer a wider range of context to a decision and reduce harm to civilians. But the ICRC also warns that, in legally mandated decisions on lawful targeting, AI outputs may inform but must not replace human judgment, or else the human risks becoming a “human rubber stamp.” Even the U.S. military’s own directive says commanders and operators must exercise appropriate levels of human judgment over the use of force. Anthropic itself says today’s models are not reliable enough for fully autonomous weapons.

Stanford, too, has entered the debate. HAI faculty and fellows have called for explicit policy on AI-designed weaponry usage and warned of AI’s power for individual surveillance, information key to military operations. HAI is well-placed to publish the conditions under which human-centered AI principles call for constraining a military application and name the categories of judgment that should not be so automated to erode human control. For example, adding friction to the design to avoid inappropriate reliance on automation. “Augment, don’t replace” works only if it means more than leaving a person at the end of an opaque, machine-built process without time to question or refuse its conclusion.

We do not yet know everything about Minab, but we know enough about AI acceleration to reject the fiction that a human is in charge simply because they are present. In war, the cost of substituting a person with technology is too high to risk. Stanford helped define human-centered AI, the very thing meant to resist the dangers that we’re watching play out in the U.S. military. Now is the time to prove that the definition holds.

Utsav Gupta is a Stanford Master of Liberal Arts candidate researching human purpose in the age of AI. He is founder and co-CEO of Filarion, building AI litigation tools and spatial computing systems, and serves as commissioner for Palo Alto Utilities. Views are his own.

The post There is no battlefield exception to human-centered AI appeared first on The Stanford Daily.