West Pointers can be trained to better evaluate, appreciate AI, study finds

Although Americans generally trust AI less than, say, Chinese people, they are often willing to accept a chatbot’s wrong answers. As Pentagon leaders push broader use of such tools, a new paper offers some reassuring news: West Point cadets can be trained to be more appropriately skeptical of AI’s output—while remaining broadly optimistic about its potential.  

Researchers from Georgetown University, the University of Pennsylvania, and the U.S. Military Academy published a paper last week comparing 236 West Point cadets to a demographically similar sample of 702 members of the public. The paper explores automation bias—humans’ tendency to over-rely on automation—and algorithm aversion, an inclination to “prematurely distrust automated outputs in ways that increase the risk of accidents or mistakes.”

One hypothesis they sought to test was that the pressures of battle would make military members would be more inclined to trust faulty outputs from a decision-support system, or DSS.

“Major powers such as the United States, China, and Russia are experimenting with integrating AI DSS into their command structures, thereby decreasing their sensor-to-shooter timelines,” they write. This might lead officers “to delegate decision-making authority to AI DSS.”

The military has long grappled with AI-enabled decision assistants. But the recent arrival of large language models like ChatGPT, Claude, and Gemini led West Point to declare a special academic focus for the 2024-25 school year:  “The Human and the Machine: Leadership on the Emerging Battlefield.”

This and other efforts seemed to work. In the study, West Pointers demonstrated “AI knowledge scores” nearly twice as high as the general public, and were less than half as likely to commit an automation bias error—that is, to trust a mistaken chatbot.

Cadets were also more likely to assess AI output by looking at the tool’s own confidence indicators, something few regular users do.

“These findings align with the way the USMA seeks to train cadets so they can achieve justified confidence, that is, to be properly calibrated so their expectations of an AI system’s accuracy match the reality of the accuracy of the system,” the researchers wrote.

In the study, about twice as many cadets expressed worry about the dangerous consequences of AI than did members of the public, but they were much less likely to describe AI as “sinister.” 

Cadets were also more enthusiastic about the potential of the technology: 87.7 percent saw strong beneficial applications for AI compared to 72.5 percent of the public, while 79.5 percent described it as “exciting,” versus 61.6 percent of the public.

UPenn’s Michael C. Horowitz, a former deputy assistant defense secretary for force development and emerging capabilities, told Defense One that the study “shows what some of the opportunities might look like for further training within the U.S. military at least. This suggests that training to reduce the risk of automation bias for military personnel using artificial intelligence could be effective.”

Horowitz noted that cadets’ training is “probably not representative of the average military right now, but it shows the path forward, at least for the U.S. military.”

Such training might also benefit the U.S. public, which might soothe the worries of technologists and leaders who say Americans’ declining trust in AI, compared to populations in China, is a national security concern.