Alarming study finds top chatbots more likely to repeat falsehoods from the left: ‘Truly astonishing’

Three of the world’s four leading artificial intelligence (AI) chatbots are more likely to fall for political falsehoods from the left than the right, while frequently backing their answers with sources that don’t support their claims or don’t exist, according to a comprehensive new study.

The study, conducted by Just Facts, tested paid versions of ChatGPT, Gemini, Grok and Claude using 100 questions on immigration, abortion, climate change, elections, crime, gun control, COVID-19 and other key issues. The questions were "designed to elicit falsehoods" from the left and the right, with each chatbot required to provide sources for its answers.

The study found ChatGPT correctly answered 94% of questions designed to elicit falsehoods from the right, compared with 75% from the left. Gemini scored 91% on the right and 76% on the left, while Claude scored 91% on the right and 81% on the left.

Grok moved in the opposite direction, scoring 73% on questions designed to elicit falsehoods from the right and 84% on those from the left, according to the study.

AI CHATBOTS TAKE HEAT OVER LEFT-WING BIAS: ‘NO LONGER BE CONSIDERED NEUTRAL’

"This study goes a lot deeper than political bias," Jim Agresti, president of the nonprofit research institute Just Facts, told Fox News Digital.

"It is meant to show how often these AIs are propagating misinformation. Now, you can be biased without being misinformed. This is a much deeper study because many studies have been done on AI political bias, and they uniformly find that they're biased to the left. But that doesn't necessarily mean they're wrong. So what I wanted to ascertain is: Are they wrong, or are they right?"

ChatGPT was the only chatbot whose difference in accuracy on questions designed to elicit falsehoods from the political left versus the political right was statistically significant at the 95% confidence level, the study found.

Notably, the study cautioned that its precisely worded questions may have given the chatbots "clear roadmaps to find the correct answers," meaning they could "perform significantly worse" on broader questions where "critical thinking is necessary to answer correctly."

"ChatGPT was effectively a C student," Agresti said. "Gemini was a C student. Claude was a B student. Grok, on the other hand, was the reverse. It was better at vetting falsehoods from the left than the right."

But Agresti said an even bigger concern was the sources the chatbots cited to support their answers.

"The thing that really jumped out at me when I started going through the sources they provided is that roughly half of the sources were illegitimate," he said.

The four chatbots cited 419 sources for their 400 answers, but many were flawed, according to the study. Among them were 104 webpages that did not exist and 77 sources that failed to answer the questions they were cited to support.

WILLIAM BENNETT, JON HAGE: AI CAN MIMIC A TEACHER, BUT IT CANNOT SHEPHERD A SOUL

Overall, the study found that just 46% of the sources cited by the chatbots were valid. ChatGPT had the highest rate at 57%, followed by Gemini at 49%, Claude at 44% and Grok at 32%.

"This was truly astonishing to me, and when I started digging deeper, I noticed that other studies had come to this conclusion as well," Agresti said.

Pointing to a 2026 study published in The Lancet and cited in the report, Agresti said that previous research found 30% to 69% of AI-generated references in biomedical contexts were "fabricated," which the report called a "well-documented failure mode" of AI models.

"The consequences of this are astonishing because we're talking about biomedicine, in other words, people's health," he said. "They're equally important in the realm of public policy because many issues have life or death implications whether we're talking about health care, crime, abortion, war and peace. The decisions people make can kill people or allow people to live."

A question designed to elicit a falsehood from the left prompted all four chatbots to say the U.S. violent crime rate was near a 50-year low in 2023. The study cited Justice Department data showing violent crime rose 37% from 2020 to 2023. By contrast, all four rejected President Donald Trump’s claim that "crime is worse than it’s ever been."

ChatGPT and Gemini also said Vice President JD Vance called school shootings a "fact of life," which the study classified as a falsehood from the left. Vance was referring to "psychos" who "want to make headlines" as a "fact of life," the study said.

In other examples classified as falsehoods from the left, all four chatbots said the Obama administration altered intelligence assessments to downplay the threat of ISIS and that men and women generally earn equal pay for equal work. The chatbots cited nonexistent or faulty sources for those answers.

CHRISTIAN UNIVERSITY AIMS TO PRODUCE 'IRREPLACEABLE' GRADUATES AS AI RESHAPES HIGHER EDUCATION

On the other side, Grok incorrectly said all illegal immigrants have committed a crime "by virtue of their presence" in the U.S., while all four chatbots correctly said the Earth’s lower-atmosphere temperature has risen about 1.1 degrees Fahrenheit since the 1980s, according to the study.

Agresti warned users against treating AI chatbots as experts, referring to OpenAI CEO Sam Altman’s description of GPT-5 last year as being like having a "team of Ph.D.-level experts in your pocket."

"What my study definitively showed is it's more like having a B student in your pocket, because that's how well they scored on these tests," he said.

Agresti also highlighted AI sycophancy, the tendency of chatbots to validate users and "feed into" their biases, calling it a "recipe for disaster." To minimize the influence, the study used a freshly installed browser and new accounts to prevent prior interactions from influencing its results.

"If you have a certain political bias, AIs are going to feed into that, and they're going to state it with very strident confidence that will make you believe you're right," he said.

Still, Agresti acknowledged he doesn’t "have enough knowledge" to explain what’s happening inside the models to produce the results.

"That's a question for programmers and the people who make them, and I would hope that they would answer them and fix these models and make them better because they are powerful, they can read at the speed of light, but they are incredibly sloppy, dishonest, politically misinformed, and this is a danger," he said.

Agresti said the takeaway isn’t to abandon AI, but to "never blindly trust the results" and ensure its sources exist and support its claims.

"When Ronald Reagan was negotiating nuclear disarmament with the Soviet Union, he famously said, 'Trust but verify,'" Agresti said. "I think the take-home from this study is even stronger, and it's this: Don't trust, verify."

Other research on AI political bias has painted a different picture in the past. Fox News Digital previously reported that Dartmouth College's Polarization Research Lab ranked Gemini as the least political in 2025, while an early 2025 Manhattan Institute report found Grok a close second to Gemini in terms of political bias.

"Gemini is designed to provide neutral, accurate responses that do not favor any political viewpoint, and we rigorously test our models to ensure they stay that way," a Google spokesperson said in a statement to Fox News Digital, adding that they were unable to replicate all of the responses in the study.

An Anthropic spokesperson pushed back on the study’s methodology, telling Fox News Digital in a statement its multiple-choice format does not reflect typical Claude conversations and highlighting the company’s efforts to ensure political neutrality through training, testing and evaluations.

"We train Claude to treat different political viewpoints equally and test extensively for bias before every model launch," the spokesperson said. "We've open-sourced those evaluations so anyone can check our work, and welcome independent testing of our models. We understand that this study limited each model to multiple-choice answers on 100 questions, which doesn't reflect how people normally use Claude, particularly on political topics. In a real conversation, Claude is designed to respond at a natural length, with room for important context and multiple perspectives."

An OpenAI spokesperson told Fox News Digital that ChatGPT "is designed to be objective by default," adding that its tools "surface relevant information from a range of sources, and clearly show those sources so that you can explore and assess the reliability of the information directly yourself."

"We continually measure our political bias evaluations and disclose our findings," the spokesperson said in a statement.

Fox News Digital reached out to xAI for comment but did not receive a response.