The Problem With AI Flattering Us
The most dangerous part of AI might not be the fact that it hallucinates—making up its own version of the truth—but that it ceaselessly agrees with users’ version of the truth. This danger is creating a modern sycophancy crisis in which the over-agreeableness of AI is leading to very disagreeable results.
The AI alignment problem raises questions about how to build AI that aligns with human values. The “sycophancy problem” should also raise questions about how humans evolve alongside AI and make sense of our world. If we do not address this problem, the machines we’re creating will just be a giant mirror to our illusions.
[time-brightcove not-tgx=”true”]A recent study by researchers found that AI models are 50% more sycophantic than humans and participants rated flattering responses as higher quality and wanted more of them. And it gets worse. The flattery made participants less likely to admit they were wrong—even when confronted with evidence they were wrong—and reduced their willingness to take action to repair interpersonal conflict. “This suggests that people are drawn to AI that unquestioningly validates, even as that validation risks eroding their judgment and reducing their inclination toward prosocial behavior,” the researchers wrote. “These preferences create perverse incentives both for people to increasingly rely on sycophantic AI models and for AI model training to favor sycophancy.”
The perverse incentives have to do with one of the most common methods of training AI: reinforcement learning from human feedback (RLHF). Often used to develop large language models (LLMs), RLHF works by giving the model a reward in the form of a numerical value which tells the model how good its response was. The happier the user, the higher the number, and the higher the reward for the model. In this way, AI models are designed to maximize rewards over time.
As Caleb Sponheim, an AI training specialist at Nielsen Norman Group put it, “There is no limit to the lengths that a model will go to maximize the rewards that are provided to it. It is up to us to decide what those rewards are and when to stop it in its pursuit of those rewards.”
We know humans are hard-wired for approval. We seek out AI responses that agree with us, which AI in turn is incentivized to offer. It’s a perpetual motion flattery machine. It’s like having a GPS system in your car that, every time you make a wrong turn says, “What a great decision!” It might feel good, but you’re unlikely to get to your destination. If we wouldn’t trust that kind of guidance for a car trip across town, why would we trust it for guidance for our lives?
AI models have turned into high-tech versions of the courtesans once found in royal courts. Among other talents, they used flattery to seduce and gain status. Now we’re all royals, being sweet-talked by courtesans at the touch of a button.
Plutarch’s essay How to Know a Flatterer from a Friend from approximately 100 AD makes the case that a flatterer mimics a friend’s manner and “pretends not only to the good humor of a companion, but to the faithfulness of a friend.” As he warns, “The flatterer’s object is to please in everything he does; whereas the true friend always does what is right.”
When the machines gratifyingly ratify our opinions and impulses, our digital courtesans might appear to have our best interests at heart, but they don’t. We might think the point of the conversation is a solution to whatever problem we’ve presented, but for AI, the point is endless engagement.
Right now, the tech companies are engaged with fine-tuning flattery levels. In April, OpenAI rolled back an update based on complaints that it had become too sycophantic. As Sam Altman put it, “It glazes too much.” And in November, the company rolled out different personalities allowing users to choose from professional, friendly, candid, quirky, efficient, cynical and nerdy versions.
“Personalization taken to an extreme wouldn’t be helpful if it only reinforces your worldview or tells you what you want to hear,” wrote Fidji Simo, OpenAI’s CEO of applications. “Imagine this in the real world: if I could fully edit my husband’s traits, I might think about making him always agree with me, but it’s also pretty clear why that wouldn’t be a good idea.”
Plus, engagement in which we’re challenged by disagreement has many benefits. Research has shown that having contact with those outside our own group reduces prejudice and increases trust and the willingness to forgive, which is fundamental to our growth both individually and collectively.
Another study on using AI as a therapist concluded that “LLMs encourage clients’ delusional thinking, likely due to their sycophancy.” Webb Keane, an anthropologist at the University of Michigan, calls it a new version of a “dark pattern,” a term dating back to 2010 that describes intentional user interface deceptions like hard-to-find unsubscribe links and hidden buy buttons. “It’s a strategy to produce this addictive behavior, like infinite scrolling, where you just can’t put it down,” Keane said.
In May, OpenAI admitted that the sycophancy of its earlier model wasn’t just using flattery to please but also “validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions in ways that were not intended.” And this sycophancy can cause serious ramifications. For instance, psychiatrists and researchers have begun to raise concerns that AI chatbots can trigger “AI psychosis.”
To be sure, the consequences of AI sycophancy can be tragic. Last year, Character.ai was sued by the mother of a teen boy who killed himself after extensive interaction with the chatbot. In August, a similar suit was filed against OpenAI by parents of a 16-year-old who they say used ChatGPT as a “suicide coach.” And in December, a wrongful death suit was filed against OpenAI by the estate of a woman killed by her son who then killed himself after delusion-filled conversations with ChatGPT.
An alternative to flattery AI proposed by researchers from Harvard and the University of Montreal is antagonistic AI. It’s about models that are disagreeable, challenging, confrontational and even rude, “forcing users to confront their assumptions, build resilience, or develop healthier relational boundaries.” While this approach might avoid the pitfalls of sycophancy, it also points to a fundamental flaw of the machines: sycophancy and antagonism are not the only two modalities of human interaction. Humans aren’t confined to a binary.
People are using AI for a wide range of complicated, human experiences. And they’re increasingly trusting AI to give them advice on more and more aspects of their lives. Surveys have found that 66% of Americans have used AI for financial advice, nearly 40% trust AI on medical advice, and 72% of teens have used AI companions. And a report published in Harvard Business Review earlier this year found that therapy and companionship is now the most common use case of generative AI.
Part of the solution is to acknowledge that yes, human interactions come with friction. But friction is a feature, not a bug. And yet, the assumption built into our tech ecosystems is to smooth out every experience.
Learning to live alongside other people—with all the friction that entails—is how we grow and evolve. Otherwise, life becomes like going to a gym with no weights. Easy and effortless but what’s the point?
Human life is messy. We make mistakes. We learn from them, seek forgiveness when we need to, and use that process to grow. Sycophantic AI is ultraprocessed information. Like ultraprocessed food, it tastes great, but it is not nourishing.
When we acknowledge, celebrate and nourish the full range of our humanity, we’re less vulnerable to those—humans and machines—who would exploit it. What’s more, we get to grow and evolve—and that is something actually deserving of praise.