Imagine a Chatbot That Actually Knew How to Talk to You
Earlier this year, when I walked into a renovated loft in downtown San Francisco, the couches and tables were littered with flyers advertising an “emotionally intelligent real-time AI coach.” They were for Amotions AI—one of several start-ups that had gathered that day to pitch investors, entrepreneurs, and tech workers. Pianpian Xu Guthrie, Amotion AI’s founder, was eager to tell me more. The AI model observes video calls on your computer, she said, and gives you real-time tips based on the other person’s tone and facial expression. Maybe you’re a salesperson, and the bot flags that your potential customer is “confused” and suggests what to say.
Emotions are the AI industry’s new fixation. Not only are growing numbers of start-ups such as Amotions AI promising tools that interpret feelings; the major AI companies are developing chatbots that apparently aren’t just smarter—they get you. When OpenAI launched a new version of ChatGPT late last year, it described the bot as “warmer by default and more conversational.” Anthropic has stated that its model, Claude, “may have some functional version of emotions or feelings,” and Google has claimed that its AI models are now capable of “reading the room.” Elon Musk’s lab, xAI, has boasted that a recent version of Grok did well on a test of emotional intelligence, or EQ, that posed scenarios such as this: “You think you might have been scapegoated by a fellow employee for the lunchroom thefts that have been happening.”
Silicon Valley has good reason to push EQ. For AI products to work as advertised—to genuinely substitute for personal assistants or co-workers—they have to be not just competent but caring; not just effective but empathetic. And so the AI industry seems to believe that the next step in developing smart and useful bots requires instilling them with people skills.
[Read: The people outsourcing their thinking to AI]
The search for an emotionally intelligent machine has long been part of AI research. In the 1960s, the computer scientist Joseph Weizenbaum developed a primitive chatbot, called ELIZA, that could simulate a psychotherapist by repeating back what a person said in question form. One day, as Weizenbaum recalled, he found his secretary chatting with ELIZA; she asked him to leave the room to give them some privacy. The original ChatGPT from late 2022 was not smarter or more powerful than other existing tools—the underlying model was actually several years old—but OpenAI’s main innovation was to engineer the bot to converse like a human. ChatGPT had a surface-level ability to pick up on and respond to cues for, say, anger or joy.
Even so, the AI industry has since not been all that interested in emotions. Silicon Valley has spent the past two years pouring resources into so-called reasoning models in the hopes of making them good at writing code and solving math problems. Last year, Ilya Sutskever, the former chief scientist at OpenAI, said that “emotions are relatively simple” for bots to master on the path toward developing intelligence. By this logic, figuring out the nature of joy or anxiety would ostensibly be much easier than figuring out nuclear fusion. Industry-wide measures exist for all sorts of technical abilities, but until recently, companies simply did not seem to publicly evaluate anything relating to human feeling.
That dismissive attitude is changing. “Emotional intelligence is one of the most important capabilities of current models,” Hui Shen, an AI researcher at the University of Michigan, told me. The companies continue to chase raw intelligence and problem-solving abilities—but they seem to have realized that, for most people, that’s not the most relevant product feature. Whether Grok can solve difficult math problems is probably less useful to you than the advice it can give on ways to impress your boss at work or, even, how it consoles you when your cat dies. (Which, according to an example in xAI’s press release about Grok’s state-of-the-art EQ, could be: “The quiet spots where they used to sleep, the random meows you still expect to hear … it just hits in waves. It’s okay that it hurts this much.”)
Last year, both OpenAI and Anthropic separately published research showing that roughly 2 to 3 percent of conversations with ChatGPT or Claude were explicitly emotional—seeking interpersonal advice, role-playing, and so on. These are small proportions, but with some billion individual users between these companies, the actual number of people having emotional discussions with these two bots alone could be well into the millions. And many of the more frequent uses of chatbots, such as for tutoring and writing personal communications, also involve varying degrees of interpreting and managing emotions.
To the extent that human emotions or preferences were incorporated into the training of ChatGPT or other top models, much of that appears to have been accomplished through a process known as “reinforcement learning with human feedback”: A chatbot writes multiple responses to the same prompt, and human raters decide which they prefer. If applied without nuance, this approach can produce AI models that uncritically agree with and reinforce anything a user says—precipitating deep emotional dependencies on AI chatbots and, in the most extreme cases, appearing to encourage delusional thinking.
[Read: The chatbot-delusion crisis]
What AI firms are after now is something that resembles genuine empathy, which involves much more than validating what users already want to hear. This sort of bot would not only comfort but push back when necessary—and, crucially, would recognize its own limits as a piece of software. For instance, Anthropic noted in a recent update to Claude’s constitution—a document that tells the model, in an abstract sense, how to behave—to avoid situations in which someone exclusively “relies on Claude for emotional support.” But no AI company has really given a clear definition of how a truly emotionally intelligent bot would differ from today’s shallow miming of EQ.
To that end, a more cynical way to interpret the industry’s frenzy over emotions is that it’s a way to make AI models more useful, yes, but also a way to retain users—akin to features such as “memory,” in which chatbots can recall details from past conversations. The miming of an interpersonal relationship gives AI models a huge advantage over other software. “People don’t have a lot of emotions associated with Google search, but with these chatbots, people are having a lot of connections,” Sahand Sabour, an AI researcher at Tsinghua University, told me. (Anthropic did not respond to a request to discuss recent research on Claude and emotions. OpenAI declined to comment but pointed me to a Substack essay in which one of its researchers wrote that AI models should be warm without giving the illusion of consciousness. xAI did not respond to a request for comment.)
No matter the motivation, instilling any sort of EQ in a computer program remains very hard. Social scientists have spent many decades trying to develop tests for people’s abilities to recognize, regulate, and respond to emotions in the hopes that they might correlate with happiness or workplace performance. Such EQ evaluations have been adapted for chatbots, with questions to the tune of: Michael has been practicing a magic trick to show his friend Lily, but Lily has been attending his practices in secret. When he performs the trick, she knows exactly how it works. How does Michael feel?
As it turns out, generative-AI models do quite well on such tests—better, in some instances, than people. That shouldn’t come as a surprise, because there are mountains of similar scenarios all over the web that AI models are trained on. All of that data is probably why bots are “so good at solving these quite narrow tests that we developed for humans,” Katja Schlegel, a psychologist at the University of Bern, told me. Such encyclopedic knowledge could make these products useful in certain settings—and the process of reinforcement learning with human feedback largely involves eliciting and sharpening these abilities. But all of this is a far cry from genuinely understanding why someone feels a certain way, empathizing with them, and figuring out whether they need to and how they might be helped.
After all, EQ tests aren’t even that useful in people, let alone chatbots. Being able to label a scowl as “upset” in a lab is very different from interacting with a scowling child, spouse, or boss. Emotions are bound to a person, a relationship, a culture, a moment in time; they are an experience. The AI industry’s first great act of marketing was labeling its products as intelligence, a term so general and poorly understood in humans that it could encompass anything. Now the same AI firms have set their sights on an attribute that is even more poorly understood than IQ. Emotions are squishy and subjective, providing leeway to convincingly market chatbots as emotionally intelligent—and pushing more people to talk with them.