What if the real risk of AI isn’t deepfakes — but daily whispers?
Most people don’t appreciate the profound threat that AI will soon pose to human agency. A common refrain is that “AI is just a tool,” and like any tool, its benefits and dangers depend on how people use it. This is old-school thinking. AI is transitioning from tools we use to prosthetics we wear. This will create significant new threats we’re just not prepared for.
No, I’m not talking about creepy brain implants. These AI-powered prosthetics will be mainstream products we buy from Amazon or the Apple Store and marketed with friendly names like “assistants,” “coaches,” “co-pilots” and “tutors.” They will provide real value in our lives — so much so that we will feel disadvantaged if others are wearing them and we are not. This will create rapid pressure for mass adoption.
The prosthetic devices I’m referring to are “AI-powered wearables” like smart glasses, pendants, pins and earbuds. Your wearable AI will see what you see and hear what you hear, all while tracking where you are, what you’re doing, who you’re with and what you are trying to achieve. Then, without you needing to say a word, these mental aids will whisper advice into your ears or flash guidance before your eyes.
The difference between a tool and a prosthetic may seem subtle, but the implications for human agency are profound. This is best understood through a simple analysis of input and output. A tool takes in human input and generates amplified output. A tool can make us stronger, faster or allow us to fly. A mental prosthetic, on the other hand, forms a feedback loop around the human, accepting input from the user (by tracking their actions and engaging them in conversation) and generating output that can immediately influence the user’s thinking.
This feedback loop changes everything. That’s because body-worn AI devices will be able to monitor our behaviors and emotions and could use this data to talk us into believing things that are untrue, buying things we don’t need or adopting views we’d otherwise realize are not in our best interest. This is called the AI Manipulation Problem, and we are not ready for the risks. This is an urgent issue because big tech is racing to bring these products to market.
Why are feedback loops so dangerous?
In today’s world, all computing devices are used to deploy targeted influence on behalf of paying sponsors. Wearable AI products will likely continue this trend. The problem is, these devices could easily be given an “influence objective” and be tasked with optimizing their impact on the user, adapting their conversational tactics to overcome any resistance they detect. This transforms the concept of targeted influence from social media buckshot into heat-seeking missiles that skillfully navigate past your defenses. And yet, policymakers don’t appreciate this risk.
Unfortunately, most regulators still view the danger of AI in terms of its ability to rapidly generate traditional forms of influence (deepfakes, fake news, propaganda). Of course, these are significant threats, but they’re not nearly as dangerous as the interactive and adaptive influence that could soon be widely deployed through conversational agents, especially when those AI agents travel with us through our lives inside wearable devices.
This is coming soon
Meta, Google and Apple are racing to launch wearable AI products as quickly as they can. To protect the public, policymakers need to abandon their “tool-use” framing when regulating AI devices. This is difficult because the tool-use metaphor goes back 35 years to when Steve Jobs colorfully described the PC as a “bicycle of the mind.” A bicycle is a powerful tool that keeps the rider firmly in control. Wearable AI will flip this metaphor on its head, making us wonder who is steering the bicycle — the human, the AI agents whispering in the human’s ears, or the corporations that deployed the agents? I believe it will be a dangerous mix of all three.
In addition, users will likely trust the AI-voices in their heads more than they should. That’s because these AI agents will provide us with useful advice and information throughout our daily life — educating us, reminding us, coaching us, informing us. The problem is, we may not be able to distinguish when the AI agent has shifted its objective from assisting us to influencing us. To appreciate the difference, you might watch the award-winning short film Privacy Lost (2023) about the dangers of AI-powered wearable devices. This is especially true when devices include invasive features such as facial recognition (which Meta is reportedly adding to their glasses).
What can we do to protect the public?
First and foremost, policymakers need to realize that conversational AI enables an entirely new form of media that is interactive, adaptive, individualized and increasingly context-aware. This new form of media will function as “active influence,” because it can adjust its tactics in real time to overcome user resistance. When deployed in wearable devices, these AI systems could be designed to manipulate our actions, sway our opinions and influence our beliefs — and do it all through seemingly casual dialog. Worse, these agents will learn over time what conversational tactics work best on each of us on a personal level.
The fact is, conversational agents should not be allowed to form control loops around users. If this is not regulated, AI will be able to influence us with superhuman persuasiveness. In addition, AI agents should be required to inform users whenever they transition to expressing promotional content on behalf of a third party. Without such protections, AI agents will likely become so persuasive that they will make today’s targeted influence techniques look quaint.
Louis Rosenberg is a pioneer of augmented reality and a longtime AI researcher. He earned his PhD from Stanford, was a professor at California State University, and authored several books on the dangers of AI, including Arrival Mind and Our Next Reality.