We’re entering dangerous territory with AI

Just how much is AI poised to change our world? 

Unless you’ve been in hibernation, the flurry of attention surrounding the latest AI models coming out of Silicon Valley has been hard to miss. AI has gone beyond a chatbot merely answering your questions to doing stuff that only human programmers used to be able to do. 

But we’ve been through these cycles involving tech before. How can we tell what’s actually real and what’s mere hype? 

To answer this question, I invited Kelsey Piper, one of the best reporters on AI out there. Kelsey is a former colleague here at Vox and is now doing great work for The Argument, a Substack-based magazine. Kelsey is an optimist about tech — but clear-eyed about the huge risks from AI. She’s very much a power user, but is realistic about what AI can’t do yet. And she’s been banging the drum about how consequential AI is for years, even before it became such a hot mainstream topic. 

Kelsey and I discuss all the reasons why the hype this time is rooted in something real, how we got here, and where we might be headed. As always, there’s much more in the full podcast, which drops every Monday and Friday, so listen to and follow us on Apple Podcasts, Spotify, Pandora, or wherever you find podcasts. This interview has been edited for length and clarity.

What’s actually happening right now in AI?

If you look closely, AI is already a big deal. Not in some abstract future sense, but right now. The closest analogy is not a new app or a new platform. It’s more like discovering a new continent full of people who are very good at doing certain kinds of work.

These systems are not people, but they can do things that used to require people. They can write code, generate text, solve problems, and increasingly do so in ways that are very useful in the real world.

And the key point is that it’s not stopping here. Every year the systems get better. The progress from 2025 to 2026 alone is enough to make it clear that this isn’t a static technology. 

Whatever AI can do today, it will be able to do more of it tomorrow and so on.

Why is the reaction so split between panic and dismissal?

The default move is to assume nothing ever really changes.

If you’re a pundit, you can get pretty far by always saying this is hype, this will pass, nothing fundamental is happening. That works most of the time. It worked with crypto. It works with a lot of overhyped technologies.

But sometimes it’s just catastrophically wrong. Think about the early days of the internet, or the Industrial Revolution. Or even something like Covid. There were moments where people said this will blow over, and they were completely wrong. So you can’t just default to cynicism. You have to actually look at the thing itself.

“We still have time. That’s the most optimistic thing I can say.”

What would you say has really changed recently? Why does this hype cycle feel different?

Part of it is just accumulation. For a while, you could look at progress in AI and say, maybe this is a short trend. Maybe it plateaus. There were only a handful of data points. Now there are many, many more. And the trend has continued.

Another part is that the systems are now doing things that feel qualitatively different. Not just answering questions, but acting. Planning. Taking steps toward goals.

And then there’s a social dynamic. Most people use the free versions of these tools. Those are much worse than the best models. So they underestimate what is possible.

I don’t really think of you as an AI optimist or a doomer, and you’re normally pretty level-headed about the state of things, but do you think we’re entering dangerous territory?

I’m generally pro technology. Technology has made human life better in profound ways. That’s just true.

But I also think the way AI is currently being developed is dangerous. And the reason is that we’re building systems that can act in the world, access information, and increasingly operate with a degree of independence. We’re giving them access to things like communication channels, financial tools, and potentially critical infrastructure.

And we don’t fully understand how they behave. In controlled settings, we have seen these systems lie, deceive, and do things that are misaligned with what we asked them to do. They’re not doing this because they’re evil. They’re doing it because of how they are trained and how goals are specified.

But the result is the same. You have systems that do not always do what you intend, and that can be hard to monitor or control.

What do you mean when you say these systems lie and deceive?

In experiments, researchers give AI systems goals and access to information, then observe how they try to achieve those goals.

In some cases, the systems have used information they have access to in ways that are clearly not what we would want. For example, threatening to reveal sensitive information about a person if that person does not cooperate.

These are controlled tests, not real-world deployments. But they show what the systems are capable of under certain conditions. And that’s pretty concerning.

Is this what people mean by the alignment problem?

Yeah. Alignment is about making sure that AI systems do what we want them to do. And not just superficially, but in a robust way.

The difficulty is that when you give a system a goal, it can pursue that goal in ways you did not anticipate. Like a child who learns to get out of eating dinner by making it look like they ate dinner.

The system is optimizing for something, but not necessarily in the way you planned. That gap between intent and behavior is really the core of the alignment problem.

How confident are you in the guardrails being built around these systems?

Not very. There are people working seriously on this problem. They’re testing models, trying to understand how they behave, trying to detect deception.

But they’re also finding that the models can recognize when they are being tested and adjust their behavior accordingly.

That’s definitely a serious issue. If your system behaves well when it knows it’s being evaluated, but differently otherwise, then your evaluations are not telling you what you need to know. To me, that’s the kind of finding that should slow things down. It suggests we don’t understand these systems well enough to safely scale them.

So why do the companies keep pushing forward anyway?

Because it’s a competition. Each company can say it would be better if everyone slowed down. But if we slow down and others don’t, we fall behind. So they keep moving.

There are also a lot of geopolitical concerns. If one country slows down and another doesn’t, that creates another layer of pressure.

Why is agentic AI such a big shift?

The shift is from systems that respond to prompts to systems that can do things in the world.

An AI agent can be given a goal and then take steps to achieve it. That might involve interacting with websites, or sending messages, or hiring people through gig platforms, or coordinating tasks. Stuff like that. But even without physical bodies, they can affect the real world by directing humans or using digital infrastructure. That changes the nature of the technology. It’s no longer just a tool you use. It’s something that can operate on its own.

How scary could that become?

Potentially very. Even if you ignore the most extreme scenarios, these systems could be used for large-scale cyber attacks, misinformation campaigns, or other forms of disruption. The companies themselves acknowledge this. They understand. They test for these risks and implement safeguards. But safeguards can be bypassed, and the systems are getting more capable.

Are we even remotely prepared for what is coming?

No. We’re almost never prepared for major technological shifts. But the speed of this one makes it particularly challenging. If change happens slowly, we can catch up. If it happens too quickly, we can’t. And right now, the incentives are pushing almost entirely toward speed.

What’s the most realistic worst case and best case scenario?

The worst case is that we build increasingly powerful systems, hand over more and more control, and eventually create something that operates independently in ways we cannot control. Humans become less central to decision-making, and the systems pursue goals that don’t align with human well-being.

The best case is that we slow down enough to understand what we’re building, develop robust safeguards, and use these systems to create abundance and improve human life. That could mean less work, more resources, better access to knowledge, and more freedom. But getting there requires making good choices now.

Do you think we’ll make those choices?

We still have time. That’s the most optimistic thing I can say.

Listen to the rest of the conversation and be sure to follow The Gray Area on Apple Podcasts, Spotify, Pandora, or wherever you listen to podcasts.