Cloudflare and GoDaddy team up to help websites fend off Big Tech's AI bot swarm

• Cloudflare and GoDaddy are addressing AI's impact on web traffic and revenue.
• The firms implemented an AI Crawl Control to manage AI bot interactions on websites.
• The partnership aims to create a permission-based system for AI-driven web content.

Cloudflare and GoDaddy are teaming up to tackle a growing problem at the heart of the internet: the collapse of the web's traditional business model in the age of AI.

A new partnership between the two companies aims to address this imbalance.

GoDaddy will integrate Cloudflare's AI Crawl Control into its web hosting site, allowing site owners to control how AI bots interact with their content. They can give AI bot crawlers access, block them, or potentially charge for access. These bots, run by giant AI companies, crawl the web, sucking up content for free to power chatbot answers and other AI model outputs.

For decades, websites have relied on a simple system. Search engines index content, users click links, and publishers earn money through ads, subscriptions, or sales. But that model is breaking down as AI answer engines increasingly deliver information directly to users, often without sending them back to the original source.

This shift has created tension between content creators and AI companies. Automated crawlers, run by Anthropic, OpenAI, Google, and other AI companies, now scan vast portions of the web to generate outputs, but they return far less traffic than traditional search engines. For many publishers and small businesses, that means declining revenue despite their content being widely used.

Cloudflare manages and protects traffic for about 20% of the web, while GoDaddy is the world's largest domain name registrar, helping millions of people name and manage websites. In combination, these companies have the power to change how business is done online.

The companies are also backing new standards, including Agent Name Service and Web Bot Auth, to verify the identity of AI agents and make their behavior more transparent.

The broader goal is to create a permission-based system for the AI-driven web. Instead of bots freely scraping content, creators would have clearer control and visibility — and a way to participate in the value created by AI systems.

This builds on Cloudflare's earlier moves to block AI crawlers by default and introduce tools that let publishers monetize access to their data.

The stakes are high. If AI systems continue to extract value without sending traffic back, the incentive to create original content could erode.

Cloudflare and GoDaddy are betting that new infrastructure, focused on identity, control, and compensation, can help rebuild a sustainable economic model for the next phase of the internet.

Sign up for BI's Tech Memo newsletter here. Reach out to me via email at abarr@businessinsider.com.