Cloudflare launches a way to charge AI bots for crawling sites

Cloudflare is launching a bold new experiment to address the growing tension between content creators and artificial intelligence companies. As the internet shifts toward AI-powered tools, publishers have struggled to maintain control over their data. Cloudflare, which serves 20% of the web, believes the answer lies in creating a transactional model for AI crawling. On Tuesday, the company unveiled a marketplace that lets publishers charge for access to their content—potentially changing how web data is monetized.

A New Marketplace for the AI Era

Cloudflare’s new initiative is called Pay per Crawl. The company is testing it in a private beta phase. Website owners who join the experiment can choose to let individual AI crawlers access their site at a micropayment rate. They can also decide to allow free crawling or block access entirely. The system gives publishers flexibility while granting visibility into who is crawling and for what purpose—whether it’s AI model training, AI search response generation, or other uses.

Cloudflare stated that these tools build on earlier efforts it launched in 2023 and 2024. These included a one-click solution to block AI bots and a dashboard that tracks crawler activity. CEO Matthew Prince explained to TechCrunch in 2024 that these were early steps toward building a new kind of content distribution system. Now, that vision is starting to materialize.

Shifting the Balance of Power

For decades, the relationship between publishers and platforms like Google was based on traffic exchange. Google scraped content, returned it in search results, and sent users to publisher websites. That flow created advertising revenue. But with the rise of AI agents that summarize content directly, that balance is deteriorating.

Cloudflare data from June 2025 paints a stark picture. Google’s crawler scraped sites 14 times for every user referral it provided. OpenAI’s crawler scraped 1,700 times for each referral. Anthropic’s crawler did so 73,000 times. Cloudflare presented these ratios as evidence that the AI ecosystem is extracting far more value from publishers than it returns.

Publishers are already reacting. Many major outlets—including TIME, The Atlantic, ADWEEK, Conde Nast, The Associated Press, and Fortune—have adopted Cloudflare’s default setting to block all AI crawlers. Cloudflare now configures new websites it hosts to block AI bots automatically unless granted permission. The company describes this as a “default of control.”

Publisher Challenges in the Age of AI

Not all publishers are in a position to negotiate with AI companies. A handful of major outlets have signed licensing agreements with firms like OpenAI and Google. Others, such as The New York Times, have filed lawsuits against tech companies, alleging unauthorized use of copyrighted material to train models.

But smaller websites and independent publishers lack leverage. Without direct deals, they risk being bypassed entirely as chatbots become a primary source of information. Cloudflare hopes that Pay per Crawl can level the playing field. By enabling even small publishers to set crawl rates, the marketplace introduces a structure where AI firms pay for what they use.

This marks a fundamental shift. Instead of content being passively harvested, it becomes part of an active, priced exchange. Cloudflare positions itself as the broker—facilitating payments and enforcing permissions.

Technical Simplicity, Strategic Implications

The implementation is relatively straightforward. Both AI companies and publishers need Cloudflare accounts. Each party sets rates for crawling. Cloudflare then acts as the intermediary—charging the AI firm per crawl and forwarding earnings to the publisher.

Spokesperson Ripley Park clarified to TechCrunch that the system does not involve cryptocurrencies or stablecoins, despite speculation that digital payments would suit the model. All transactions occur within traditional financial systems.

The model is remain experimental. It should make everyone onboard so that it can succeed. Though some of the publishers will be willing to join it, the majority of the AI companies have been accustomed to using it without payment and will be happy not to have to pay. The issue here is that Cloudflare will have to invite as many stakeholders as they will be able to find to create a viable ecosystem.

Preparing for an “Agentic” Internet Future

Cloudflare emphasized that the full value of Pay per Crawl may emerge as AI agents become more prevalent. These agents—software bots that perform tasks on behalf of users—are already being built by companies like Google and OpenAI. They visit websites, gather data, and deliver synthesized results without sending users to the original source.

In this context, Cloudflare imagines a future where agents are given a budget. That budget is then used to pay for content access dynamically. In a blog post, the company suggested this could apply to tasks ranging from compiling legal research to selecting restaurants.

Cloudflare envisions the entire process occurring at the “network edge,” meaning it can be automated and fast. Publishers could be paid instantly for each interaction—without subscriptions, ads, or complex licensing agreements.

An Uncertain but Necessary Shift

The digital publishing industry faces deep structural challenges. Traffic from traditional search is declining. Advertising revenues are unreliable. AI chatbots are changing how users find and consume information. Cloudflare’s marketplace does not solve every problem, but it proposes a new model—one based on consent, pricing, and traceability.

There’s no assurance that this model will succeed. AI companies may prefer to continue scraping freely. Smaller publishers may find the marketplace too complex or unprofitable. And pricing crawl access fairly remains a challenge.

Nevertheless, Cloudflare is one of the companies that have sufficient infrastructure access and standing to make an effort at this. Reaching a fifth of the web it can in a unique position enforce permissions and can handle micropayments at scale. The relocation can act as the roadmap of how the internet may operate in the era of AI dominance.

Whether this marketplace achieves critical mass will depend on adoption from both sides—but it’s a step toward rebalancing power in an increasingly agent-driven web.