Cloudflare tightens its course against unauthorized access by AI crawlers. In the future, known bots that collect content from websites for training purposes will be blocked by default. New customers will be actively asked whether they want to allow such access at all. The goal is to put control over digital content back into the hands of the creators – and to create monetization opportunities at the same time.

With the newly introduced "Pay Per Crawl" model, selected publishers can set their own prices for AI system access. AI providers have the option to view these prices, register for paid usage, or refrain from accessing. The offer is initially only available to an exclusive group of leading media houses and content creators.

Cloudflare responds to the growing importance of generative AI applications and large language models, which increasingly automate reading content on the web. As early as 2023, the company introduced initial blocking options based on the voluntary guidelines of robots.txt. Later, the feature was expanded to block all known AI bots regardless of their compliance. What's new is that these blocks are now active by default for all new domains at Cloudflare.

Additionally, since March, Cloudflare has been offering a technical barrier called "AI Labyrinth," which deliberately leads unwanted crawlers into infinite loops to complicate scraping activities.

Supporters of the new guidelines include major publishers such as The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora. They are all united by the concern that generative AI models increasingly use original content without obtaining the appropriate rights or providing compensation. "People now trust AI systems more than search engines. This leads to them reading fewer original sources," explained Cloudflare CEO Matthew Prince.

Furthermore, Cloudflare is working on a verification model for AI crawlers. These will in the future openly indicate whether content is used for training, inference, or search. Website operators can then individually decide which access they allow. "Original content makes the internet one of the greatest inventions of the last century," Prince continues. "Our goal is to protect the rights of creators without stifling innovation.