The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them from “accessing content without permission or compensation,” according to an announcement on Tuesday. With the change, Cloudflare will start asking new domain owners whether they want to allow AI scrapers, and will even let some publishers implement a “Pay Per Crawl” fee.
The Pay Per Crawl program will let publishers set a price for AI scrapers to access their content. AI companies can then view pricing and choose whether to register for the “Pay Per Crawl” fee or turn away. This is only available for “a group of some of the leading publishers and content creators” for now, but Cloudflare says it will ensure “AI companies can use quality content the right way — with permission and compensation.”
Cloudflare has been helping domain owners fight AI crawlers for a while now. The company started letting websites block AI crawlers in 2023, but it only applied to ones that abide by a site’s robots.txt file, the unenforceable agreement that signals whether bots can scrape its content. Cloudflare began allowing websites to block “all” AI bots last year — whether they respect a site’s robots.txt file or not — and now this setting is enabled by default for new Cloudflare customers. (The company identifies scrapers to block by comparing them to its list of known AI bots.) Cloudflare also rolled out a feature in March that sends web-crawling bots into an “AI Labyrinth” to deter them from scraping sites without permission.
Several major publishers and online platforms, including The Associated Press, The Atlantic, FortuneStack Overflow, and Quora, are on board with Cloudflare’s new AI crawler restrictions, as websites contend with a future where more people are finding information through AI chatbots, rather than search engines. “People trust the AI more over the last six months, which means they’re not reading original content,” Cloudflare CEO Matthew Prince said during the Axios Live event last week.
Additionally, Cloudflare says it’s working with AI companies to help verify their crawlers and allow them to “clearly state their purpose,” such as whether they’re using the content for training, inference, or search. Website owners can then review this information and determine which crawlers to let in.
“Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it,” Prince said in the press release. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.”
About The Author
