The major internet Content Delivery Network (CDN), Cloudflare, has declared war on AI companies. Starting July 1, Cloudflare now blocks by default AI web crawlers accessing content from your websites without permission or compensation. The change addresses a real problem. My own small site, where I track all my stories, Practical Technology, has been slowed dramatically at times by AI crawlers. It’s not just me. Numerous website owners have reported that AI crawlers, such as OpenAI’s GPTBot and Anthropic’s ClaudeBot, generate massive volumes of automated requests that clog up websites so they’re as slow as sludge. GoogleBot alone reports that the cloud-hosting service Vercel bombards the sites it hosts with over 4.5 billion requests a month. These AI bots often crawl sites far more aggressively than traditional search engine crawlers. They sometimes revisit the same pages every few hours or even hit sites with hundreds of requests per second. While the AI companies deny that their bots are to blame, the evidence tells a different story. Thus, on behalf of its two million-plus customers, 20% of the web, Cloudflare now blocks AI crawlers. For any new website signing up for its services, AI crawlers will be automatically blocked from accessing its content unless the site owner grants explicit permission. Additionally, Cloudflare promises to detect “shadow” scrapers — bots that attempt to evade detection — by using behavioral analysis and machine learning. What’s good for the AI goose is good for the gander.

Full report : The era of unrestricted AI crawling appears to be ending – well, at least for the fifth of the internet that flows through Cloudflare’s pipes.