Publishers are stepping up efforts to protect their websites from tech companies that hoover up content for new AI tools. The media companies have sued, forged licensing deals to be compensated for the use of their material, or both. Many asked nicely for artificial-intelligence bots to stop scraping. Now, they are working to block crawlers from their sites altogether. “You want humans reading your site, not bots, particularly bots that aren’t returning any value to you,” said Nicholas Thompson, the chief executive of the Atlantic. Scraping is nearly as old as the web itself. But the web has changed significantly since the 1990s, when Google was a scrappy startup. Back then, there were benefits to letting Google crawl freely: sites that were scraped would pop up in search results, driving traffic and ad revenue. A new crop of AI-fueled chatbots, from ChatGPT to Google’s Gemini, now deliver succinct answers using troves of data taken from the open web, eliminating the need for many users to visit websites at all. Search traffic has dropped precipitously for many publishers, who are bracing for further hits after Google began rolling out AI Mode, which responds to user queries with far fewer links than a traditional search. Scraping activity has jumped 18% in the past year, according to Cloudflare, an internet services company.

Full report : Websites are building fences around their content in an effort to cut off AI bot crawlers that don’t pay for content.