Back to News

Cloudflare will now, by default, block AI bots from crawling its clients’ websites

MIT Technology Review - AI
Jul 1, 2025 10:00
Peter Hall
1 views
airesearchtechnology

Summary

Cloudflare has announced it will now automatically block AI bots from crawling websites it hosts, though clients can choose to allow or ban specific bots. The company is also introducing a "pay-per-crawl" service, enabling website owners to receive compensation from AI companies that wish to access their content. This move could significantly impact how AI models gather training data and may lead to new norms around data access and compensation in the AI field.

The internet infrastructure company Cloudflare announced today that it will now default to blocking AI bots from visiting websites it hosts. Cloudflare will also give clients the ability to manually allow or ban these AI bots on a case-by-case basis, and it will introduce a so-called “pay-per-crawl” service that clients can use to receive compensation…
Read Full ArticleMore News

Related Articles

4 Top Crypto Coins Poised For Big Gains: BlockDAG, RNDR, XLM and CRO

Analytics InsightJul 4

The article highlights four cryptocurrencies—BlockDAG, RNDR, XLM, and CRO—as having strong growth potential, with particular emphasis on RNDR (Render Token) due to its role in powering decentralized GPU rendering for AI and graphics applications. RNDR's integration into AI workflows is seen as a driver for increased demand and value. This trend underscores the growing intersection of blockchain and AI, where decentralized computing resources support AI development and deployment.

How to Use ChatGPT: Supercharge Your Productivity

Analytics InsightJul 4

The article outlines practical ways to use ChatGPT to boost productivity, such as drafting emails, brainstorming ideas, and summarizing documents. It highlights how AI tools like ChatGPT can streamline everyday tasks, making them valuable assets for both individuals and businesses. The widespread adoption of such AI assistants signals a shift toward more efficient, AI-augmented workflows in various professional fields.

The environmental impact of AI is only half the story

Hacker News - AIJul 4

The article argues that while concerns about AI's environmental impact—such as energy consumption and carbon emissions—are important, they represent only part of the issue. It emphasizes that broader societal effects, including AI's influence on power dynamics, labor, and decision-making, are equally critical and must be addressed by the AI field.