Cloudflare shuts down AI Crawler for unauthorized data extraction!

Cloudflare, Inc. (NYSE: NET), a leading connectivity cloud company, today announced that Cloudflare is the first Internet infrastructure company to block AI crawlers that access content without permission or payment by default. Going forward, website owners will be able to choose which AI crawlers they want to access their content, and decide how those AI companies use their data. AI companies can now clearly define how their AI crawlers will use their datafor training, inference, or searchto help website owners decide what data they allow their AI crawlers to access. By making this the default setting for Cloudflare, we are taking the first step towards creating a more sustainable future for both content creators and AI creators.

For decades, the Internet has operated on a simple principle of exchange: search engines index content and direct users to the original site, generating traffic and advertising revenue for websites of all sizes. This cycle rewards quality content creators with revenue and followers while helping users discover new, relevant information. But this principle has broken down as AI crawlers collect content like text, articles, and images to generate answers without sending visitors to the original source, costing content creators revenue and reducing the satisfaction of knowing that people are visiting their content. Without the incentive to create quality, original content, society will collapse and the future of the Internet will be at risk.

If we are to survive in the age of AI, we need to give content owners and publishers the tools to control what content they consume and build a new economic model that works for everyone: creators, consumers, founders, the future of AI, and the future of the web, said Matthew Prince, co-founder and CEO of Cloudflare. Original content is what made the internet one of the greatest inventions of the last century, and its important that creators continue to create it. By enabling AI crawlers to collect unlimited amounts of content, were aiming to put the power back in the hands of creators, and let AI companies continue to innovate. Its about securing a future of a free and vibrant internet with a new model that works for everyone.

Cloudflares innovative approach to blocking AI crawlers is a game changer for publishers and sets a new standard for respecting content online. When AI companies can no longer take what they want for free, it opens the door for sustainable innovation built on permission and collaboration. This is an important step towards creating a fairer exchange of value on the internet that protects creators, supports quality journalism, and holds AI companies accountable, said Roger Lynch, CEO of Condé Nast.

Weve long said that AI platforms must pay originating publishers and creators fairly for using our content, said Neil Vogel, CEO of Dotdash Meredith. Now we can restrict access to our content to only AI partners willing to negotiate fair terms. We are proud to support Cloudflare and look forward to using Cloudflares tools to protect our content and the open web.

As the largest publisher in America, with USA TODAY and more than 200 local publications across the USA TODAY network, it is critically important that we prevent unauthorized scraping and unjustified use of our original content. As our industry faces these challenges, we believe Cloudflares technology can help combat this theft of valuable intellectual property, said Renn Turiano, Chief Consumer and Product Officer of Gannett Media.

Content creators and publishers around the world use Pinterest to grow their businesses, reach new audiences and measure their success directly, said Bill Ready, CEO of Pinterest. As AI continues to reshape the digital landscape, we are committed to building a strong internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive.

Content creators and publishers around the world use Pinterest to grow their businesses, reach new audiences and measure their success directly, said Bill Ready, CEO of Pinterest. As AI continues to reshape the digital landscape, we are committed to building a strong internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive.

We applaud Cloudflare for supporting a sustainable digital ecosystem that benefits all stakeholders: consumers who rely on reliable data, publishers who invest in data creation, and advertisers who support the dissemination of data, said Vivek Shah, CEO of Ziff Davis.

Enforcement of a Permission-Based Model for the Internet

Cloudflare powers one of the worlds largest networks, helping manage and protect 20% of the webs traffic. Handling trillions of requests per day, the company has the worlds most advanced bot management solution to precisely distinguish between human users and AI crawlers. Cloudflare introduced a single-click option to block AI crawlers in September 2024. Since then, more than one million customers have chosen the solution, a robust yet easy-to-use solution that stops data extraction while customers define their own AI strategy.

Cloudflare is now taking the next step in enforcing its permission-based model for AI crawlers. Going forward, AI companies will need to explicitly grant a website permission before fetching data. When signing up with Cloudflare, all new domains will be asked if they want to allow AI crawlers, giving customers the ability to explicitly allow or deny access to AI crawlers from the outset. This major change means that all new domains will start with default controls, removing the need for webpage owners to manually configure settings to opt out. Customers can easily review their settings and enable crawling at any time, if they want to keep their content open.

Permission-based model for AI crawlers is used by leading content owners, publishers, media companies, and global technology companies.

Leading content, media and technology company supporting the creation of a sustainable future that prioritizes original content such as:ADWEEK, The Arena Group, The Associated Press, The Atlantic, Atlas Obscura, BuzzFeed, Inc., Condé Nast, Digital Content Next, DOC, Dotdash Meredith, Drupal & Acquia, EngineEars, Evolve Media, Fortune, Gannett Media, Groundviews.org, จดหมายข่าว Half Baked, Hyperscience, IAB Tech Lab, Independent Media, International Center for Journalists, Internet Brands, Linkup, News/Media Alliance, O'Reilly Media, PMC, Pinterest, ProRata AI, Quora, Raptive, Reddit, SimpleFeed, Sky News Group, Snopes.com, SourceForge, Sovrn, Inc., Stack Overflow, StockTwits, SustainableMedia.Center, Third Door Media, TIME, Universal Music Group, Webflow และ Ziff Davis.

AI companies can now more reliably audit their data collection programs.

Cloudflare is now making its content ecosystem more transparent to AI companies and creators. The company recently offered new ways for AI bots to prove their identity and for websites to identify them, giving creators and website owners a new identity mechanism and control over what data they allow to collect. Cloudflare is also working on a new protocol to provide bot owners and AI agent developers with a publicly standardized way to identify themselves.

CLOUDFLARE / FAQ Co., Ltd. (PR)