Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model

Empowers leading publishers and AI companies to stop the scraping and use of original content without permission

Cloudflare, Inc. (NYSE:NET), the leading connectivity cloud company, today announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default. Starting today, website owners can choose if they want AI crawlers to access their content, and decide how AI companies can use it. AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow. Cloudflare's new default setting is the first step toward a more sustainable future for both content creators and AI innovators.

For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information. That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.

"If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow's AI founders, and the future of the web itself," said Matthew Prince, co-founder and CEO of Cloudflare. "Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone."

"Cloudflare's innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership," said Roger Lynch, CEO of Condé Nast. "This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable."

"We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements," said Neil Vogel, CEO of Dotdash Meredith. "We're proud to support Cloudflare and look forward to using their tools to protect our content and the open web."

"As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorized scraping and the use of our original content without fair compensation is critically important," said Renn Turiano, Chief Consumer and Product Officer of Gannett Media. "As our industry faces these challenges, we are optimistic the Cloudflare technology will help combat the theft of valuable IP."

"Creators and publishers around the world leverage Pinterest to expand their businesses, reach new audiences and directly measure their success. As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive," said Bill Ready, CEO of Pinterest.

"AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what," said Steve Huffman, co-founder and CEO of Reddit. "The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare's efforts are a step in the right direction for everyone."

"We applaud Cloudflare for advocating for a sustainable digital ecosystem that benefits all stakeholders — the consumers who rely on credible information, the publishers who invest in its creation, and the advertisers who support its dissemination," said Vivek Shah, CEO of Ziff Davis.

Enforcing a Permission-Based Model for the Internet

Cloudflare powers one of the world's largest networks, helping to manage and protect traffic for 20% of the web. The company handles trillions of requests daily, and thus has the world's most advanced bot management solutions, accurately distinguishing between human users and AI crawlers. In September 2024, Cloudflare introduced the option to block AI crawlers in a single click. More than one million customers have since chosen this option, meant to be an aggressive but easy solution that halts scraping while they determine their AI strategy.

Now, Cloudflare is taking the next step to enforce a permission-based model for AI crawlers. AI companies will now be required to obtain explicit permission from a website before scraping. Upon sign-up with Cloudflare, every new domain will now be asked if they want to allow AI crawlers, giving customers the choice upfront to explicitly allow or deny AI crawlers access. This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out. Customers can easily check their settings and enable crawling at any time if they want their content to be freely accessed.

Top Global Publishers, Media, & Technology Companies Embrace a Permission-Based Model for AI Crawling

Leading content, media, and technology companies are in support of creating a more sustainable future that values original content, including: ADWEEK, The Arena Group, The Associated Press, The Atlantic, Atlas Obscura, BuzzFeed, Inc., Condé Nast, Digital Content Next, DOC, Dotdash Meredith, Drupal & Acquia, EngineEars, Evolve Media, Fortune, Gannett Media, Groundviews.org, Half Baked Newsletter, Hyperscience, IAB Tech Lab, Independent Media, International Center for Journalists, Internet Brands, Linkup, News/Media Alliance, O'Reilly Media, PMC, Pinterest, ProRata AI, Quora, Raptive, Reddit, SimpleFeed, Sky News Group, Snopes.com, SourceForge, Sovrn, Inc., Stack Overflow, StockTwits, SustainableMedia.Center, Third Door Media, TIME, Universal Music Group, Webflow, and Ziff Davis.

AI Companies Can Now More Reliably Verify Their Crawlers

Now Cloudflare is making the content ecosystem more transparent for AI companies and creators. The company recently proposed new ways for AI bots to authenticate themselves as well as for websites to identify those bots – giving creators and website owners new identification mechanisms and control over what crawlers they want to allow. Cloudflare is participating in the development of a new protocol to provide bot owners and AI agent developers with a public, standard way to identify themselves.

Companies in Support of a Permission-Based Approach for AI Crawling

• ADWEEK: "As the front page and homepage for marketing, advertising and media industry leaders, ADWEEK's position has been clear that we must be compensated for our investment grade journalism and information. I am thrilled Cloudflare has created a marketplace and mechanism that will enable us to properly participate in the promise LLMs have for our industry." – Will Lee, CEO, ADWEEK
• The Arena Group: "We think of our writers and content creators as entrepreneurs. Their work deserves protection. By blocking unauthorized AI crawlers, Cloudflare is not just defending content – it's defending the future of creators and storytellers. This is a vital move toward a digital economy built on trust, permission and fair value." – Paul Edmondson, CEO, The Arena Group
• The Associated Press: "The information landscape continues to change rapidly but the value of accurate, factual, nonpartisan journalism has never been more essential. We're pleased to participate in this important framework that will help ensure intellectual property is protected and all content creators are fairly compensated for their work." – Kristin Heitmann, Chief Revenue Officer, The Associated Press
• The Atlantic: "For too long, giant AI companies have built businesses on training data that they never paid for, and by scraping sites from whom they haven't even asked permission. Now, thanks to Cloudflare, and its long-time commitment to the health of the open Web, this dynamic is finally going to change." – Nicholas Thompson, CEO, The Atlantic
• Atlas Obscura: "As a media CEO, a former chief technology officer, and a longtime journalist, I am constantly reflecting on the moment we are in and how pivotal it is to the economics of public discourse. We have to get the calculus right now, so that publishers are treated fairly. As a customer of Cloudflare, I'm glad Cloudflare is looking for ways to create fair economics in the age of AI." – Louise Story, CEO, Atlas Obscura
• BuzzFeed, Inc.: "Cloudflare's initiative is a crucial first step in publishers taking control of their content in the AI era. BuzzFeed is proud to stand with Cloudflare and our peers as part of our multipronged approach to fuel innovation responsibly while also securing the future of quality content and journalism." – Ken Blom, Chief Business Officer, BuzzFeed, Inc.
• Digital Content Next: "Permission is the law when it comes to copyrighted content - full stop. Cloudflare's move brings technology to help ensure AI companies can't sidestep the rules. It's a critical step toward restoring a fair value exchange for trusted publishers and content creators." – Jason Kint, CEO, Digital Content Next
• DOC: "Generative AI has given the entire Internet industry an opportunity to reimagine the broken value exchange between publishers, consumers, and marketers. It takes companies with courage and conviction to break the decades-old practice of ‘steal first, apologize later' that dominates the business models of most Internet giants. Kudos to Cloudflare and its founding participants for taking decisive and consequential action while there's still time." – John Battelle, Co-founder and CEO, DOC
• Drupal & Acquia: "Large sites can afford the technical infrastructure to detect and block crawlers. Some can even pursue direct licensing deals with AI companies. However, most content creators can do neither. Cloudflare's solution isn't complete, but it establishes a crucial principle: AI training data has a price, and creators deserve to share in the value AI generates from their work." – Dries Buytaert, Founder and Project Lead of Drupal and Co-founder and Executive Chair of Acquia
• EngineEars: "In the world of music, copyright protection is everything for the livelihood of creators. EngineEars, the operating system for music creation, wholeheartedly supports Cloudflare pioneering technology that protects creator's art and human expression from being stolen by AI firms." – Derek Ali, 4x Grammy award winning audio engineer, founder and CEO, EngineEars
• Evolve Media: "The rise of AI brings incredible opportunity, and there's a clear need for a future where creators, publishers, and AI companies can thrive together. Cloudflare's initiative supports that vision by promoting a more sustainable Internet through protecting the value of original content." – Aaron Broder, Founder and CEO, Evolve Media
• Fortune: "The rise of AI presents incredible opportunities but also the moment for when publishers lock in proper attribution and compensation for our valuable intellectual property and carefully created content. Fortune sees a future of both active cooperation with AI companies to license content and suitable pay per read models. Because of that, we support Cloudflare's initiative to provide a framework that ensures equitable use of content by AI companies, and which contributes to sustainability for publishers." – Anastasia Nyrkovskaya, CEO, Fortune
• Groundviews.org: "After over two decades creating and editing critical content, especially in violent contexts defined by democratic backsliding, I've negotiated firsthand how civic media platforms struggle to operate, and rely on traffic to original work in order to sustain operations. Today, we face a chilling reality where AI systems scrape knowledge that citizen journalists often risk their lives to produce, and publish – extracting value without attribution or compensation. Cloudflare's permission-based approach represents a novel, and urgent, and meaningful intervention. For independent, small, local language based media networks, where international visibility often provides the only protection against censorship or worse, maintaining control over how our content is accessed, and monetised isn't merely about sustainability. It's about preserving the incentive structures that enable bearing witness to continue. This initiative helps ensure that the next generation of investigative journalists, and civic media platforms can still afford to speak truth to power." – Dr. Sanjana Hattotuwa, Founding Editor, Groundviews.org
• Half Baked Newsletter: "As a small publisher, we rely on the trust and engagement of our readers. Cloudflare's move gives us the control we need to protect our content and continue building something real in a world of AI noise. It protects the creative spark that made the Internet worth exploring in the first place." – Darragh Lucey, CEO, Half Baked Newsletter
• Hyperscience: "As AI reshapes the way we create, consume, and interact with information, the value of original content, and the rights of those who create it, have never been more important. At Hyperscience, our success is built on original research and proprietary ML models, which our customers rely on to run transparent, compliant, and accurate AI workflows. We remain steadfast in our belief that creators, researchers, and technologists deserve recognition and protection in this new era of AI. We are excited to support this visionary Cloudflare initiative and believe that it will provide the framework to ensure that creators in every domain – including software development, healthcare and biotech, art and music, and journalism – benefit from an equitable and productive exchange with the LLMs and AI systems that leverage their work." – Andrew Joiner, CEO, Hyperscience
• IAB Tech Lab: "What we're hearing from publishers is consistent. Their content is being pulled into AI systems without notice, and they're seeing real drops in traffic as a result. That's why we launched the LLM Content Ingest API Initiative. To give publishers and brands the ability to decide how and when their content is accessed by AI tools. Cloudflare is helping make that possible by implementing parts of the framework, building the enforcement infrastructure behind it and moving the industry toward a more permission-based approach to content access." – Shailley Singh, EVP of Product and COO, IAB Tech Lab
• Independent Media: "Trusted, truly independent journalism is vital for us all, so it's great to see Cloudflare demonstrating that ingenuity and innovation, not just legislation, can play an important role in securing a sustainable model for how publishers and AI companies co-exist. Creating a marketplace for high-quality content from responsible publishers is crucial – for the AI companies as well as the news industry." – Christian Broughton, CEO, The Independent & Independent Media
• International Center for Journalists: "We see journalists across the world providing vital, original reporting to their communities, yet AI bots scrape their work for free while newsrooms struggle to stay open. At ICFJ+, we are working with small news sites -- beginning in Africa and across a variety of languages -- to help them protect and reclaim the value of their original work in the age of AI. We welcome this very promising initiative from Cloudflare." – Sharon Moshavi, co-CEO of ICFJ+ and president of ICFJ
• Linkup: "High-quality, trusted content is the lifeblood of Linkup's answer engine, ensuring we deliver reliable insights to our customers. Cloudflare's program gives us a clear, AI-ready path to that content while respecting and rewarding its creators. We're thrilled to help pioneer a model that safeguards a vibrant, sustainable future for the Internet." – Phil Mizrahi, CEO and Co-founder, Linkup
• News/Media Alliance: "The rise of AI presents exciting opportunities, but in order for the industry to grow sustainably, it must do so in cooperation with publishers. Cloudflare's tools provide a strong framework for a more equitable exchange, offering a path for both industries to grow and thrive together. By valuing and protecting the rights of publishers, we're ensuring that they can continue to create the high-quality content that fuels AI innovation." – Danielle Coffey, President & CEO, News/Media Alliance
• O'Reilly Media: "It's so great to see Cloudflare standing up for publishers! Building a sustainable economy for AI starts with giving creators and publishers control over their content, rather than simply letting it be expropriated by AI platforms and application developers. Colonialism doesn't age well as a business model." – Tim O'Reilly, Founder and CEO, O'Reilly Media
• PMC: "We are very pleased to see the market-based solution that Cloudflare has offered and encourage others to join. Waiting for legal or government interventions to remedy the current challenges in our ecosystem isn't a strategy. Supporting sustainable models that compensate content creators fairly is an obligation that we all share as news and media organizations." – Craig Perreault, Chief Strategy Officer, PMC
• ProRata AI: "Our whole mission is to protect and elevate human creativity in the AI era. That's why we're proud to be one of the first AI companies to participate in Cloudflare's initiative to create a new permission-based model for the Internet. We believe that creators and publishers deserve to be fairly compensated for the value they bring, and we're thrilled to support Cloudflare in helping make that vision a reality." – Bill Gross, Founder & CEO, ProRata AI
• Quora: "Publishers are essential to both the Internet's future and AI's growth. At Quora, we believe these two industries can thrive together, and we're committed to supporting initiatives like Pay Per Crawl that create mutual value and sustainable growth for the long term." – Ricky Arai-Lopez, Head of Product, Quora
• Raptive: "Digital publishers fuel the Internet with original ideas and expertise, and they deserve control over how their work is used. This move toward a permission-based model is a meaningful step—restoring fair value, protecting creativity, and ensuring a better future for those who make the web worth visiting." — Michael Sanchez, CEO, Raptive
• SimpleFeed: "As the leader in content syndication, SimpleFeed is on the front lines of the battle with AI bots. We are excited for solutions that bring publishers greater transparency and fair compensation for their work. The world needs healthy publishers and we welcome opportunities to partner with companies building a new ecosystem on more equitable incentives." – Mark Carlson, CEO, SimpleFeed
• Sky News Group: "This permission-based model will help secure the future of quality digital journalism, which is our commitment. Sky News is all about providing ‘the full story, first'– so we wanted to be among the first to join Cloudflare's framework for setting fair terms of trade in news. We'll help design the future of these services as video becomes an ever-larger part of both crawling and publishing." – David Rhodes, Executive Chairman, Sky News Group
• Snopes.com: "If the shift toward AI continues to erode web traffic, I worry that most premium publishers will have no choice but to adopt a subscription-only model. The whole Internet behind a paywall isn't good for anyone. That's why I'm optimistic about Cloudflare's efforts to protect publishers and help us find a sustainable solution that benefits all sides." – Chris Richmond, CEO, Snopes.com
• SourceForge: "At SourceForge, we empower millions of businesses and developers by providing a trusted platform for discovering software through authentic user-generated B2B software reviews and open source projects. Cloudflare's permission-based model for AI crawlers is a vital advancement that helps protect original content while fostering innovation. By giving publishers control over how their content is accessed and used, Cloudflare supports a fairer digital ecosystem that benefits publishers, creators, developers, and users alike. We're excited to partner with Cloudflare in promoting transparency and responsible AI practices across the Internet." – Logan Abbott, President, SourceForge
• Sovrn, Inc.: "For more than a decade, Sovrn has relentlessly championed the open and free Internet. While AI represents incredible opportunities to advance human knowledge and abundance, we are putting at risk creators and publishers who make the Internet indispensable. We collectively need to ensure a balance where creators and publishers can continue their important work and thrive in the future, and that's why Cloudflare's work is so critical to this next chapter of the Internet." – Walter Knapp, CEO, Sovrn, Inc.
• Stack Overflow: "We applaud Cloudflare for all they are doing to support the new business models of the modern day Internet. Community platforms that fuel LLMs should be compensated for their contributions so they can invest back in their communities. We've been very vocal around the importance and integrity of socially responsible AI practices. Our mission is to set new standards with vetted, trusted, and accurate data that will be the foundation on which technology solutions are built and delivered to our users. We believe attribution is non-negotiable, that human review of content is necessary and community feedback is the only way to ensure GenAI tools accelerate innovation and not misinformation." – Prashanth Chandrasekar, CEO, Stack Overflow
• StockTwits: "At Stocktwits, original content and real-time conversation are at the heart of our community. As AI transforms how information is discovered and used, it's essential that innovation doesn't come at the expense of the creators and platforms driving it. We're proud to support Cloudflare's efforts to ensure a more transparent, fair, and sustainable Internet." – Howard Lindzon, Founder and CEO, StockTwits
• SustainableMedia.Center: "In order for the Internet to remain the critical source of news and information it has been for web users, we need to dramatically re-think the business model for users and publishers. Cloudflare's vision aligns strongly with the Sustainable Media Center's mission to give GenZ users control of their digital lives. We applaud this critical effort." – Steven Rosenbaum, Co-founder and Executive Director, SustainableMedia.Center
• Third Door Media: "For years, publishers large and small have played by the rules, creating high-quality content in exchange for visibility and traffic. But AI shifts the game. Cloudflare's approach recognizes that content has value beyond clicks. It's time we build a new framework, one that respects the work of journalists and creators while fueling the future of AI responsibly. I, for one, am hoping to see more solutions for publishers in the marketplace." – Marc Sirkin, CEO, Third Door Media
• TIME: "At TIME, we're committed to advancing innovation without compromising the integrity of original journalism. Cloudflare's initiative is a meaningful step toward building a healthier AI ecosystem—one that respects the value of trusted content and supports the creators behind it." – Mark Howard, Chief Operating Officer, TIME
• Universal Music Group: "We welcome this new initiative from Cloudflare, that will help address the indiscriminate, disruptive, and unauthorized scraping of both creative and commercial IP by AI model developers and support new licensing. At UMG, we have always embraced innovation and new technologies, and firmly believe that AI, when used ethically, transparently, and respectfully of copyright and human creativity, has the opportunity to introduce significant new avenues for creativity and future monetization." – Boyd Muir, Chief Operating Officer, Universal Music Group
• Webflow: "With millions of websites powered by Webflow, we see how quickly AI-driven discovery is becoming the norm. Cloudflare's opt-in model brings structure and permission into that shift, giving content owners and marketing teams more control over how their work is accessed and reused. It's a necessary evolution, and a meaningful step toward a more accountable AEO ecosystem." – Allan Leinwand, CTO, Webflow

To learn more, please check out the resources below:

• Blog: Content Independence Day: no AI crawl without compensation!
• Blog: Control content use for AI training with Cloudflare's managed robots.txt and blocking for monetized content
• Blog: Message Signatures are now part of our Verified Bots Program, simplifying bot authentication
• Blog: Introducing Pay Per Crawl: enabling content owners to charge AI crawlers for access
• Blog: From Googlebot to GPTBot: Who's crawling your site in 2025
• Blog: The crawl before the fall… of referrals: understanding AI's impact on content providers
• Apply for Early Access: Cloudflare's Pay Per Crawl beta
• Cloudflare Radar: AI Insights: Cloudflare's hub for Internet traffic trends and insights

About Cloudflare

Cloudflare, Inc. (NYSE:NET) is the leading connectivity cloud company on a mission to help build a better Internet. It empowers organizations to make their employees, applications and networks faster and more secure everywhere, while reducing complexity and cost. Cloudflare's connectivity cloud delivers the most full-featured, unified platform of cloud-native products and developer tools, so any organization can gain the control they need to work, develop, and accelerate their business.

Powered by one of the world's largest and most interconnected networks, Cloudflare blocks billions of threats online for its customers every day. It is trusted by millions of organizations – from the largest brands to entrepreneurs and small businesses to nonprofits, humanitarian groups, and governments across the globe.

Learn more about Cloudflare's connectivity cloud at cloudflare.com/connectivity-cloud. Learn more about the latest Internet trends and insights at radar.cloudflare.com.

Follow us: Blog | X | LinkedIn | Facebook | Instagram

Forward-Looking Statements

This press release contains forward-looking statements within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended, which statements involve substantial risks and uncertainties. In some cases, you can identify forward-looking statements because they contain words such as "may," "will," "should," "expect," "explore," "plan," "anticipate," "could," "intend," "target," "project," "contemplate," "believe," "estimate," "predict," "potential," or "continue," or the negative of these words, or other similar terms or expressions that concern Cloudflare's expectations, strategy, plans, or intentions. However, not all forward-looking statements contain these identifying words. Forward-looking statements expressed or implied in this press release include, but are not limited to, statements regarding the capabilities and effectiveness of Cloudflare's products and technology, the benefits to Cloudflare's customers from using Cloudflare's products and technology, the potential opportunity for Cloudflare to attract additional customers and to expand sales to existing customers through Cloudflare's new solutions, the timing of when Cloudflare's new solutions will be generally available to all current and potential Cloudflare customers, Cloudflare's technological development, future operations, growth, initiatives, or strategies, and comments made by Cloudflare's CEO and others. Actual results could differ materially from those stated or implied in forward-looking statements due to a number of factors, including but not limited to, risks detailed in Cloudflare's filings with the Securities and Exchange Commission (SEC), including Cloudflare's Quarterly Report on Form 10-Q filed on May 8, 2025, as well as other filings that Cloudflare may make from time to time with the SEC.

The forward-looking statements made in this press release relate only to events as of the date on which the statements are made. Cloudflare undertakes no obligation to update any forward-looking statements made in this press release to reflect events or circumstances after the date of this press release or to reflect new information or the occurrence of unanticipated events, except as required by law. Cloudflare may not actually achieve the plans, intentions, or expectations disclosed in Cloudflare's forward-looking statements, and you should not place undue reliance on Cloudflare's forward-looking statements.

© 2025 Cloudflare, Inc. All rights reserved. Cloudflare, the Cloudflare logo, and other Cloudflare marks are trademarks and/or registered trademarks of Cloudflare, Inc. in the U.S. and other jurisdictions. All other marks and names referenced herein may be trademarks of their respective owners.

View source version on businesswire.com: https://www.businesswire.com/news/home/20250701003239/en/

Cloudflare, Inc.

Daniella Vallurupalli

Vice President, Head of Global Communications

[email protected]