https://techcrunch.com/2024/09/23/cloudflares-new-marketplace-lets-websites-charge-ai-bots-for-scraping/ TechCrunch Logo Login * Search * Startups * Venture * Apple * Security * AI * Apps * Events * Startup Battlefield * More + Fintech + Cloud Computing + Layoffs + Hardware + Google + Microsoft + Transportation + EVs + Meta + Instagram + Amazon + TikTok + Newsletters + Podcasts + Partner Content + Crunchboard Jobs + Contact Us Search [ ] AI Cloudflare's new marketplace will let websites charge AI bots for scraping Maxwell Zeff 6:19 AM PDT * September 23, 2024 Comment [Matthew-Prince-CloudflareDSC00249]Image Credits: Noam Galai/Getty Images / Getty Images Cloudflare announced plans on Monday to launch a marketplace in the next year where website owners can sell AI model providers access to scrape their site's content. The marketplace is the final step of Cloudflare CEO Matthew Prince's larger plan to give publishers greater control over how and when AI bots scrape their websites. "If you don't compensate creators one way or another, then they stop creating, and that's the bit which has to get solved," said Prince in an interview with TechCrunch. As the first step in its new plan, on Monday, Cloudflare launched free observability tools for customers, called AI Audit. Website owners will get a dashboard to view analytics on why, when, and how often AI models are crawling their sites for information. Cloudflare will also let customers block AI bots from their sites with the click of a button. Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial. A demo of AI Audit shared with TechCrunch showed how website owners can use the tool, which is able to see where each scraper that visits your site comes from, and offers selective windows to see how many times scrapers from OpenAI, Meta, Amazon, and other AI model providers are visiting your site. [Screenshot-2024-09-23-at-6]Demo of AI audit. Image Credits: Cloudflare Cloudflare is trying to address a problem looming over the AI industry: How will smaller publishers survive in the AI era if people go to ChatGPT instead of their website? Today, AI model providers scrape thousands of small websites for information that powers their LLMs. While some larger publishers have struck deals with OpenAI to license content, most websites get nothing, but their content is still fed into popular AI models on a daily basis. That could break the business models for many websites, reducing traffic they desperately need. Earlier this summer, AI-powered search startup Perplexity was accused of scraping websites that deliberately indicated they did not want to be crawled using the Robots Exclusion Protocol. Shortly after, Cloudflare released a button to ensure customers could block all AI bots with one click. "That was out of frustration we were hearing, where people were feeling like their content was being stolen," said Prince. Some website owners told Business Insider that AI bots were scraping their websites so much, it felt like a DDoS attack was crippling their servers. Having your website scraped can not only be upsetting, but it can literally run up your cloud bill and impact your service. But what if you wanted to block Perplexity's bots, but not OpenAI's? Prince tells TechCrunch that Cloudflare's customers are asking for tools that allow them to choose which AI models have access to their sites. Cloudflare's new tools launching today will allow customers to block some AI crawlers, while letting others through. Even large publishers that have struck licensing deals with OpenAI -- such as TIME, Conde Nast, and The Atlantic -- have relatively little insight into how much ChatGPT is scraping their websites, according to Prince. Many of them have to accept what OpenAI tells them, but the answer determines if the publishers are getting a good licensing deal or not. But Cloudflare's marketplace, launching sometime in the next year, aims to allow small publishers to strike deals with AI model providers as well. "Let's give all of you the ability to do what only Reddit, Quora, and the big publishers of the world have done previously," said Prince. "What if we let you set, effectively, a price for accessing and taking your content to ingest into these systems." While it's a bold idea, Cloudflare is not sharing a fully fleshed-out idea of what its marketplace will look like. Prince says websites could charge AI model providers based on the rates at which they're scraping individual websites, but it's unclear how much they will really pay. Further, he says websites could charge a monetary price to be scraped, or simply ask AI labs to give them credit. The details are fuzzy. While AI companies may not initially be excited about paying for content they currently get for free, Cloudflare's CEO says he thinks this is ultimately good for the AI ecosystem. Prince says the current landscape, where some AI companies don't pay for content ever, is not sustainable. More TechCrunch Get the industry's biggest tech news Explore all newsletters TechCrunch Daily News Every weekday and Sunday, you can get the best of TechCrunch's coverage. Startups Weekly Startups are the core of TechCrunch, so get our best coverage delivered weekly. TechCrunch Fintech The latest Fintech news and analysis, delivered every Tuesday. TechCrunch Mobility TechCrunch Mobility is your destination for transportation news and insight. Email address (required) [ ] Subscribe By submitting your email, you agree to our Terms and Privacy Notice. Tags AI, AI model, ChatGPT, cloudflare, scrapers, web scraper AI Crypto scammers hack OpenAI's press account on X Kyle Wiggers 3 mins ago OpenAI's official press account on X was hacked by cryptocurrency scammers. Or, at least that's what appears to have happened. Late Tuesday afternoon, OpenAI Newsroom, an account OpenAI recently launched... Crypto scammers hack OpenAI's press account on X Venture Venture capitalists continue to play musical chairs Rebecca Szkutak 39 mins ago From Keith Rabois to Ethan Kurzweil, a lot of VCs have switched firms or spun out of storied VC institutions to launch their own funds this year. These employment changes... Venture capitalists continue to play musical chairs Space TechCrunch Space: SpaceX calls out 'systematic challenges' with FAA Aria Alamalhodaei 42 mins ago Hello and welcome back to TechCrunch Space. I wanted to flag once again that the final agenda for the Space Stage at TechCrunch Disrupt is now live. I'll be pushing... AI Here is what's illegal under California's 9 (and counting) new AI laws Maxwell Zeff 2 hours ago California Governor Gavin Newsom is currently considering 38 AI-related bills, including the highly contentious SB 1047, which the state's legislature sent to his desk for final approval. These bills try... Here is what's illegal under California's 9 (and counting) new AI laws AI Sam Altman catapults past founder mode into 'god mode' with latest AI post Sarah Perez 2 hours ago Founder mode? Pffft. Who needs that when you can be the father of creation, ushering in a new age of humanity? Welcome to "god mode." Sam Altman, the CEO of... Sam Altman catapults past founder mode into 'god mode' with latest AI post Climate How Big Tech embraced nuclear power Tim De Chant 3 hours ago Companies are eyeing nuclear power as a way to reconcile their breakneck data center growth with their commitments to hit net zero carbon emissions. How Big Tech embraced nuclear power Apps X will soon make your public posts visible to accounts you've blocked Maxwell Zeff 3 hours ago X will soon change the functionality behind its block button so that if you block an account, they will still be able to see your public posts, according to changes... X will soon make your public posts visible to accounts you've blocked Security Some Kaspersky customers receive surprise forced-update to new antivirus software Lorenzo Franceschi-Bicchierai 4 hours ago After U.S. ban, Kaspersky's American customers were automatically migrated to UltraAV antivirus, in some cases without the customers' knowledge. Some Kaspersky customers receive surprise forced-update to new antivirus software Featured Article Tesla Superchargers: GM, Ford, Rivian and other EV brands with access EV owners of GM vehicles like the Chevrolet Silverado EV and Cadillac Lyriq will now officially have access to Tesla's Superchargers. Rebecca Bellan 4 hours ago Tesla Superchargers: GM, Ford, Rivian and other EV brands with access Transportation Rivian reveals first $10M in grants for long-promised Rivian Foundation Sean O'Kane 4 hours ago Rivian has revealed the first $10 million worth of grants from the Rivian Foundation, three years after the EV maker vowed the philanthropy would receive 1% of its equity to... Rivian reveals first $10M in grants for long-promised Rivian Foundation Enterprise Ephos wants to shatter the market for AI and quantum chips with a new design based on glass Ingrid Lunden 5 hours ago A theoretical physicist believes he has made a breakthrough in photonics research that will enable us to have faster and better processors -- a major need in artificial intelligence, quantum... Ephos wants to shatter the market for AI and quantum chips with a new design based on glass AI Rabbit's web-based 'large action model' agent arrives on r1 on October 1 Devin Coldewey 5 hours ago Rabbit is ready to release the first generic version, which is to say not specific to any app or interface, of its Large Action Model. Rabbit's web-based 'large action model' agent arrives on r1 on October 1 TechCrunch Disrupt 2024 Consumer, culture, and creators with Erin and Sara Foster at TechCrunch Disrupt 2024 Dominic-Madori Davis 5 hours ago Two of the industry's most famous sisters, Erin and Sara Foster, sit down alongside business partner Phil Schwarz at TechCrunch Disrupt 2024 to talk about consumer investing, culture curation, and... Consumer, culture, and creators with Erin and Sara Foster at TechCrunch Disrupt 2024 Robotics Boston Dynamics' Spot can now autonomously unlock doors Brian Heater 5 hours ago The "embedded digital access credentials" work over Bluetooth. Boston Dynamics' Spot can now autonomously unlock doors AI Letta, one of UC Berkeley's most anticipated AI startups, has just come out of stealth Julie Bort 7 hours ago AI memory management startup Letta just emerged from stealth with $10 million in seed led by Felicis and a bunch of big-name angels. Letta, one of UC Berkeley's most anticipated AI startups, has just come out of stealth Transportation First TikTok, now smart cars: How Biden's new proposed ban will affect US automakers Rebecca Bellan 7 hours ago The White House issued a long-anticipated proposal Monday that would ban Chinese smart cars because internet-connected vehicles pose a national security risk. The proposal, made amid an escalating trade war,... First TikTok, now smart cars: How Biden's new proposed ban will affect US automakers Hardware Meta Connect 2024: How to watch the metaverse and generative AI event Brian Heater 7 hours ago Meta Connect starts Wednesday at 10 a.m. PT and is set to focus on Meta's XR platforms, the metaverse, and its generative AI platform, Llama. Meta Connect 2024: How to watch the metaverse and generative AI event TechCrunch Disrupt 2024 TechCrunch Disrupt 2024 Side Events schedule: Women in Tech, SignalFire, Llama Lounge, and more to host TechCrunch Events 7 hours ago With TechCrunch Disrupt 2024 right around the corner, we're thrilled to introduce the companies hosting Side Events that will extend the buzz and excitement to the thousands of attendees and... TechCrunch Disrupt 2024 Side Events schedule: Women in Tech, SignalFire, Llama Lounge, and more to host Hardware Google TV receives a major update ahead of the launch of its new streaming box Lauren Forristal 8 hours ago Ahead of the launch of Google TV Streamer, the company's new set-top streaming box, the tech giant is also bringing updates to all Google TV devices. This includes a home... Google TV receives a major update ahead of the launch of its new streaming box Featured Article Zin Boats' bigger, faster electric leisure craft is built from the hull up with PNW pride After taking on water during the pandemic, Zin Boats is back with a bigger, better electric watercraft that it has built from the hull up -- again. Devin Coldewey 8 hours ago Zin Boats' bigger, faster electric leisure craft is built from the hull up with PNW pride TechCrunch Disrupt 2024 5 days left to grab rebooted ticket prices for TechCrunch Disrupt 2024 TechCrunch Events 8 hours ago The countdown to TechCrunch Disrupt 2024 is on, and so are rebooted ticket prices! Save up to $600 on individual ticket types before September 27. Take advantage of these huge... 5 days left to grab rebooted ticket prices for TechCrunch Disrupt 2024 Apps TikTok launches expanded subscriptions feature for creators Aisha Malik 8 hours ago TikTok announced on Monday that its redesigned "Subscription" monetization offering is rolling out to eligible creators in select regions, including Brazil, France, Germany, Spain, the U.K., Indonesia, Italy, Japan, South... TikTok launches expanded subscriptions feature for creators Fundraising Pyka fields interest from defense as $40M round goes to scaling up its electric autonomous planes Devin Coldewey 9 hours ago Though it briefly worked on a passenger plane, the company decided after raising some money in 2022 that a cargo variant of the Pelican was more practical in the short... Pyka fields interest from defense as $40M round goes to scaling up its electric autonomous planes Fundraising All Iron Ventures rebrands as Acurio Ventures with a new EUR150M 'follow-on' fund Mike Butcher 9 hours ago The new fund has already made around 20 investments, and it will operate with a generalist thesis, investing across the whole of Europe. All Iron Ventures rebrands as Acurio Ventures with a new EUR150M 'follow-on' fund AI Cloudflare's new marketplace will let websites charge AI bots for scraping Maxwell Zeff 9 hours ago Cloudflare announced plans on Monday to launch a marketplace in the next year where website owners can sell AI model providers access to scrape their site's content. The marketplace is... Cloudflare's new marketplace will let websites charge AI bots for scrapingImage Credits: Noam Galai/Getty Images / Getty Images Transportation Thor and Harbinger's new hybrid RV will let you spend more time at the campsite Sean O'Kane 10 hours ago Legacy automakers are experiencing a sort of existential crisis as they grapple with whether to stick to plans to go all-electric or hedge with hybrids. This sudden appetite for options... Thor and Harbinger's new hybrid RV will let you spend more time at the campsite Apps The new Reeder app is built for RSS, YouTube, Reddit, Mastodon, and more Ivan Mehta 10 hours ago For the longest time, RSS readers have followed an "Inbox Zero" design philosophy by showing an unread count against each source. If you have more than a dozen feeds plugged... The new Reeder app is built for RSS, YouTube, Reddit, Mastodon, and more Climate David Energy is going up against Goliath energy incumbents Rebecca Szkutak 11 hours ago James McGinniss has been obsessed with decarbonization and the energy grid since he was a high schooler over a decade ago. Now, his startup David Energy has a lofty goal:... David Energy is going up against Goliath energy incumbents Enterprise Kestra raises another $8M for its open source orchestration platform Romain Dillet 11 hours ago Data orchestration platform Kestra just raised an $8 million funding round led by Alven, with existing investors Isai and Axeleo participating once again. Kestra raises another $8M for its open source orchestration platform Startups Jump raises $12M to help freelancers get benefits just like employees Romain Dillet 15 hours ago Jump offers full-time contracts to freelancers looking for some stability and the benefits involved with a full-time job. Jump raises $12M to help freelancers get benefits just like employees About * TechCrunch * Staff * Contact Us * Advertise * Crunchboard Jobs * Site Map Legal * Terms of Service * Privacy Policy * RSS Terms of Use * Privacy Placeholder 1 * Privacy Placeholder 2 * Privacy Placeholder 3 * Privacy Placeholder 4 * Code of Conduct * About Our Ads Trending Tech Topics * Apple Intelligence * Cruise Robotaxis * Amazo Video Generator * iOS 18 * Snap Spectacles * Tech Layoffs * ChatGPT * Facebook * X * YouTube * Instagram * LinkedIn * Mastodon * Threads (c) 2024 Yahoo. All rights reserved. Powered by WordPress VIP