[HN Gopher] We can't have nice things because of AI scrapers
___________________________________________________________________
We can't have nice things because of AI scrapers
Author : LorenDB
Score : 77 points
Date : 2026-01-13 21:57 UTC (1 hours ago)
(HTM) web link (blog.metabrainz.org)
(TXT) w3m dump (blog.metabrainz.org)
| SchemaLoad wrote:
| Cloudflare has a service for this now that will detect AI
| scrapers and send them to a tarpit of infinite AI generated
| nonsense pages.
| yakattak wrote:
| Do you have a link to that?
| SchemaLoad wrote:
| https://blog.cloudflare.com/ai-labyrinth/
| bitbasher wrote:
| Wow, so to prevent AI scrapers from harvesting my data I need
| to send all of my traffic through a third party company that
| gets to decide who gets to view my content. Great idea!
| rester324 wrote:
| You can implement this yourself, who is stopping you?
| ranger_danger wrote:
| Modern scrapers are using headless chromium which will not see
| the invisible links, so I'm not sure how long this will be
| effective.
| timpera wrote:
| Unfortunately, Cloudflare often destroys the experience for
| users with shared connections, VPNs, exotic browsers... I had
| to remove it from my site after too many complaints.
| inferiorhuman wrote:
| Which is still a far worse experience than if Cloudflare's
| services weren't needed.
| squigz wrote:
| > The /metadata/lookup API endpoints (GET and POST versions) now
| require the caller to send an Authorization token in order for
| this endpoint to work.
|
| > The ListenBrainz Labs API endpoints for mbid-mapping, mbid-
| mapping-release and mbid-mapping-explain have been removed. Those
| were always intended for debugging purposes and will also soon be
| replaced with a new endpoints for our upcoming improved mapper.
|
| > LB Radio will now require users to be logged in to use it (and
| API endpoint users will need to send the Authorization header).
| The error message for logged in users is a bit clunky at the
| moment; we'll fix this once we've finished the work for this
| year's Year in Music.
|
| Seems reasonable and no big deal at all. I'm not entirely sure
| what "nice things" we can't have because of this. Unauthenticated
| APIs?
| yakattak wrote:
| I agree its not a big deal. Unauthenticated APIs are nice
| though, especially for someone who's maybe not as familiar with
| how APIs work.
| lysace wrote:
| At some point they must become more cost efficient by pure market
| economics mechanisms. That implies less load on sites. Much of
| the scraping that I see is still very dumb/repetative. Like
| Googlebot in like 2001.
| incompatible wrote:
| They seem to have so much bubble money at the moment that the
| cost of scraping is probably a rounding error in their pocket
| change.
| lysace wrote:
| Exactly.
| lep_qq wrote:
| This is frustrating to watch. MetaBrainz is exactly the kind of
| project AI companies should be supporting--open data, community-
| maintained, freely available for download in bulk. Instead
| they're: [?] Ignoring robots.txt (the bare minimum web courtesy)
| [?] Bypassing the provided bulk download (literally designed for
| this use case) [?] Scraping page-by-page (inefficient for
| everyone) [?] Overloading volunteer-run infrastructure [?]
| Forcing the project to add auth barriers that hurt legitimate
| users The irony: if they'd just contacted MetaBrainz and said
| "hey, we'd like to use your dataset for training," they'd
| probably get a bulk export and maybe even attribution. Instead,
| they're burning goodwill and forcing open projects to lock down.
| This pattern is repeating everywhere. Small/medium open data
| projects can't afford the infrastructure to handle aggressive
| scraping, so they either: 1. Add authentication (reduces
| openness) 2. Rate limit aggressively (hurts legitimate API users)
| 3. Go offline entirely (community loses the resource) AI
| companies are externalizing their data acquisition costs onto
| volunteer projects. It's a tragedy of the commons, except the
| "commons" is deliberately maintained infrastructure that these
| companies could easily afford to support. Have you considered
| publishing a list of the offending user agents / IP ranges? Might
| help other projects protect themselves, and public shaming
| sometimes works when technical measures don't
| tensegrist wrote:
| Scraping page-by-page (inefficient for everyone)
|
| you know what else is "(inefficient for everyone)"? posting the
| output instead of the prompt
| tensegrist wrote:
| the more time passes the more i'm convinced that the solution is
| to--somehow--force everyone to have to go through something like
| common crawl
|
| i don't want people's servers to be pegged at 100% because a
| stupid dfs scraper is exhaustively traversing their search
| facets, but i also want the web to remain scrapable by ordinary
| people, or rather go back to how readily scrapable it used to be
| before the invention of cloudflare
|
| as a middle ground, perhaps we could agree on a new /.well-known/
| path meant to contain links to timestamped data dumps?
| nostrademons wrote:
| That's sorta what MetaBrainz did - they offer their whole DB as
| a single tarball dump, much like what Wikipedia does. I
| downloaded it in the order of an hour; if I need a MusicBrainz
| lookup, I just do a local query.
|
| For this strategy to work, people need to actually _use_ the DB
| dumps instead of just defaulting to scraping. Unfortunately
| scraping is trivially easy, particularly now that AI code
| assistants can write a working scraper in ~5-10 minutes.
| Imustaskforhelp wrote:
| If someone wants to scrape. I mean not levels of complete
| internet similar to how google does but at a niche level (like
| you got a forum you wish to scrape)
|
| I like to create tampermonkey scripts regarding these. They are
| like more lightweight/easier way to build extensions mostly imo
|
| Now I don't like AI but I don't know anything about scraping so
| I used AI to generate the scraping code and paste it in
| tampermonkey and let it run
|
| I recently used this for where I effectively scraped a website
| which had list of vps servers and their prices and I built
| myself a list of that to analyze as an example
|
| Also I have to say this that I usually try to look out for
| databases so much so that on a similar website like this
| related to something, I contacted them about db but no
| response, their db of server prices were private and only
| showed lowest
|
| So I picked the other website and did this. I also scraped all
| headlines of lowendtalk ever with their links for semi purposes
| of archival and semi purposes of scraping the headlines and
| parsing it to LLM to find a list of vps providers as well
| nikanj wrote:
| And then YC funds a startup who plans to leapfrog the
| competition by doing their own scrape instead using the
| standard data everyone else has
| fartfeatures wrote:
| Good idea and perhaps a standard that means we only have to
| grab deltas or some sort of etag based give me all the database
| dumps after the one I have (or if something has changed).
| tpmoney wrote:
| I'll propose my pie in the sky plan here again. We should
| overhaul the copyright system completely in light of AI and
| make it mostly win-win for everyone. This is predicated on the
| idea that the NIST numbers set is sort of the "hello world"
| dataset for people wanting to learn machine vision and having
| that common data set is really handy. Numbers made up off the
| top of my head/subject to tuning but the basic idea is this:
|
| 1) Cut copyright to 15-20 years by default. You can have 1
| extension of an additional 10-15 years if you submit your work
| to the "National Data Set" within say 2-3 years of the initial
| publication.
|
| 2) Content in the National set is well categorized and cleaned
| up. It's the cleanest data set anyone could want. The data set
| is used both to train some public models and also licensed out
| to people wanting to train their own models. Both the public
| models and the data sets are licensed for nominal fees.
|
| 3) People who use the public models or data sets as part of
| their AI system are granted immunity from copyright violation
| claims for content generated by these models, modulo some
| exceptions for knowing and intentional violations (e.g.
| generating the contents of a book into an epub). People who
| choose to scrape their own data are subject to the current
| state of the law with regards to both scraping and use (so you
| probably better be buying a lot of books).
|
| 4) The license fees generated from licensing the data and the
| models would be split into royalty payments to people whose
| works are in the dataset, and are still under copyright
| protection, proportional to the amount of data submitted and
| inversely proportional to the age of that data. There would be
| some absolute caps in place to prevent slamming the national
| data sets with junk data just to pump the numbers.
|
| Everyone gets something out of this. AI folks get clean data,
| that they didn't have to burn a lot of resources scraping.
| Copyright holders get paid for their works used by AI and
| retain most of the protections they have today, just for a
| shorter time), the public gets usable AI tooling without
| everyone spending their own resources on building their own
| data sets, site owners and the like get reduced bot/scraping
| traffic. It's not perfect, and I'm sure the devil is in the
| details, but that's the nature of this sort of thing.
| Pet_Ant wrote:
| I wish more resources were available legitimately. There is a
| dataset I need for legitimate research that I cannot even find a
| way to contact the repo owners.
|
| Mind you I take effort to not be burdensome by downloading only
| what I need and taking time between each request of a couple
| seconds, and the total data usage is low.
|
| Ironically, I supposed you could call it "AI" what I'm using it
| for, but really it's just data analytics.
| StephenHerlihyy wrote:
| I don't know why anyone would still be trying to pull data off
| the open internet. Too much signal to noise. So much AI influence
| already baked into the corpus. You are just going to be
| reinforcing existing bias. I'm more worried about the day Amazon
| or Hugging Face take down their large data sets.
| saaaaaam wrote:
| MetaBrainz is a fairly valuable "high signal" dataset though.
| smallerfish wrote:
| Bear in mind that some portion of this could be human directed
| research. I'm doing a research project right now with 1000 things
| that I'm building profiles on; to build a full profile requires
| an agent to do somewhere around 100 different site lookups. Where
| APIs exist, I've registered API keys and had the agent write a
| script to query the data in that manner, but that required me to
| be deliberate about it. Non technie plebs aren't likely to be
| directed to use an API by the agent.
| blell wrote:
| Seems a mistake to disable the (I assume) faster-to-generate api.
| Bots will go back to scraping the website itself, increasing
| load.
| hi-wintermute wrote:
| Setting the API to require a token and adding a honeypot to the
| pages themselves seems like a decent solution.
| dannyobrien wrote:
| Metabrainz is a great resource -- I wrote about them a few years
| ago here: https://www.eff.org/deeplinks/2021/06/organizing-
| public-inte...
|
| There's something important here in that a public good like
| Metabrainz would be fine with the AI bots picking up their
| content -- they're just doing it in a frustratingly inefficient
| way.
|
| It's a co-ordination problem: Metabrainz assumes good intent from
| bots, and has to lock down when they violate that trust. The bots
| have a different model -- they assume that the website is
| adversarially "hiding" its content. They won't believe a random
| site when it says "Look, stop hitting our API, you can pick all
| of this data in one go, over in this gzipped tar file."
|
| Or better still, this torrent file, where the bots would briefly
| end up improving the shareability of the data.
| fartfeatures wrote:
| > They won't believe a random site when it says "Look, stop
| hitting our API, you can pick all of this data in one go, over
| in this gzipped tar file."
|
| What mechanism does a site have for doing that? I don't see
| anything in robots.txt standard about being able to set
| priority but I could be missing something.
| jacksnipe wrote:
| It's not great, but you could add it to the body of a 429
| response.
| squigz wrote:
| The mechanism is putting some text that points to the
| downloads.
| bodantogat wrote:
| I feel the pain -- it's very difficult to detect many of the less
| ethical scrapers. They use residential IP pools, rotate IPs, and
| provide valid user agents.
| ranger_danger wrote:
| And many proxy/scraping providers now use real browsers that
| can automatically bypass cloudflare captchas as well, and a bot
| with a real browser similarly won't be clicking on any
| invisible links, so... I am skeptical just how long this will
| make an appreciable difference.
| arjie wrote:
| Someone convinced me last time[0] that these aren't the well-
| known scrapers we know but other actors. We wouldn't be able to
| tell, certainly. I'd like to help the scrapers be better about
| reading my site, but I get why they aren't.
|
| I wish there were an established protocol for this. Say a
| $site/.well-known/machine-readable.json that instructs you on a
| handful of established software or allows pointing to an
| appropriate dump. I would gladly provide that for LLMs.
|
| Of course this doesn't solve for the use-case where the AI
| companies are trying to train their models on how to navigate
| real world sites so I understand it doesn't solve all problems,
| but one of the things I think I'd like in the future is to have
| my own personal archive of the web as I know it (Internet Archive
| is too slow to browse and has very tight rate-limits) and I was
| surprised by how little protocol support there is for robots.
|
| robots.txt is pretty sparse. You can disallow bots and this and
| that, but what I want to say is "you can get all this data from
| this git repo" or "here's a dump instead with how to recreate
| it". Essentially, cooperating with robots is currently under-
| specified. I understand why: almost all bots have no incentive to
| cooperate so webmasters do not attempt to. But it would be cool
| to be able to inform the robots appropriately.
|
| To archive Metabrainz there is no way but to browse the pages
| slowly page-by-page. There's no machine-communicable way that
| suggests an alternative.
|
| 0: https://news.ycombinator.com/item?id=46352723
| saaaaaam wrote:
| As referenced in the article, there absolutely is an
| alternative.
|
| https://metabrainz.org/datasets
|
| Linked to from the homepage as "datasets".
|
| I may be too broadly interpreting what you mean by "machine-
| communicable" in the context of AI scraping though.
| squigz wrote:
| > To archive Metabrainz there is no way but to browse the pages
| slowly page-by-page. There's no machine-communicable way that
| suggests an alternative.
|
| Why does there have to be a "machine-communicable way"? If
| these developers cared about such things they would spend 20
| seconds looking at this page. It's literally one of the first
| links when you Google "metabrainz"
|
| https://metabrainz.org/datasets
| OutOfHere wrote:
| It is nonsense since AI is the nicest thing.
| falloutx wrote:
| Its not just AI scrappers doing it by themselves but now users
| are also being trained to put the link in the claude chat/chat
| gpt and ask it to summarise it. And off course that would show up
| on the website end as a scraper.
|
| In fact firefox now allows you to preview the link and get key
| points without ever going to the link[1]
|
| [1] https://imgur.com/a/3E17Dts
| zx8080 wrote:
| Nothing prevents scraper from creating a free account and sending
| auth token in API requests.
|
| I'm not saying the API changes are pointless, but still, what's
| the catch?
| dherls wrote:
| It's much easier to detect a single account abusing your API
| and ban them/require payment. Trying to police an endpoint open
| to the internet is like playing g whackamole
___________________________________________________________________
(page generated 2026-01-13 23:00 UTC)