[HN Gopher] Trapping misbehaving bots in an AI Labyrinth
___________________________________________________________________
Trapping misbehaving bots in an AI Labyrinth
Author : pabs3
Score : 76 points
Date : 2025-03-19 13:31 UTC (3 days ago)
(HTM) web link (blog.cloudflare.com)
(TXT) w3m dump (blog.cloudflare.com)
| windhaven wrote:
| From the post: "It is important to us that we don't generate
| inaccurate content that contributes to the spread of
| misinformation on the Internet, so the content we generate is
| real and related to scientific facts, just not relevant or
| proprietary to the site being crawled."
|
| Okay, why should I care if a crawler that is clearly doing
| something it shouldn't receives misinformation?
| bithavoc wrote:
| I guess if the crawlers can't actually see the trap,
| misinformation would be attributed to your website in case
| model responses expose content attribution tags to end users.
| dns_snek wrote:
| LLMs already make up citations, everyone would assume it was
| just the model spewing nonsense citations.
| mentalgear wrote:
| That's actually a good strategy. It avoids adding more false
| information in the infosphere while de-incentivising the
| crawlers from returning to the site (since they don't find the
| information they are looking for there).
| d4rkn0d3z wrote:
| "When we detect unauthorized crawling..."
|
| How did you do that?
| mog_dev wrote:
| Simple, you add the trapped paths to robots.txt Well behaved
| robots will not crawl them.
| d4rkn0d3z wrote:
| Nifty trick.
| hombre_fatal wrote:
| Just consider how you click around HN versus how your crawler
| would behave if you wanted to crawl every page of HN starting
| from the homepage.
| theyknowitsxmas wrote:
| The toggle doesn't work for me.
|
| API Request Failed: PUT /api/v4/zones/xxx/bot_management (504)
| visarga wrote:
| Like burning down you house so squatters can't use it. A
| poisoning attack that makes communication itself untrusted.
| Wondering if we will get to extreme reciprocal mistrust
| eventually.
|
| This also reminds me of art images doctored to break models if
| they get into the training set, by applying invisible features
| with different semantics.
| hinkley wrote:
| Making requests in bad faith is also an attack on communication
| itself.
| nonchalantsui wrote:
| More like having a series of fake doors and rooms built out in
| front of your home, with most of them leading back outside and
| not into the home.
| DangitBobby wrote:
| Something of a labyrinth for AI crawlers, if you will.
| TeMPOraL wrote:
| More like it being a store, not a house, inside being a
| complex maze of obnoxious ads, inhabited by performance
| artists who distract you so pickpockets can rob you - and
| because locals figured out blind people are immune to this,
| they started paying them to buy stuff for them, and now you
| retrofit the maze to have confusing tactile markings, as to
| direct blind people back out of the store.
|
| The AI paranoia is getting out of hand. Worrying about bots
| spamming you is one thing, but discriminating on crawlers
| specifically because they're from AI companies - and
| conveniently omitting the difference between a bot that's
| crawling (and should obey robots.txt) vs. a bot that's acting
| as _user agent_ (and should _not_ care about robots.txt) -
| isn 't just poisoning communication; it's setting the commons
| on fire.
|
| See also: The Dog in the Manger.
| egypturnash wrote:
| There's been multiple articles on the front page of HN
| about how there's a ton of AI crawlers that are _really bad
| citizens_ - ignoring robots.txt, ignoring cache, re-
| scanning pages multiple times a day. The commons is already
| on fire and it's not because of the actions of any of the
| "locals".
| TeMPOraL wrote:
| How is that different than non-AI crawlers doing the same
| for the past decade or so? Tons of businesses engage in
| site crawling and scrapping, and many of them are bad
| citizens.
|
| My issue isn't with blocking bad-behaving bots - it's
| with singling out LLMs (both training and use), or worse,
| assuming the problem is being associated with AI and not
| bad bot behavior.
| theamk wrote:
| When you say "reciprocal mistrust", what parties do you have in
| mind? Websites not trusting visitors and visitors ot trusting
| websites?
|
| because the latter was already the case, and AI made it much
| worse. Any unfamiliar websife could be AI generated and
| therefore void of original cobtent and full of unverified
| facts.
| bob1029 wrote:
| I'm still struggling with how you would address the bots that
| don't care for speed at all.
|
| How much do we care about stopping crawlers that are slower than
| the average human user? Is this even possible to do given perfect
| wire-level emulation of a typical UA?
|
| Should I expect pages protected by this technology to
| periodically Turing test me?
| theamk wrote:
| this is explicitly "misbehaving bots" - it's for performance,
| not against scraping in general. A bot which is slower than
| average user is likely well-behaving, and therefore is out of
| scope.
| Terr_ wrote:
| I imagine part of the problem is "fast" bots can be disguised
| as many many different slow ones, and it's not always trivial
| to figure out that they're all tentacles of the same malign
| being.
| karaterobot wrote:
| I wonder how these hidden links interact with screen readers. The
| article says they only get served when Cloudflare already
| believes you're a bot, but due to my privacy settings and VPN, a
| lot of Cloudflare-fronted web pages think I'm a bot when I'm just
| browsing around the web. I suppose that having invisible links in
| the page wouldn't hurt me much, but would they bug someone using
| a screen reader? Honestly just wondering.
| momojo wrote:
| In this 'arms race', will this serve as an actual deterrent? Can
| anyone involved in scraping chime in?
| yuvalr1 wrote:
| I am not involved in scraping, but to me this sounds like
| simply another tool in the arsenal. They say it's hard for the
| scraper to realize it has been caught this way because it's not
| being blocked. However, I don't see anything preventing
| scrapers from implementing heuristics to realize that.
| daralthus wrote:
| > No real human would go four links deep into a maze of AI-
| generated nonsense.
|
| sure... sure nobody.
| mdaniel wrote:
| I'm looking forward to all this vibe coding agentic world
| stopping the just facepalm level dupes around here
|
| from _two days ago_ https://news.ycombinator.com/item?id=43421525
| thayne wrote:
| Wouldn't generating the content be much more expensive than the
| crawling done by the bots? How is it economical to deal with bots
| by spending a bunch of resources to feed them fake content.
| palmotea wrote:
| They answered your question in the OP:
|
| > To generate convincing human-like content, we used Workers AI
| with an open source model to create unique HTML pages on
| diverse topics. Rather than creating this content on-demand
| (which could impact performance), we implemented a pre-
| generation pipeline that sanitizes the content to prevent any
| XSS vulnerabilities, and stores it in R2 for faster retrieval.
| We found that generating a diverse set of topics first, then
| creating content for each topic, produced more varied and
| convincing results. It is important to us that we don't
| generate inaccurate content that contributes to the spread of
| misinformation on the Internet, so the content we generate is
| real and related to scientific facts, just not relevant or
| proprietary to the site being crawled.
|
| Personally, I wish they would have generated deliberately
| inaccurate content. It would be a further disincentive to do
| unauthorized crawling. Just as long as the inaccuracies don't
| intersect with typical "misinformation on the Internet"
| inaccuracies, I think it'd be totally fine and ethical (e.g.
| Queen Elizabeth II was the 34th president of the United States,
| the TV show Saved by the Bell aired for 14 seasons with the
| original cast, making it the longest running live action teen
| drama in Canada).
| TeMPOraL wrote:
| Who would you want to target with this? Companies crawling
| the Internet for training data have enough of it to filter
| out such deliberately inaccurate content - if not directly,
| then by betting it'll cancel out in training due to volume of
| data. Now, if you want to mess with people who use LLMs to
| drive search instead of going straight to Google - what's the
| point? Just causally screw with other people? Why not also
| slash tires of their cars while you're at it?
| conartist6 wrote:
| I don't think it matters if the content is accurate, feeding
| it its own slop as if it were new data is deeply damaging to
| the model
| MoonGhost wrote:
| The opposite of it would be an agent which deliberately generates
| expensive but useless requests. Like search. If it detects
| labyrinth.
| callamdelaney wrote:
| There's already a project that does this and it didn't need to
| use AI.
___________________________________________________________________
(page generated 2025-03-22 23:00 UTC)