[HN Gopher] Using lots of little tools to aggressively reject th...
___________________________________________________________________
Using lots of little tools to aggressively reject the bots
Author : archargelod
Score : 145 points
Date : 2025-05-31 08:06 UTC (14 hours ago)
(HTM) web link (lambdacreate.com)
(TXT) w3m dump (lambdacreate.com)
| loloquwowndueo wrote:
| Nice - I like that most of the ai scraper bot blocking was done
| using Nginx configuration. Still, once fail2ban was added to the
| mix (meaning: additional service and configuration), I wonder if
| considering something like Anubis (https://anubis.techaro.lol/)
| would have been more automatic. I've seen Anubis verification
| pages pop up more frequently around the web!
| rovr138 wrote:
| FWIW, the reason I like their approach is that fail2ban is
| still lean, works off of the same logs, and doesn't start with
| the requirement to affect everyone's experience due to bad
| actors.
| nulbyte wrote:
| Making the visitor put up with your bot because you don't like
| someone else's is just bizarre to me. I disable js by default,
| so Ive been seeing this more and more. It's all rubbish these
| days.
| Proofread0592 wrote:
| It is nice that the AI crawler bots honestly fill out the `User-
| Agent` header, I'm shocked that they were the source of _that_
| much traffic though. 99% of all websites do not change often
| enough to warrant this much traffic, let alone a dev blog.
| grishka wrote:
| They also respect robots.txt.
|
| _However_ , I've also seen reports that after getting blocked
| one way or another, they start crawling with browser user-
| agents from residential IPs. But it might also be someone else
| misrepresenting their crawlers as
| OpenAI/Amazon/Facebook/whatever to begin with.
| rovr138 wrote:
| We ended up writing similar rules to the article. It was just
| based on frequency.
|
| While we were rate limiting bots based on UA, we ended up
| also having to apply wider rules because traffic started
| spiking from other places.
|
| I can't say if it's the traffic shifting, but there's
| definitely a big amount of automated traffic not identifying
| itself properly.
|
| If you look at all your web properties, look at historic
| traffic to calculate <hits per IP> in <time period>. Then
| look at the new data and see how it's shifting. You should be
| able to identify the real traffic and the automated very
| quickly.
| cratermoon wrote:
| > They also respect robots.txt
|
| All the reports I've heard from organizations dealing with AI
| crawler bots say they are not honest about their user agent
| and do not respect robots.txt
|
| "It's futile to block AI crawler bots because they lie,
| change their user agent, use residential IP addresses as
| proxies, and more." https://xeiaso.net/notes/2025/amazon-
| crawler/
| eesmith wrote:
| Further info along the same lines at
| https://drewdevault.com/2025/03/17/2025-03-17-Stop-
| externali...
|
| > If you think these [AI] crawlers respect robots.txt then
| you are several assumptions of good faith removed from
| reality. These bots crawl everything they can find,
| robots.txt be damned, including expensive endpoints like
| git blame, every page of every git log, and every commit in
| every repo, and they do so using random User-Agents that
| overlap with end-users and come from tens of thousands of
| IP addresses - mostly residential, in unrelated subnets,
| each one making no more than one HTTP request over any time
| period we tried to measure - actively and maliciously
| adapting and blending in with end-user traffic and avoiding
| attempts to characterize their behavior or block their
| traffic.
|
| Sourcehut (the site described) used Anubis before swithing
| "to go-away, which is more configurable and allows us to
| reduce the user impact of Anubis (e.g. by offering
| challenges that don't require JavaScript, or support text-
| mode browsers better)."
| https://sourcehut.org/blog/2025-05-29-whats-cooking-q2/
| immibis wrote:
| However, there's no evidence those bots are really OpenAI
| et al.
| grishka wrote:
| Speaking from my own experience, which is admittedly
| limited, but still -- I had AI bots crawling my fediverse
| server, I added them to my robots.txt as "Disallow: *",
| they stopped.
|
| As I said, it might be someone else entirely using
| OpenAI/Amazon/Meta/etc user agents to hide their real
| identity while ignoring robots.txt. What's to stop them?
| People blame those companies anyway.
| reconnecting wrote:
| Creator of tirreno [1] here.
|
| While our platform is primarily designed for live, logged-in
| users, it also works well for bot detection and blocking.
|
| We anonymize IP addresses by replacing the last octet with an
| asterisk, effectively grouping the same subnet under a single
| account. You can then use the built-in rule engine to
| automatically generate blacklists based on specific conditions,
| such as excessive 500 or 404 errors, brute-force login attempts,
| or traffic from data center IPs.
|
| Finally, you can integrate tirreno blacklist API into your
| application logic to redirect unwanted traffic to an error page.
|
| Bonus: a dashboard [2] is available to help you monitor activity
| and fine-tune the blacklist to avoid blocking legitimate users.
|
| [1] https://github.com/tirrenotechnologies/tirreno
|
| [2] https://play.tirreno.com/login (admin/tirreno)
| reconnecting wrote:
| We also have work in progress to block bots based on publicly
| available IP ranges through the same dashboard. Any suggestions
| are welcome.
| mindslight wrote:
| > _We anonymize IP addresses by replacing the last octet with
| an asterisk, effectively grouping the same subnet under a
| single account_
|
| So as a user, not only do I have to suffer your blockwall's
| false positives based on "data center IPs" (ie most things that
| aren't naively browsing from the information-leaking address of
| a last-mile connection like some cyber-bumpkin). But if I do
| manage to find something that isn't a priori blocked (or manage
| to click through 87 squares of traffic lights), I still then
| get lumped in with completely unrelated address-neighbors to
| assuage your conscience that you're not building a user
| surveillance system based on nonconsentually processing
| personal information.
|
| Just please make sure you have enough of a feedback process
| that your customers can see that they are losing real customers
| with real dollars.
| reconnecting wrote:
| You're right, blanket blocking based on IP ranges (like TOR
| or DC) often creates false positives and punishes privacy-
| conscious users. Therefore, unlike the traditional way of
| blocking an IP just because it is from a data center, tirreno
| suggests using a risk-based system that takes into account
| dozens or hundreds of rules.
|
| As in my example, if the IP is from a data center and creates
| a lot of 404 errors, send it to a manual review queue or to
| automatic blocking (not recommended).
|
| Personally, I prefer to manually review even bot activity,
| and tirreno, even if it's not directly designed for bot
| management, works great for this, especially in cases when
| bad bots are hidden behind legitimate bot UA's.
| wincy wrote:
| I keep forgetting that metacritic just doesn't work with
| any VPN I've tried, but in a way that's really annoying (it
| loads the site then never loads any scores or anything you
| actually want to see).
|
| It's so annoying.
| reconnecting wrote:
| I was literally shocked that one European PaaS blocked
| all TOR exit nodes for all their clients web resources
| because of security concerns instead of setting up proper
| online fraud prevention.
| nixgeek wrote:
| Increasing amounts of traffic are flowing through Google VPN
| if you're on a Pixel, and on Apple, there is iCloud Private
| Relay. I'd have thought the address-neighbors issue would be
| especially likely to catch out these situations?
|
| Overly simplistic solutions like this absolutely will
| actively cost you real customers and real revenue.
| reconnecting wrote:
| tirreno has a rule for Private Relay/Starlink IP addresses,
| which can be configured as either a positive or a negative
| signal, depending on specific needs.
|
| There is a extra cost only if you choose to block users
| automatically, regardless of the tool used.
| rovr138 wrote:
| Great article and sleuthing to find the information.
|
| I know you're processing them dynamically as they come in and
| break the rules. But if you wanted to supplement the list, might
| be worth sourcing the ones from https://github.com/ai-robots-
| txt/ai.robots.txt at some frequency.
| sneak wrote:
| You don't have to fend off anything, you just have to fix your
| server to support this modest amount of traffic.
|
| Everyone else is visiting your site for entirely self-serving
| purposes, too.
|
| I don't understand why people are ok with Google scraping their
| site (when it is called indexing), fine with users scraping their
| site (when it is called RSS reading), but suddenly not ok with AI
| startups scraping their site.
|
| If you publish data to the public, expect the public to access
| it. If you don't want the public (this includes AI startups) to
| access it, don't publish it.
|
| Your website is not being misused when the data is being
| downloaded to train AI. That's literally what public data is
| _for_.
| red369 wrote:
| Is it because people viewed it as Google scraping the site to
| make an index so that people could find the site, while the AI
| scraping is intended so people won't need to visit the site at
| all?
| ehutch79 wrote:
| Also, google is relatively considerate when crawling.
| DamonHD wrote:
| Usually though from time to time over many many years from
| when G was still wet behind the ears I have had to write
| specific defences effectively just to deal with it.
| ehutch79 wrote:
| Yeah, "Relatively" is doing a lot of heavy lifting there.
|
| I remember people with lax security and a tags that went
| to deletion endpoints... we all learned a lot back in the
| day.
| owebmaster wrote:
| The AI apps (namely chatgpt and Claude) are evolving to
| display external data with widgets that will potentially
| drive more traffic than google has been doing for a long
| time. Might be worth changing focus as SEO killed google.
| eitland wrote:
| Kagi already behave really nicely, giving a small summary
| with numbered footnotes that links to where Kagi got its
| information from.
|
| Which IMO shows that is already possible to do this and and
| it has been for a while already.
| bentley wrote:
| > I don't understand why people are ok with Google scraping
| their site (when it is called indexing), fine with users
| scraping their site (when it is called RSS reading), but
| suddenly not ok with AI startups scraping their site.
|
| I'm pretty open-minded about AI, and have no visceral objection
| to AI scraping in theory. However... the rise of AI scrapers is
| the first time in twenty years of running tiny websites that my
| servers--all my servers, consistently--hit CPU and bandwidth
| usage levels far above baseline, and even brush against my
| hosts' monthly limits. When I check server logs, the traffic is
| largely bots repeatedly trawling infinite variations of version
| control history pages. Googlebot never crawled these enough to
| become a problem; Internet Archive never crawled these enough
| to become a problem. But the current scrapers are, and
| literally everyone I talk to who hosts their own VCS webservers
| is having the same problem right now. Many are choosing to
| reduce the problem by injecting Cloudflare or Anubis, which I
| as a user hate, and refuse to do myself--but what alternative
| can I propose to them? People running these servers are just
| users of Gitea and CGit and whatnot, they're not capable of
| rewriting the underlying tools to reduce the CPU usage of diff
| generation.
| sneak wrote:
| Seems to me like standard dumb rate limiting (max 100r/min or
| like 250r/5min) per IP would solve the problem easily without
| affecting human users at all.
|
| Surprised this isn't a feature in Gitea already (even though
| it is better done in the reverse proxy web server).
| DamonHD wrote:
| Some scummy scrapers are snowshoeing their way through
| scads of (residential) IPs, just for one example.
|
| Assuming that there are simple universal solutions, eg
| based on IP or UA, suggests that you have been lucky enough
| not to deal with many varieties of this misbehaviour.
| isaacremuant wrote:
| Your expectations are arbitrary and you're not hosting it so
| they can take whatever measures they want. Just like some
| publish "public websites" and immediately show a "subscribe"
| button if you want to see more. It's their prerogative.
| loloquwowndueo wrote:
| Tell me you haven't had your site scraped by an abusive AI
| crawler without telling me you haven't had your site scraped by
| an abusive AI crawler.
|
| Google and traditional crawlers are _fine_. This new breed of
| AI crawlers have no scruples, no rate limits, and will just
| load pages and follow links as fast as they can, very likely
| overwhelming smaller web servers in the process.
|
| I too have blocked many of them to avoid them filling my logs
| with nonexistent page accesses, for example.
| sneak wrote:
| I run a public Gitea at https://git.eeqj.de and while I see
| it getting scraped, I haven't noticed any effects whatsoever
| from the additional load. No mitigations have been necessary.
| tomsmeding wrote:
| The OP notes at some point that the scrapers were
| generating lots of source tarballs individual commits. That
| takes server resources, and a small server can't sustain
| many of those requests per second. I wonder why you
| (apparently) don't suffer from this -- in fact one can even
| wonder why such tarballs are generated in the first place,
| it sounds very pointless.
| bob1029 wrote:
| > The OP notes at some point that the scrapers were
| generating lots of source tarballs individual commits.
|
| I refuse to believe OAI has a scraper that is designed to
| push commits. It is obvious to me that we are conflating
| other forms of malicious use with the boogeyman.
| kassner wrote:
| They are talking about the standard "Download zip", which
| can also be done for commits individually. They aren't
| precomputed, which causes the load.
| OtherShrezzing wrote:
| >I don't understand why people are ok with Google scraping
| their site (when it is called indexing), fine with users
| scraping their site (when it is called RSS reading), but
| suddenly not ok with AI startups scraping their site.
|
| Google and the publisher share the value derived from that
| scraping event (somewhat equitably) when a user clicks a link
| in Google's index. OpenAI, Anthropic, and co extract all of the
| value, and incur costs to the publisher, from their scraping
| events.
| treve wrote:
| The answer is attribution. also putting things public does not
| imply no copyright or license
| vachina wrote:
| I've turned off logging on my servers precisely because it's
| growing too quickly due to these bots. They're that relentless,
| and would fill every form, even access APIs otherwise accessible
| only by clicking around the site. Anthropic, openAI and Facebook
| are still scraping to this day.
| davidmurdoch wrote:
| > even access APIs otherwise accessible only by clicking around
| the site
|
| How else?
| eGQjxkKF6fif wrote:
| Would you mind sharing information on these crawlers accessing
| APIs only usable for clicking around on websites?
|
| And to clarify,
|
| It's a part of the UI or something and only a human should be
| pressing it, and there's no other way to access that API or
| something?
|
| AI agents exist now, there is virtually no way to distinguish
| between real user and bot if they mimic human patterns.
| atomman747 wrote:
| I get that the author "doesn't care anymore", but I saw Google,
| ripe.net, and semrush in the banned IPs.
|
| Of these, I certainly wouldn't ban Google, and probably not the
| others, if I wanted others to see it and talk about it.
|
| Even if your content were being scraped for some rando's AI bot,
| why have a public site if you don't expect your site to be used?
|
| Turning the lights off on the motel sign when you want people to
| find it is not a good way to invite people in.
| DamonHD wrote:
| Semrush misbehaved so badly for such a long time with various
| levels of incompetence that I have special notes in my
| robots.txt files going back at least 8 years and eventually got
| someone in legal to pay attention. I see zero value to me or
| any potential users of mine in letting an 'SEO' firm lumpishly
| trample all over my site barging out real visitors. And some of
| Semrush's competitors were just as bad.
|
| The current round of AI nonsense also very poor. Again had to
| send legal notes to investor relations and PR depts in at least
| one well-known case, as well as all the technical measures, to
| restore some sort of decorum.
| akudha wrote:
| My experience with them has been the same. On one of my
| employer's websites, the top three bots were Google,
| Bytedance and Semrush. It is a small website for a niche
| audience, not even in English, and changes very infrequently
| (like once or twice a quarter). That did not stop these three
| bots from hammering the site, every second
| phyzome wrote:
| Because the bot requests are consuming significant amounts of
| bandwidth, memory, CPU, and disk space. Like the intro says,
| it's just rude, and there's no reason to serve traffic to
| harvesters like that.
|
| Google also runs an AI scraper, which might be what you saw
| represented there?
| globie wrote:
| From the article it's sure starting to seem like people
| across the internet are just starting to realize what happens
| when you don't have just 3-4 search engines responsible for
| crawling for data anymore. When data becomes truly
| democratized, its access increases dramatically, and we can
| either adjust or shelter ourselves while the world moves on
| without us.
|
| Did Google never ever scrape individual commits from Gitea?
| HumanOstrich wrote:
| > When data becomes truly democratized...
|
| That is not at all what is happening.
| globie wrote:
| I know, we're locking everything down behind WAFs and
| repeating captchas so only attested identities can get
| access in the end.
| HumanOstrich wrote:
| Yep, everyone is building their own little walled gardens
| instead of adapting.
| itsafarqueue wrote:
| Friend, this IS them adapting.
| HumanOstrich wrote:
| More like maladapting.
| taormina wrote:
| There are also bad actors who pretend to be the Google scraper.
| Google once upon a time had a reputation for respectfully
| scrapping, but if he's getting the traffic he needs with or
| without the Googlebot, why should he care?
| BlueTemplar wrote:
| You haven't started educating people in the last ten years that
| it's just not acceptable to use Google any more ?
|
| Especially when this happens ?
|
| _Google is using AI to censor independent websites like mine_
|
| https://news.ycombinator.com/item?id=44124820
|
| Sure sounds like we've reached the point where it's more of a
| liability !
| miladyincontrol wrote:
| A lot of less scrupulous crawlers just seem to imitate the big
| ones. I feel a lot of people make assumptions because the user
| agent has to be true, right?
|
| My fave method is still just to have bait info in robots.txt that
| gzip bombs and autoblocks all further requests from them. Was
| real easy to configure in Caddy and tends to catch the worst
| offenders.
|
| Not excusing the bot behaviours but if a few bots blindly take
| down your site, then an intentionally malicious offender would
| have a field day.
| horsawlarway wrote:
| Your last comment feels pretty apt.
|
| Maybe I'm just a different generation than the folks writing
| these blog posts, but I really don't understand the fixation on
| such low resource usage.
|
| It's like watching a grandparent freak out over not turning off
| an LED light or seeing them drive 15 miles to save 5c/gallon on
| gas.
|
| 20 requests per second is just... Nothing.
|
| Even if you're dynamically generating them all (and
| seriously... Why? Time would have been so much better spent
| fixing that with some caching than this effort) it's just not
| much demand.
|
| I get the "fuck the bots" style posts are popular in the
| Zeitgeist at the moment, but this is hardly novel.
|
| There are a lot more productive ways to handle this that waste
| a lot less of your time.
| rnmg wrote:
| Can you expand on the better ways of handling bots? Genuinely
| curious.
| layer8 wrote:
| He's saying that a modern web server setup should be able
| to handle the traffic without any bot-specific handling.
| Personally, I don't follow.
| vladvasiliu wrote:
| I sympathize with the general gist of your post, but I've
| seen many a bot generate more traffic than legitimate users
| on our site.
|
| Never had any actual performance issue, but I can see why a
| site that expects generally a very low traffic rate might
| freak out. Could they better optimize their sites? Probably,
| I know ours sucks big time. But in the era of autoscaling
| workloads on someone else's computer, a misconfigured site
| could rack up a big ass bill.
| eGQjxkKF6fif wrote:
| It's not fuck the bots, it's fuck the bot owners for using
| the websites as they want, and not at minimum, asking. Like
| 'hey cool if I use this tool to interact with your site for
| this and that reason?'
|
| No, they just do it. So that can scrape data, which at this
| point in time for AI which has hit the cap on what it can
| consume knowledge wise, scrapes it because live updates and
| new information is most valuable to them.
|
| So they will find tricky, evil ways to hammer resources that
| we as site operators own; even minimally to use site data to
| their profit, their success, their benefits while blatantly
| saying 'screw you' as they ignore robots.txt or pretend to be
| legitimate users.
|
| There's a digital battle field going on. Clients are coming
| in as real users using IP lists like from
| https://infatica.io/
|
| A writeup posted to HN about it
|
| https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-
| Part-...
|
| A system and site operator has every right to build the tools
| they want to protect their systems, data, and have a user
| experience that benefits their audiences.
|
| Your points are valid and make sense, but; it's not about
| that. It's about valuing authentic works, intellectual
| properties, and some dweeb that wants to steal it doesn't get
| to just run their bots against resources at others
| detriments, and their benefits.
| spookie wrote:
| A friend of mine had over 1000 requests/sec on his Gitea at
| peaks. Also, you aren't taking into account some of us don't
| have a "server", just some shitbox computer in the basement.
|
| This isn't about mere dozen requests. It gets pretty bad. It
| also slows down his life.
| nickpsecurity wrote:
| Some of us have little money or optimized for something else.
| I spent a good chunk of this and last year with hardly any
| groceries. So, even $30 a month in hosting and CDN costs was
| large.
|
| Another situation is an expensive resource. This might be
| bandwidth hogs, CPU heavy, or higher licensing per CPU's in
| databases. Some people's sites or services dont scale well or
| hit their budget limits fast.
|
| In a high-security setup, those boxes usually have limited
| performance. It comes from the runtime checks, context
| switches, or embedded/soft processors. If no timing channels,
| one might have to disable shared caches, too.
|
| Those systems run slow enough that whatever is in front
| usually needs to throttle the traffic. We'd want no wasted
| traffic given their cost ranges from $2,000 / chip (FPGA) to
| six digits a system (eg XTS-500 w/ STOP OS). One could say
| the same if it was a custom or open-source chip, like Plasma
| MIPS.
|
| Many people might be in the poor category. A significant
| number are in the low-scalability category. The others are
| rare but significant.
| ThePinion wrote:
| Can you further elaborate on this robots.txt? I was under the
| impression most AI just completely ignores anything to do with
| robots.txt so you may just be hitting the ones that are maybe
| attempting to obey it?
|
| I'm not against the idea like others here seem to be, I'm more
| curious about implementing it without harming good actors.
| hananova wrote:
| Back in the 90's, me as a dumb kid got a phone call from my ISP
| telling me that my system had been enlisted in a botnet and that
| they were turning off my connection until I fixed it.
|
| Maybe it's time to once again block entire ASN's of ISP's that
| tolerate being used as residential proxies, and this shit might
| end.
| NitpickLawyer wrote:
| The residential proxies are not offered by ISPs but by world-
| wide users (knowingly or not) installing dubious software in
| exchange for access / pennies / whatever. There was a pretty
| good article about it a while ago on hn.
| aorth wrote:
| Yep! Eye opening report on residential proxies from Trend
| Micro.
|
| https://www.trendmicro.com/vinfo/us/security/news/vulnerabil.
| ..
| vhcr wrote:
| And? If your network allows malware you should be blocked.
| RamRodification wrote:
| I could be wrong, but the comment for the rate limiting (1r/s)
| doesn't seem to line up with the actual rate limit being set
| (5r/s). # Define a rate limit of 1 request per
| second every 1m limit_req_zone $binary_remote_addr
| zone=krei:10m rate=5r/s;
| tasuki wrote:
| I wonder, why do people maintain personal git forges?
|
| Is it so others can submit issues and pull requests? Do others
| actually bother to create accounts at your personal git forge so
| they can contribute?
| kassner wrote:
| Personal choices. I want to selfhost all my stuff, and avoid
| relying on GitHub, and also having some codebases not being
| used in AI training. I achieve the latter by not having my
| forge public, so I don't have much of a dog in this fight.
|
| I could just have a directory in another server, but I like
| being able to link to a specific file/line/commit, and I prefer
| viewing the code in the browser in some situations. Forgejo
| takes the same effort to selfhost as gitweb on my setup, so why
| not?
| tasuki wrote:
| Ok, fair.
|
| I have several remotes for my projects: one on a machine I
| control, another on GitHub, so I can send links to people
| etc.
| nullc wrote:
| I can't use mouser or digikey anymore thanks to anti-crawler
| crusading.
| opan wrote:
| I have always gotten the short end of the stick with this stuff
| as well. Captchas are annoying, Cloudflare challenges are
| usually not passable, etc. For some sites I use regularly,
| they'll log me out and stop me logging in a few times per year
| and then I have to tweet at them about it, which is usually
| followed by something about them having just turned up their
| bot protection settings, but then often they'll fix whatever
| they did and I can get back in.
|
| Would it be so crazy to just let it happen? When ordinary users
| are suffering as much or more than the "criminal" targets,
| maybe it's not worth it. I understand there are cases where
| bandwidth usage would become too much, but there's gotta be a
| better way. Maybe let some bots through so that fewer humans
| are blocked.
| ThomW wrote:
| It ticks me off that bots no longer respect robots.txt files at
| all. The authors of these things are complete assholes. If you're
| one of them, gfy.
| aswegs8 wrote:
| Ah, good old botkilling. Gotta love those.
| immibis wrote:
| I consider the disk space issue a bug in Gitea. When someone
| downloads a zip, it should be able to stream the zip to the
| client, but instead it builds the zip in temporary space, serves
| it to the client, and doesn't delete it.
|
| I solved it by marking that directory read-only. Zip downloads,
| obviously, won't work. If someone really wants one, they can
| check out the repository and make it theirself.
|
| If I really cared, of course I'd fix the bug or make sure there's
| a way to disable the feature properly or only enable it for
| logged-in users.
|
| Also I server-side redirect certain user-agents to
| https://git.immibis.com/gptblock.html . This isn't because they
| waste resources any more but just because I don't like them, what
| they're doing is worthless anyway, and because I can. If they
| really want the data of the Git repository they can clone the Git
| repository instead of scraping it in a stupid way. That was
| always allowed.
|
| 8 requests per second isn't that much unless each request
| triggers intensive processing and indeed it wasn't a load on my
| little VPS other than the disk space bug. I blocked them because
| they're stupid, not because they're a DoS.
| dakiol wrote:
| OpenAI is the biggest offender according to my server's logs.
| Nevertheless we keep praising them over here. We are so stupid.
| DamonHD wrote:
| Who's "we"?
| nickpsecurity wrote:
| My sites get 10-20k requests a day. Mostly AI scrapers. One thing
| I noticed is many look for specific, PHP pages. If you dont use
| PHP, you might be able to autoblock any IP requesting PHP pages.
| If you have PHP, block those requesting pages you dont have.
|
| Some of us are happy to train AI's but want to block overload.
| For instance, I'm glad they're scraping pages about the Gospel
| and Biblical theology. It might help to put anything large that
| you dont want scraped into specifi directories. Then, upon
| detecting a bot, block the IP from accessing those.
|
| In my case, I also have a baseline strategy to deal with a large
| number of requests. That's text only, HTML/CSS presentation,
| other stuff externally hosted, and BunnyCDN with Perma-Cache
| ($10/mo + 1 penny / GB). The BunnyCDN requests go to $5/mo. VM's
| on Digital Ocean. I didnt even notice AI scrapers at first since
| (a) they didn't affect performance and (b) a month of them
| changed my balance from $30 to $29.99.
|
| (Note to DO and Bunny team members that may be here: Thanks for
| your excellent services.)
| AStonesThrow wrote:
| Chris Siebenmann has shared many thoughts about crawlers, aka
| spiders.
|
| https://utcc.utoronto.ca/~cks/space/?search=spider
|
| https://utcc.utoronto.ca/~cks/space/?search=crawlers
|
| It's interesting how in 2005 he was advocating for playing
| nicely, because yes, search engines were hypothetically driving
| traffic to sites they crawled. And there don't seem to be a lot
| of complaints about performance hits. Though typically, his
| comments and observations are restricted to his blog's corner of
| his department.
___________________________________________________________________
(page generated 2025-05-31 23:00 UTC)