I self-host a small forum on the surface web, it's on a niche topic
(old-school RPGs), so it doesn't get much traffic. I was surprised
this morning when I checked the forum for updates and could not load
it. I hopped on the server to find a load of ~100, with almost all
4GB of memory exhausted and all four CPUs pegged by DB processes. My
shell session was very slow, so rather than try to do any
diagnostics, I just killed apache outright to get some breathing
room. As soon as I did that, the load started to drop.

The forum access logs were huge - 150MB just for today - and in them
I saw what appeared to be otherwise normal forum requests from 5k+
unique IPs, all with "normal" user agent strings, coming in at what
I estimated were hundreds of requests per second. A whois on some of
the top abusers showed the IPs were from Alibaba and Tencent cloud
ranges, so I blocked all of the assigned CIDR blocks I could find
for both cloud providers at the host-based firewall (I blocked 24
networks in the end).

Once that was done, I restarted apache and once again had a working
forum. I could see in the firewall stats that for 18 of the blocked
networks, the dropped packet counts were increasing by over a
hundred per second. I don't think this is a deliberate DoS attack on
such a insignificant site, but most likely AI scraper bots, who are
rotating IPs and randomizing user agents (and of course completely
ignoring robots.txt). It's hard to believe the web is getting even
worse than it was, but here we are.

Recently I've read stories from other webmasters of similar problems
with AI scrapers, and am glad for gopher. While I love the
old-internet style forum, it's still possible to participate in a
community on gopher, at an albeit slower pace (and see my other
gopher site [0] if you're interested in old-school RPGs). I worry
about less technical webmasters, who might give up even running a
forum under such circumstances. I don't blame them, and that makes
me sad for what the web could have been.

[0]: gopher://gopher.smolderingwizard.com