Post AxQMSW52VWebGLbuiG by m0xEE@nosh0b10.m0xee.net
 (DIR) More posts by m0xEE@nosh0b10.m0xee.net
 (DIR) Post #AxQMSTKsiAXkkc01UO by neverpanic@chaos.social
       2025-08-21T21:07:03Z
       
       0 likes, 0 repeats
       
       I've had it with the aggressive #AI #crawlers now. Some bot  has been hitting #MacPorts with a legitimate enough user agent that I can't block it without also blocking users.Yesterday, it sent 377k requests (62 % of the total), 369k to URLs forbidden in robots.txt from 274k unique IPs. Most of it for content that could be analyzed quicker using `svn checkout` or `git clone`.Dynamic content on the #web is broken. There's just no way to do that anymore. What a waste of energy.
       
 (DIR) Post #AxQMSV16SfuXxqlFwG by monk@social.unboiled.info
       2025-08-22T07:35:59.686073Z
       
       0 likes, 0 repeats
       
       @neverpanic> Dynamic content on the #web is broken. There's just no way to do this anymore.If that's how we stop building apps out of pages, all hail Ai crawlers!
       
 (DIR) Post #AxQMSW52VWebGLbuiG by m0xEE@nosh0b10.m0xee.net
       2025-08-22T09:50:22Z
       
       0 likes, 0 repeats
       
       @monk@social.unboiled.info @neverpanic@chaos.socialSadly, this mostly causes issues for dynamic content that the server returns, not for "Web apps" — those are mostly unaffected. So it is the opposite of good 😩
       
 (DIR) Post #AxQMSbnXFWDWznT1GK by neverpanic@chaos.social
       2025-08-21T21:09:11Z
       
       0 likes, 0 repeats
       
       I'd love to spend my time on something more productive. Instead, I'm dealing with this shit on a daily basis now.And I've been reading the bots have now learned to also solve Anubis challenges, so now the only defense is to make it so expensive to solve these challenges that it drains the proxies' batteries, but that will also make it very bad user experience for everybody else.