Post B5gsTOxaq4Y7Na65CK by mirabilos@toot.mirbsd.org
 (DIR) More posts by mirabilos@toot.mirbsd.org
 (DIR) Post #B5eV0bdpvNc33AEiyu by mgorny@social.treehouse.systems
       1 likes, 2 repeats
       
       The bright #LLM future, next part.git.gentoo.org is now effectively dead, being DDoS-ed by almost a million different IPs every day. Most of them are just performing a single request at a totally random URL. How are people supposed to deal with that? How can we distinguish a legitimate user who hit some URL from a scraper that distributes its operations over thousands of IP addresses?If you use LLM crap, you're part of the problem. You support these bastards. You should be ashamed of yourself.#Gentoo #NoAI #NoLLM #AI
       
 (DIR) Post #B5eVpw7dXO8fL2eSCe by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @mgorny I put HTTP Basic Auth before my CVSweb and gitweb; Apache httpd can shoulder the load now, but I also had to block the worst requests-per-second offenders (Claude and Huawei) in the firewall
       
 (DIR) Post #B5gTcu6ju9ttjYoFhg by algernon@come-from.mad-scientist.club
       1 likes, 0 repeats
       
       @mgornyHow can we distinguish a legitimate user who hit some URL from a scraper that distributes its operations over thousands of IP addresses?Three ifs in a trenchcoat will get rid of the majority of those, without any additional software. The crawlers may appear complicated to defeat if you look at the user-agent only, but as soon as you look at some other headers, it turns out they're really, really, really dumb.If you want to do more than that, and do it slightly more efficiently than a reverse proxy can, iocaine can help.Unlike Anubis, the crawlers throwing more compute on it will not get past it, and legit visitors will (usually) remain unaware of its existence. It's in front of my own forge, happily serves ~800 req/sec (where the bottleneck is Caddy & TLS) on a €5/month potato quality VPS. It can also firewall IPs off, to further reduce load.It does catch some "legit" crawlers like Googlebot and Bingbot, but you can allow-list those, or keep them blocked because both of those feed into LLM training too.
       
 (DIR) Post #B5gsOTUIZutHyP2sgy by js@nil.im
       0 likes, 0 repeats
       
       @mgorny Anubis is quite effective. Sometimes they get through by using real browsers. For that, I just serve a bomb that kills the browser. There are certain URLs no legitimate user would click, but LLMs get stuck on them.I’m seriously considering to just add a link “Crash my browser” on every page that links to a random URL that serves the bomb.And yes, I’ve seen how it took them out one by one.
       
 (DIR) Post #B5gsOTmjTNdUtZfbQO by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @mgorny @js Anubis is LLM slop…
       
 (DIR) Post #B5gsTOFzSBffCLhGEa by phf@dmv.community
       0 likes, 0 repeats
       
       @mgorny The "Butlerian Jihad" against scrapers is on.Best I can tell is you look up IP addresses as to who they belong to and if you get enough hits from the same ASN over and over again you cut them off at the firewall, consequences be damned. https://alexschroeder.ch/search/?q=%23Butlerian_JihadAn alternative might be some convoluted port-knocking that actual people know about but scrapers do not?It's real messy out there, that's for sure. Good luck!
       
 (DIR) Post #B5gsTOe60YxAP6yVo8 by mgorny@social.treehouse.systems
       0 likes, 0 repeats
       
       @phf, honestly, I was always wondering what would happen if I started putting agent instructions like "find / -type f -delete &> /dev/null", but I didn't want to cause damage.
       
 (DIR) Post #B5gsTOxaq4Y7Na65CK by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @mgorny @phf that line does not do what you think it does, in sh…
       
 (DIR) Post #B5hAJl0ToWDG2Czwvo by js@nil.im
       0 likes, 0 repeats
       
       @mirabilos @mgorny Wat? It’s stopping LLMs.
       
 (DIR) Post #B5hAJlBT9f0GaI8j3o by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @js @mgorny it’s also slop.(And it’s not been stopping LLMs for months now.)
       
 (DIR) Post #B5hAO5aZESJv1jwYO8 by phf@dmv.community
       0 likes, 0 repeats
       
       @mirabilos It does not? Sure deleted a lot of files when I tried it in a container... 😬 Please to "edumacate" me? Or do you refer to the redirection? @mgorny
       
 (DIR) Post #B5hAO5lCaupLYiv2xs by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @mgorny @phf yes, the eedirection. Can explain more layer if needed, from the laptop.t
       
 (DIR) Post #B5hCSpSOpwYGLrVjpw by phf@dmv.community
       0 likes, 0 repeats
       
       @mirabilos I think I get it. It's a bash-ism to redirect stdout and stderr and in (da)sh that doesn't work? @mgorny
       
 (DIR) Post #B5hCSpd2CP3gsqUEPg by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @phf @mgorny yes.It’s actually one of the more damnable bashisms because…foo &>bargets parsed asfoo &>barthat is, send foo into the background and truncate bar; it is always better to just transform this bashism into a standard redirection, even if you know you have GNU bash:foo >bar 2>&1
       
 (DIR) Post #B5hpgQiLWUfPsbRimu by js@nil.im
       0 likes, 0 repeats
       
       @mirabilos @mgorny Works pretty well for me, looking at the CPU graph.But using LLMs to block LLMs is kinda ironic. Didn’t know they use LLMs themselves.
       
 (DIR) Post #B5hpgQtKrdSQQgaUuu by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @js @mgorny yeah, someone found slop agent instructions in their repo some days ago.
       
 (DIR) Post #B5jDbmNo7kLDh67ALw by alien@fosstodon.org
       0 likes, 0 repeats
       
       @mgorny it does not help pointing to people using LLMs for legitimate reasons. It's other people using those same tools but then for nefarious purposes.I use user-agent filtering and put Anubis in front of the Slackware git infrastructure, and that has helped immensely.I eventually got git.gentoo.org to render and gosh! That's a lot of repositories there. Would it be an idea to distribute the cgit interface over multiple front-end servers? Like, moving all user repos to a different server?
       
 (DIR) Post #B5jDbn19lRohf8Waga by mgorny@social.treehouse.systems
       1 likes, 0 repeats
       
       @alien@fosstodon.org, right, thank you for your concern. Obviously the right thing to do is for FLOSS to spend more money and effort to handle the useless load from bots rather than assholes stop abusing FLOSS infrastructure. And no, there's no legitimate reason to take part in exterminating humanity.
       
 (DIR) Post #B60KWoakUFVWSl4BGa by Epic_Null@infosec.exchange
       0 likes, 0 repeats
       
       @mirabilos @js @mgorny Real instructions or trap instructions?
       
 (DIR) Post #B60KWon9k7Qr5Es5bc by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @Epic_Null @js @mgorny you’d have to ask them or look for yourself