Post Av0yHgn2PPvClTMYBE by reidrac@mastodon.gamedev.place
 (DIR) More posts by reidrac@mastodon.gamedev.place
 (DIR) Post #Av0yHfu5hhyA13efXE by reidrac@mastodon.gamedev.place
       2025-06-11T06:50:43Z
       
       0 likes, 0 repeats
       
       More aggressive scanning of repos, this time from Google Cloud IPs. Ignoring robots.txt, hitting all the expensive endpoints very fast (not even caching could deal with it; they got some 504s).The ban was in place, but I assume those aren't real users, so I promoted them to permanent.I'm starting to wonder what percentage of my daily traffic is real users.
       
 (DIR) Post #Av0yHgn2PPvClTMYBE by reidrac@mastodon.gamedev.place
       2025-06-11T06:53:48Z
       
       0 likes, 0 repeats
       
       The user agent is Scrapy, from https://scrapy.org -- so I can add that to my list of "bad bots".It is OSS apparently, but it also has "crap" in the name.
       
 (DIR) Post #Av0yHhfz77sFVt4QpE by reidrac@mastodon.gamedev.place
       2025-06-11T07:00:13Z
       
       0 likes, 1 repeats
       
       The main developer is https://www.zyte.com/"Zyte API detects and bypasses blocks in real time, routing requests through the most efficient path without manual tuning. Keep those scrapers flying."Fantastic.