Post B5kYknEqbZRB9J1U12 by bert_hubert@eupolicy.social
 (DIR) More posts by bert_hubert@eupolicy.social
 (DIR) Post #B5kYkmXxB37t0GxE9o by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       RE: https://social.treehouse.systems/@mgorny/116465292654071299There are millions of these single query scrapers. As open projects we should band together and block them preemptively. This is the only way we can put a stop to this.RT: https://social.treehouse.systems/users/mgorny/statuses/116465292654071299
       
 (DIR) Post #B5kYkmrS0Yipyk4nY0 by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       Here are 91,372  IP addresses pretending to be normal browsers, while sending a *single* query to me today. http://berthub.eu/tmp/singlequery-2026-04-26.csv - feel free to block them all!
       
 (DIR) Post #B5kYkn4DF6vkcK2zRI by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       Automated this a bit, and it is fascinating that the bulk of what looks like residential proxies has their language set to zh-CN:
       
 (DIR) Post #B5kYknEqbZRB9J1U12 by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       Meanwhile, the IP addresses used mostly come from Brazil. Iraq, USA, India, Bangladesh, Chile, Argentine, France. This from a site hosting Dutch parliamentary documents.  country | count ---------+------- BR      |  9214 IQ      |  3685 US      |  3597 IN      |  2291 BD      |  2116 CL      |  1822 AR      |  1697 FR      |  1667 CA      |  1577 PK      |  1570 TR      |  1499 ES      |  1449 GB      |  1323 PH      |  1248 IT      |  1246 CO      |  1219 RU      |  1151
       
 (DIR) Post #B5kYknUneGCJwmUDse by bert_hubert@eupolicy.social
       1 likes, 0 repeats
       
       As an experiment, I briefly sent the crawlers with a specific Chinese language setting to https://www.foreignersintaiwan.com/blog/taiwan-is-a-country-heres-why and they *all* vanished.  Very odd & I can't quite explain it. Still, not complaining!
       
 (DIR) Post #B5kYknsYDxCF8RbBtw by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       (the crawlers are back in full force! but they are all getting some great news about Taiwan in response to their queries)
       
 (DIR) Post #B5kYkoJ8d6SoSu2QLI by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       Their runs look like the graph below. I assume they task their network of residential proxies with a massive list of URLs to retrieve, and as the list gets smaller proxies run out of work:
       
 (DIR) Post #B5kYkoiJ7Wb3ixoWZc by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       So I did another experiment and redirected the crawlers to a ~10GB file. This was a brief experiment, but it is reassuring to know my setup can stream 300+Mbit/s without ill effects!
       
 (DIR) Post #B5kYkp63hDayucvUau by bert_hubert@eupolicy.social
       0 likes, 0 repeats
       
       After some suggestions I'm now serving a gzip-bomb, but since these IP addresses only visit me once, I don't know if they suffer ill effects. But rest assured knowing many dozens of gigabits of content are now being served every second!