Post AlEuNwCJ9fTDM0b8ro by bigiain@aus.social
 (DIR) More posts by bigiain@aus.social
 (DIR) Post #AlEuNvGWcVFWSnYznk by vees@epistolary.org
       2024-08-22T12:23:00Z
       
       0 likes, 0 repeats
       
       What is this nonsense from ByteDance / ByteSpider? If the static page didn't change the first five times you requested it, its not going to change the second five. Time to update my robots.txt?
       
 (DIR) Post #AlEuNwCJ9fTDM0b8ro by bigiain@aus.social
       2024-08-22T22:47:28Z
       
       1 likes, 0 repeats
       
       @vees I’d consider setting up a fail2ban rule - black hole an ip address for 24hrs or so if they break a sensible rate limit. Set it high enough that human browsing and well behaved and wanted bits (like Google Bot perhaps) never hit it. Bytedance/CCP can gtfo…