Post AdVfCjMWgrZcTRU9qa by stefano@mastodon.bsd.cafe
(DIR) More posts by stefano@mastodon.bsd.cafe
(DIR) Post #AdV6BJ4tQtroGjp6JM by stefano@mastodon.bsd.cafe
2024-01-04T11:20:01Z
0 likes, 1 repeats
This morning, an e-commerce site (built on Laravel and well-developed, hence quite efficient) started showing signs of slowing down. This had also happened a few weeks ago, and we partially managed the situation by increasing the VPS power and freeing up the physical machine from other loads. An analysis of the nginx log reveals that the server is being bombarded with requests from Bytedance. As often happens in these cases, I attempted to firewall the IPs associated with the bots. However, as soon as I block one IP, (or entire class) the crawling resumes (violently, almost like triggering a DoS attack) from another IP on another class. They don't respect the robots.txt file. The IPs they use online don't match the ones from which the requests originate; they probably constantly acquire and change IP blocks. It makes me wonder: if everyone online behaved like this, everything would collapse in a matter of minutes. #webdevelopment #serverissues #DDoS #Bytedance #firewall
(DIR) Post #AdV6KHGTXlUtFfWCYa by mms@emacs.ch
2024-01-04T11:21:38Z
0 likes, 1 repeats
@stefano I reject them in relayd based on user agenthttps://michal.sapka.me/bsd/blocking-bad-bots-openbsd/
(DIR) Post #AdV6QxkaUnZkkwn5yC by stefano@mastodon.bsd.cafe
2024-01-04T11:22:51Z
0 likes, 0 repeats
@mms That's the best way to deal with the situation.
(DIR) Post #AdV7BbTD9Q3HGWu3Oa by stefano@mastodon.bsd.cafe
2024-01-04T11:31:16Z
0 likes, 0 repeats
Added to nginx.conf:# case sensitive matching if ($http_user_agent ~ (Bytespider)) { return 403; } # case insensitive matching if ($http_user_agent ~* (bytespider)) { return 403; }Result: load is back to normal
(DIR) Post #AdVA1ZN4OrfdipwCVk by linus@telegrafverket.cc
2024-01-04T12:03:02Z
0 likes, 0 repeats
@stefano I guess we can count ourselves lucky that they at least don’t spoof their user agents. Considering how violently hostile it is to ignore robots.txt it almost feels like negligence rather than malice.
(DIR) Post #AdVA5W8KNcS8q3vO2y by stefano@mastodon.bsd.cafe
2024-01-04T12:03:48Z
0 likes, 0 repeats
@linus agree. The first thing I told the colleague is: I blocked the user agent, but I'm not sure they're not going to fake it.
(DIR) Post #AdVAr80bNUIdux0tqS by linus@telegrafverket.cc
2024-01-04T12:12:21Z
0 likes, 1 repeats
@stefano I guess another solution if there’s no user agent to go by is to use something like fail2ban to react on >N lines of a certain type in the httpd log triggering either a temporary firewall drop rule or a 403 landing page if possible. With a short block (<1m?) it might even be enough to make bots give up without inconveniencing real humans too much.
(DIR) Post #AdVG5zuKHPIsuGzITQ by itisiboller@infosec.exchange
2024-01-04T13:11:05Z
0 likes, 0 repeats
@stefano Thanks (stolen, although not as huge a problem as yours)
(DIR) Post #AdVOuwb19OBJ2RlCu8 by ClickyMcTicker@hachyderm.io
2024-01-04T14:17:00Z
0 likes, 0 repeats
@linus @stefano And if all else fails, block their entire ASN.
(DIR) Post #AdVOuxW5jBpptSSmrg by linus@telegrafverket.cc
2024-01-04T14:23:26Z
0 likes, 1 repeats
@ClickyMcTicker @stefano according to this blog post, that's not enough https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/
(DIR) Post #AdVdZkxEjllIADVZK4 by pee@mastodon.online
2024-01-04T17:34:12Z
0 likes, 0 repeats
@stefano have you tried Crowdsec?
(DIR) Post #AdVfCjMWgrZcTRU9qa by stefano@mastodon.bsd.cafe
2024-01-04T17:52:26Z
0 likes, 0 repeats
@pee I did, an interesting project. But unfortunately I don't think it would help here, and I can't install additional software on that server.