Post B9KirOtFpHWxs8bCxk by badrihippo@fosstodon.org
(DIR) More posts by badrihippo@fosstodon.org
(DIR) Post #B9KcjaW07b7Xsuc8R6 by mntmn@mastodon.social
0 likes, 0 repeats
hmmm mnt gitlab is down-ish because it's pummeled by crawlers because iptables ipset doesn't seem to work anymore? i get a lot of "ipset v7.19: Element cannot be added to the set: it's already added" from my realtime blocking script, which shouldn't be possible as these IPs shouldn't be able to pass the firewall in the first place
(DIR) Post #B9KcuIxxi01YdunqCm by mntmn@mastodon.social
0 likes, 0 repeats
i thought the iptables rules might have been flushed, and readded them, but now i just have them twice, and the pkts counter is going up... hmm. any idea what's happening? (stopped the gitlab for now while thinking about this)
(DIR) Post #B9KdZpniXCqiLooWEi by mntmn@mastodon.social
0 likes, 0 repeats
i wonder if the tcp src is altered for iptables somehow, but not for nginx...
(DIR) Post #B9KdbRGAkTR0ojt8pk by avian@toot.si
0 likes, 0 repeats
@mntmn Do you have a firewall rule with accept all with state established or something similar? That's a thing that has got me before when I was wondering why blocked IPs are trying to get blocked the second time.
(DIR) Post #B9Ke1pW1BoBzZdstfM by theraspb@aus.social
0 likes, 0 repeats
@mntmn you're not accepting x-forward-for or x-real-ip from the internet are ya? that could show addresses in the logs that the http client has set in the header, which isnt the actual layer 3 ip address
(DIR) Post #B9Ke6leQz4kD2MAwGe by mntmn@mastodon.social
0 likes, 0 repeats
sidenote, my ipset blocklist has 3134769 entries
(DIR) Post #B9KeLMccKC1fknHla4 by froztbyte@mastodon.social
0 likes, 0 repeats
@mntmn getting counters on both those input rules is a bit odd, first match should terminate processing (unless there’s bits of the rule not visible in screenshot)the error message you quite comes from the ipset binary tho, use `ipset list` etc to check there also, you may like `-L -nvx` for a general display (it’s my preferred when dealing with iptables, tho I’m trying to stick to nftables on new stuff)
(DIR) Post #B9Kema7SFY4xQxgiMS by mntmn@mastodon.social
1 likes, 0 repeats
ah, it's just that the attack is now much faster/bigger scale than before. i'm blocking almost 20 IPs per second, and some IP request multiple things in parallel so the script just tries to block them multiple times in a row
(DIR) Post #B9Kf1RZIKAafG0nS7c by mntmn@mastodon.social
0 likes, 0 repeats
@froztbyte yeah it's just because i added the rule twice (second time by accident today, first time i guess 2 weeks ago), so the top counter is the new one and just counted for a little while, while the second one has "aged" and accumulated a lot of packets before
(DIR) Post #B9Kf6wPV58N8MNpwhM by mntmn@mastodon.social
0 likes, 0 repeats
someone is really pumping the gas pedal today on crawler attacks
(DIR) Post #B9KfOoFu8j3CHTrvZQ by lanodan@queer.hacktivis.me
0 likes, 0 repeats
@mntmn Yeah kind of thing which made me wonder about disabling http2 on some vhosts to somewhat lessen their ability to make parallel requests.Although thankfully here crawlers don't cause much load issues as 99% of my stuff is static.
(DIR) Post #B9KfT2TnCHprYQNiam by Cenbe@techhub.social
0 likes, 0 repeats
@mntmn This is exactly why I shut down my web site. The web has become useless... and I can't even log in to sites without my phone in my hand.
(DIR) Post #B9KfXbQNDOFp5lxU3M by froztbyte@mastodon.social
0 likes, 0 repeats
@mntmn ah, right :) didn’t think of that because I’m used to running my iptables with ferm (which would clobber counters on ruleset reload)
(DIR) Post #B9KfoLU9jG2ca5iA3k by katco@mstdn.social
0 likes, 0 repeats
@mntmn no clue if this is related, but Guix saw an uptick and it turned out to be some kind of new browser: https://toot.aquilenet.fr/@civodul/117084682789609110RT: https://toot.aquilenet.fr/users/civodul/statuses/117084682789609110
(DIR) Post #B9KghPLL5TWmTNqKIa by mntmn@mastodon.social
0 likes, 0 repeats
@lanodan hmmmm @ http2!
(DIR) Post #B9KirOtFpHWxs8bCxk by badrihippo@fosstodon.org
0 likes, 0 repeats
@mntmn incidentally, I just noticed that @disroot is facing a DDoS attack too. Coincidence?https://status.disroot.org/
(DIR) Post #B9KkhxwMr5V1ahgUSW by mntmn@mastodon.social
1 likes, 0 repeats
i blocked out large swathes of whole /8 ranges to collect some stats in the DROP list
(DIR) Post #B9Km8whZfUlrWSC4qu by mntmn@mastodon.social
0 likes, 0 repeats
top 10 crawler traffic to my server comes from 103., 102., 45., 105., 186., 41., 37., 190., 181., 157.
(DIR) Post #B9KuzIzUqI7jLI3RtQ by mntmn@mastodon.social
0 likes, 0 repeats
strange, now it's suddenly quiet(er)
(DIR) Post #B9KvUVJD5v2eWU9V7w by mntmn@mastodon.social
0 likes, 0 repeats
the packet counters in all the DROP rules are suddenly not incrementing anymore, only here and there a rare trickle. so it is _one_ coordinated entity behind this who decided to press the stop button?! eerie
(DIR) Post #B9Kw7tZGpksl3SRoC8 by decibyte@social.data.coop
0 likes, 0 repeats
@mntmn I had a similar experience earlier today: https://social.data.coop/@decibyte/117087895845840901Ended up spending the day on future mitigation instead of writing code for a project.
(DIR) Post #B9KwT5CKmxI7Cpp2sC by gsuberland@chaos.social
0 likes, 0 repeats
@mntmn related to this?https://pony.social/@cadey/117089128496608868
(DIR) Post #B9Kwzk24VDIl7PawW8 by mntmn@mastodon.social
0 likes, 0 repeats
@gsuberland interesting, i think semi-unrelated but i also just noticed lightpanda in the logs, appeared first time on august 9. gonna block that, too
(DIR) Post #B9LdUituxImQf3Ejbs by mntmn@mastodon.social
0 likes, 0 repeats
@decibyte ah yes, that sounds very related