Post AOtuChmnREZe0gNumO by vib@fedi.absturztau.be
(DIR) More posts by vib@fedi.absturztau.be
(DIR) Post #AOtuCedT9SKYEszvKC by vib@fedi.absturztau.be
2022-10-24T18:16:30.458616Z
0 likes, 0 repeats
I don't know what in the world snug.moe is doing help
(DIR) Post #AOtuCfOcKA2ob73ZoW by vib@fedi.absturztau.be
2022-10-24T18:16:56.236465Z
0 likes, 0 repeats
Foundkey pumps up the CPU to the max and the deliver queue is very very very delayed
(DIR) Post #AOtuChmnREZe0gNumO by vib@fedi.absturztau.be
2022-10-24T18:21:54.641831Z
0 likes, 0 repeats
ok one more realistic theory: someone deleted their accounts and now foundkey struggles to send delete activities like EVERYWHERE
(DIR) Post #AOtuCiGDfq6rTw9Pdo by vib@fedi.absturztau.be
2022-10-24T18:22:28.998481Z
0 likes, 0 repeats
(DIR) Post #AOtuCiTgrksw9iSAdc by vib@fedi.absturztau.be
2022-10-24T18:18:04.500512Z
0 likes, 0 repeats
I am confused and also scared
(DIR) Post #AOtuCib8Q4q8Wnw7F2 by vib@fedi.absturztau.be
2022-10-24T18:23:44.258846Z
0 likes, 0 repeats
it is in fact delaying many many many delete activities to many instances I never heard of, that aren’t even fedi instances(anymore)
(DIR) Post #AOtuCjBIFdlOKwqzbM by vib@fedi.absturztau.be
2022-10-24T18:30:02.986881Z
0 likes, 0 repeats
yeah
(DIR) Post #AOtuCjZkmhKTYoIWjA by puniko@mk.absturztau.be
2022-10-24T18:31:08.466Z
0 likes, 0 repeats
@vib@fedi.absturztau.be wow, you beat me
(DIR) Post #AOtuPY5QoORvgzE2L2 by vib@fedi.absturztau.be
2022-10-24T18:32:23.173833Z
0 likes, 0 repeats
@puniko it’s normally not like this though…..
(DIR) Post #AOtuPYW1DXiV1RfGmO by puniko@mk.absturztau.be
2022-10-24T18:33:35.143Z
0 likes, 0 repeats
@vib@fedi.absturztau.be something stuck in queue?(also idk if foundkey has changed stuff in it)
(DIR) Post #AOtv0brNkGdu6sKGTw by vib@fedi.absturztau.be
2022-10-24T18:35:31.528395Z
1 likes, 0 repeats
@puniko well. I don’t know the details but now Foundkey tries to send delete activities in more places than it used to (to every instance it is aware of maybe?) and now someone seems to have deleted their account
(DIR) Post #AOtw5dvfZzUjvgINA8 by Jain@blob.cat
2022-10-24T18:53:13.462764Z
0 likes, 0 repeats
@vib @puniko why not let all events work through? does it affect your servers performance?
(DIR) Post #AOtwQBKV6gkfJKhg6S by vib@fedi.absturztau.be
2022-10-24T18:54:13.652981Z
0 likes, 0 repeats
@Jain @puniko I will probably just let it finish but yes all of my CPU cores are close to 100% and now new posts won’t federate until the queue gets emptier
(DIR) Post #AOtwQBtF1WXb34xQFk by Jain@blob.cat
2022-10-24T18:56:54.710298Z
0 likes, 0 repeats
@vib @puniko that reminds me of how long it takes the last time I restored the DB :blobcatlaugh:
(DIR) Post #AOtwrufsckFjlQvWaG by puniko@mk.absturztau.be
2022-10-24T19:00:22.242Z
0 likes, 0 repeats
@vib@fedi.absturztau.be @Jain@blob.cat tasks usually should hang around for 2 days or so, if they continue to fail
(DIR) Post #AOtwsCnzDLQrzmrFia by puniko@mk.absturztau.be
2022-10-24T19:00:52.252Z
0 likes, 0 repeats
@Jain@blob.cat @vib@fedi.absturztau.be it doesnt impact performance by much. some events might be delayed, but usually not by much
(DIR) Post #AOtwunXvaDFaWbaCXY by puniko@mk.absturztau.be
2022-10-24T19:01:21.579Z
0 likes, 0 repeats
@vib@fedi.absturztau.be @Jain@blob.cat you could also clear the queue as a last resort btw
(DIR) Post #AOtzFlUqbBIa9pGeaO by vib@fedi.absturztau.be
2022-10-24T19:03:02.067454Z
0 likes, 0 repeats
@puniko @Jain yeah I know but I don’t want to do that at least now. it seems to progress
(DIR) Post #AOtzFm0kgYorkmC8Jc by vib@fedi.absturztau.be
2022-10-24T19:26:37.597076Z
0 likes, 0 repeats
@puniko @Jain now at 400,000 delayeds lol
(DIR) Post #AOtzFmT6z7VLAjSmWG by Jain@blob.cat
2022-10-24T19:28:39.772877Z
0 likes, 0 repeats
@vib @puniko :blobcat3c:
(DIR) Post #AOve9n2UF0dCU0EQ9Q by tost@mk.toast.cafe
2022-10-25T14:39:16.301Z
1 likes, 0 repeats
@puniko@mk.absturztau.be @vib@fedi.absturztau.be @Jain@blob.cat we have a new mechanism that instantly drops tasks as they come up if they target a presumed-dead (automatically detected) instance, so while the queue is huge it should also resolve VERY fast
(DIR) Post #AOve9nYOKO9U4x9tse by Jain@blob.cat
2022-10-25T14:41:39.700442Z
0 likes, 0 repeats
@tost @vib @puniko custom patch? are instances which are temporarely down affected by this?
(DIR) Post #AOvehCTigBy0wBtucq by Jain@blob.cat
2022-10-25T14:47:38.351107Z
0 likes, 0 repeats
@tost @vib @puniko nicely done
(DIR) Post #AOveirTbjc4uNfdmFc by tost@mk.toast.cafe
2022-10-25T14:45:04.134Z
2 likes, 0 repeats
@Jain@blob.cat @vib@fedi.absturztau.be @puniko@mk.absturztau.benot custom, it's in mainthis is the primary commit: https://akkoma.dev/FoundKeyGang/FoundKey/commit/91a4f38871e4f39fc30e943fa913dd9ab43cc600 (minus cleanup)the tl;dr is that we already keep a statistic for "the last time we successfully communicated with an instance" (receiving an activity, sending an activity without an error, etc)if the last time that has happened is > a month, the instance is considered "dead for now" and we stop sending activities (exactly the same way as if it was suspended via the "stop sending activities" button)all that it takes to undo that is for us to receive an activity from themiirc a fetch (e.g via search or querying a user) should be sufficient, or to just have them send us somethingi.e truly automated "dead instance detection"
(DIR) Post #AOvxcE4XDzWhYtSjY0 by vib@fedi.absturztau.be
2022-10-25T18:16:05.477021Z
1 likes, 0 repeats
@Jain @puniko @tost nicey -w-