Post AlIm3prAzxDutaSPNA by Tij@eientei.org
(DIR) More posts by Tij@eientei.org
(DIR) Post #AlIm3prAzxDutaSPNA by Tij@eientei.org
2024-08-24T19:34:49.306484Z
2 likes, 2 repeats
@nukie you son of a fucking bitch why did you not leave me a spot
(DIR) Post #AlImD2PxfLjSSX7EjA by Tij@eientei.org
2024-08-24T19:36:29.572781Z
2 likes, 2 repeats
@nukie literally where the fuck did this dude go, he switched his xmpp pfp to hello kitty or some shit Im scared that he actually went trans for real this time
(DIR) Post #AlImQl0BDLl6V0bABs by mint@ryona.agency
2024-08-24T19:38:59.202554Z
2 likes, 1 repeats
@Tij @nukie Seems to still have his old Asuka avatar.
(DIR) Post #AlImiAb1JDLmnj8Juq by mint@ryona.agency
2024-08-24T19:42:07.071755Z
2 likes, 1 repeats
@Tij @nukie Anyway, at the very least his instance was useful as a guinea pig for debugging frozen federation queue problem.
(DIR) Post #AlImiodYRD92zAcuQ4 by Tij@eientei.org
2024-08-24T19:42:14.872598Z
2 likes, 2 repeats
@mint @nukie he has multiple accounts
(DIR) Post #AlImoKrY3DY6NbnuiG by Tij@eientei.org
2024-08-24T19:43:15.377050Z
1 likes, 1 repeats
@mint @nukie true, its broken beyond repair now doe
(DIR) Post #AlImsTV5sTICKOH6tU by mint@ryona.agency
2024-08-24T19:43:59.768308Z
2 likes, 1 repeats
@Tij @nukie Won't say it's beyond repair, but I don't want to kick it back until the new Oban release drops and gets pulled as dependency in Pleroma.
(DIR) Post #AlImwUs4iucvjlwTmi by Tij@eientei.org
2024-08-24T19:44:44.054111Z
1 likes, 1 repeats
@mint @nukie oh
(DIR) Post #AlIn4g1r7ghWevPwv2 by 0@pl.absolutelyproprietary.org
2024-08-24T19:46:12.615205Z
3 likes, 1 repeats
@Tij @nukie how fedi.
(DIR) Post #AlKdhHYLRnqYogIsvA by mint@ryona.agency
2024-08-25T17:10:32.539705Z
1 likes, 1 repeats
@Tij @nukie Actually I'm now tired of waiting plus I want to see how it performs now, so I merged the develop branch and changed oban dependency to be pulled from git, will update the instances later.Also @feld, apparently adding Lazarus plugin broke accessing DB config. adminfe would just show the empty settings dropdown while nu-PleromaFE throws TypeError: n.tuple is undefined. This doesn't happen after switching to develop branch, and I don't see anything out of place in /api/pleroma/admin/config aside from that. Nothing too critical, just had to change config.exs instead to reduce liability of a hostile actor on one of the instances.Screenshot_20240825_195527.png
(DIR) Post #AlKfcNpgs7AIklD076 by phnt@fluffytail.org
2024-08-25T17:32:04.907643Z
1 likes, 1 repeats
@mint @feld I updated to the latest Oban git yesterday and it survived a DB repack which is an improvement I guess. Other actions that would crash Pleroma (related to the 502 gateway issue; that issue is a special case of this) no longer seem to do so. Today Husky crapped out on me probably because it couldn't auth over the dropped API requests coming through db_connection. Increasing queue_target and queue_interval did help with that, but it might have other side effects. It's too early to tell if the newer Oban helps with the stalling federation. At least the performance isn't worse. @nukie @Tij
(DIR) Post #AlKg59FYAaHHvI1KSm by mint@ryona.agency
2024-08-25T17:37:15.998657Z
2 likes, 1 repeats
@phnt @feld @nukie @Tij >It's too early to tell if the newer Oban helps with the stalling federationDescription of the commit is a fairly close match to the symptoms we observed (Oban unaliving itself after receiving too many DB timeouts), so we'll see.
(DIR) Post #AlKinULRIfTHzw2uA4 by mint@ryona.agency
2024-08-25T18:07:41.699505Z
0 likes, 1 repeats
@phnt @Tij @feld @nukie Oh, it crashed in entirety, but got restarted by healthcheck script. Nothing in logs except hundreds of DBConnection.ConnectionErrors, but there's also a bunch of "ERROR 57014 (query_canceled)" and "unknown registry: Pleroma.Web.StreamerRegistry".
(DIR) Post #AlKjrJoGVkO8R7v3B2 by phnt@fluffytail.org
2024-08-25T18:19:35.702447Z
1 likes, 1 repeats
@mint @feld @nukie @Tij Yeah that's the issue I usually run into. I have no idea how the supervisor tree in Pleroma looks like, but my theory is that after enough db_connection errors, the error slowly goes up and eventually reaches Pleroma's own supervisor. The maximum number of restarts is set to 3 in the default config and after that is exceeded, it exits and init restarts it. There's a somewhat rare case when the Pleroma application completely shuts down, but the system process itself still exists and therefore it doesn't get restarted by init. That's the issue I talked about.
(DIR) Post #AlKlDu38YKB39pCyPI by mint@ryona.agency
2024-08-25T18:34:52.805143Z
1 likes, 1 repeats
@phnt @feld @nukie @Tij Huh, never noticed that option.https://git.pleroma.social/pleroma/pleroma/-/blob/develop/config/config.exs?ref_type=heads#L925Might as well set it to 99 or something.
(DIR) Post #AlKyBw2NURZG9CFG8e by phnt@fluffytail.org
2024-08-25T21:00:11.858864Z
1 likes, 1 repeats
@feld @nukie @Tij @mintIt's very hard to tell because even loading FE sometimes floods logs with DBConnection errors. Currently I have no way of at least somewhat reliably causing the crash.There's one log in one of the other threads that at least partially crippled Pleroma into not listening on any ports and didn't cause a restart.https://fluffytail.org/notice/Al1dQDXk8Erhmg31sWI'll look through my logs tomorrow for a proper log where Pleroma exited completely.
(DIR) Post #AlXQM3WVdoCKsFTLMG by phnt@fluffytail.org
2024-08-31T21:12:18.196992Z
1 likes, 1 repeats
@feld Sorry for the delay. I've looked through my logs and the last time Pleroma shut down and restarted was 11 days (no crash or stalled federation since then) when I was still running Oban 2.13.6.The logs are mostly the same as the ones in the other thread linked above. A lot of "connection not available and request was dropped from queue after X ms" or an occasional "connection closed by the pool, possibly due to a timeout..." messages coming from db_connection with even rarer "cancelling statement due to user request" messages coming from postgrex.The db_connection errors always come in big batches usually when disk iowait increases which isn't under my control. It's not caused by Postgres autovacuum as that runs much more frequently.@feld @Tij @mint @nukie