Post Apw9BxfaCpuzFSVHAe by saagar@federated.saagarjha.com
(DIR) More posts by saagar@federated.saagarjha.com
(DIR) Post #An07sJppGc2LN5ndLc by olivia@pl.voltrina.net
2024-10-14T15:23:57.867005Z
0 likes, 0 repeats
>pleroma database has grown by 7GB in a span of 4 dayswhat
(DIR) Post #An08KhBc2ZdBu2WN16 by i@declin.eu
2024-10-14T15:28:52.183072Z
1 likes, 0 repeats
@olivia which tables? https://pl.voltrina.net/phoenix/live_dashboard/ecto_statssome people keep seeing massive instance killing database growth but never share the details
(DIR) Post #An08TWwJVV0OjHZtFg by verita84_automata@poster.place
2024-10-14T15:31:19.469653Z
1 likes, 0 repeats
@olivia you gotta exclude t he mastodon instances via mrf that block you anyway
(DIR) Post #An0BWB4XZ791dlNbt2 by olivia@pl.voltrina.net
2024-10-14T16:04:45.950947Z
0 likes, 0 repeats
oban_jobs seems to be the culprit
(DIR) Post #An0Bz3slNOu9jsdzgO by i@declin.eu
2024-10-14T16:09:45.892212Z
1 likes, 0 repeats
@olivia how do you have so many stuck jobs lol, normal oban should hover at a couple of k jobs, if you don't mind losing the unfederated backlog, you can just delete from the table freelymind showing what pleroma=> select state, queue, attempt, count(*) from oban_jobs group by state, queue, worker, attempt order by count(*) desc; prints?
(DIR) Post #An0Caumtddl4XKwwsq by mint@ryona.agency
2024-10-14T16:16:50.928617Z
2 likes, 1 repeats
@i @olivia Bet a good half of that would happen to be a bunch of those mystery misskey.io posts.
(DIR) Post #An0D0h1LRiorMu03GK by olivia@pl.voltrina.net
2024-10-14T16:21:29.032690Z
0 likes, 0 repeats
there you go state | queue | attempt | count -----------+----------------------+---------+--------- available | background | 0 | 3773277 available | background | 2 | 1863333 available | background | 1 | 1022895 available | background | 3 | 494206 available | background | 4 | 266730 available | background | 5 | 130975 available | background | 6 | 67975 available | background | 7 | 34023 available | background | 0 | 28042 available | background | 8 | 16993 available | background | 9 | 8558 available | notifications | 0 | 6916 available | background | 10 | 4201 available | background | 11 | 2264 available | background | 12 | 988 available | background | 13 | 480 available | background | 14 | 205 completed | federator_outgoing | 1 | 150 completed | federator_incoming | 1 | 130 completed | search_indexing | 1 | 127 available | background | 15 | 117 cancelled | federator_incoming | 1 | 85 scheduled | background | 0 | 77 retryable | background | 10 | 67 retryable | background | 2 | 56 available | background | 0 | 55 retryable | background | 3 | 34 retryable | slow | 19 | 28 cancelled | slow | 1 | 25 executing | federator_incoming | 1 | 23 available | check_domain_resolve | 0 | 20 available | mailer | 0 | 20 retryable | slow | 18 | 17 retryable | background | 13 | 16 discarded | federator_incoming | 5 | 16 executing | background | 3 | 12 executing | background | 2 | 12 executing | federator_incoming | 4 | 11 retryable | background | 4 | 11 executing | federator_incoming | 2 | 10 retryable | background | 6 | 9 retryable | federator_outgoing | 4 | 9 executing | federator_incoming | 3 | 9 retryable | background | 19 | 8 available | background | 19 | 8 retryable | slow | 17 | 8 retryable | background | 12 | 6 retryable | background | 9 | 6 executing | federator_outgoing | 1 | 6 retryable | background | 5 | 6 available | background | 0 | 5 executing | federator_incoming | 5 | 5 available | background | 0 | 5 discarded | slow | 5 | 5 retryable | background | 8 | 5 executing | background | 4 | 4 retryable | background | 11 | 4 retryable | background | 7 | 4 available | background | 18 | 3 retryable | slow | 15 | 3 completed | federator_incoming | 2 | 2 executing | federator_outgoing | 5 | 1 retryable | background | 14 | 1 retryable | background | 15 | 1 retryable | slow | 11 | 1 retryable | slow | 16 | 1 executing | background | 1 | 1 completed | slow | 1 | 1 completed | web_push | 1 | 1 available | background | 17 | 1 available | background | 0 | 1(71 rows)
(DIR) Post #An0DLVhoi5HwMBnBYm by i@declin.eu
2024-10-14T16:25:05.030578Z
1 likes, 0 repeats
@mint @olivia i wonder if @feld would ever consider a dedup MRF into pleroma via simhash_ex or ex_lsh, since it would also require a cachex table, and those have to be defined ahead of time, unlike whenever we eventually switch to nebulex
(DIR) Post #An0Db4AWrL2zaOdj9M by i@declin.eu
2024-10-14T16:27:55.752954Z
2 likes, 0 repeats
@olivia forgot feld made most things a background when removing worker from select but not group, probably as mint says, remote fetch workers stuck with 20! retries, working through spamrun watch 'sudo -Hu postgres psql pleroma -c "delete from oban_jobs where attempt > 3;"' for some hours and it should clear up
(DIR) Post #An0DyAyLjJlhLaxjcG by mint@ryona.agency
2024-10-14T16:32:14.975017Z
2 likes, 1 repeats
@i @olivia Here's another report of the same shit happening: https://git.pleroma.social/pleroma/pleroma/-/issues/333520 retries might indeed be too much (I have around 1500 jobs right now, most of which are completed, a good half of those that aren't are retries for DRC since verita's :cloudflare: rules keep blocking me or something), but I'm more interested in how the fuck do they manage to pile up so hard. Forcefetched the mentioned post without any issues.
(DIR) Post #An0EKkQObESpfhqO3s by mint@ryona.agency
2024-10-14T16:36:19.573599Z
2 likes, 1 repeats
@i @olivia Maybe it's related to pinned posts; I have patched my pleromer to not fetch them when seeing a new actor.
(DIR) Post #An0F44DnqG2m6NZ0Yi by feld@friedcheese.us
2024-10-14T16:42:18.105003Z
0 likes, 0 repeats
@i @olivia @mint dedup for what exactly?
(DIR) Post #An0F45CQCsX78NvQ2q by i@declin.eu
2024-10-14T16:44:21.829391Z
0 likes, 0 repeats
@feld @olivia @mint the PASTA WITH EXTRA SPAM, like almost all the previous nuisances would have been discarded if 99% matches of the exact same text posts were ignored
(DIR) Post #An0Feo9IYvzKy4yZns by feld@friedcheese.us
2024-10-14T16:46:15.109049Z
0 likes, 0 repeats
@i @olivia @mint was there a spam campaign of mostly same text, but no links and the existing MRFs don't detect it?
(DIR) Post #An0Fepcl4t9DXjlcMS by mint@ryona.agency
2024-10-14T16:51:08.737559Z
1 likes, 1 repeats
@feld @i @olivia There was, I wasn't affected, some used AntiMentionSpam, keyword or reject policies. That said, it isn't related to current issue with RemoteFetcherWorkers piling up into millions (which I believe are only spawned by pinned post fetching pipeline in vanilla pleromer).
(DIR) Post #An0Fnu4rtDYzxfF0CW by i@declin.eu
2024-10-14T16:52:38.604455Z
0 likes, 0 repeats
@feld @olivia @mint the recent misskey one had unique text to keywordpolicy, but with https://256.lt/mrf/similar_policy.ex i didn't even have to, since it was made to combat the goldberg spam, and happily worked through the ctkpaar mess too
(DIR) Post #An0G7GGjbg4gHy1QIa by mint@ryona.agency
2024-10-14T16:56:18.044268Z
1 likes, 1 repeats
@feld @i @olivia Indeed, the three posts mentioned in the issue are the same three posts that's pinned on affected actor's profile. Don't notice anything out of ordinary in his collection aside from said posts having shitton of emojis.
(DIR) Post #An0GPoCECjmOj0l3w0 by feld@friedcheese.us
2024-10-14T16:58:06.041374Z
0 likes, 0 repeats
@mint @i @olivia weird, why would it keep fetching them? can you confirm for me the profile so I can take a closer look?also the dupes shouldn't happen with latest develop branch, at least if it tried it would cancel inserting the job every time because a duplicate one existed (until pruner kicks in and clears up old Oban jobs)
(DIR) Post #An0GPpDKQ8FnsiHSHw by mint@ryona.agency
2024-10-14T16:59:38.916312Z
0 likes, 1 repeats
@feld @i @olivia Profile is https://misskey.io/users/9mhsmldaly3m08ft, the issue mentions some transmogrifier error but I haven't gotten it when forcefetching all three posts.
(DIR) Post #An0HIQucTXBsRDZKDo by mint@ryona.agency
2024-10-14T17:09:31.325981Z
1 likes, 2 repeats
@feld @i @olivia The exact error might be irrelevant since they might also have some geoblocks or other :cloudflare: shenanigans going. What's more concerning is pileup happening in a first place; now that I'm thinking about it might be recursion.1. pleromer receives an activity referencing that guy's profile/post 2. it fetches them 3. fetch pipeline kicks in 4. pinned posts fetching happens as a part of pipeline5. pleromer inserts RemoteFetcherWorker jobs for those posts 6. said jobs try to fetch pinned posts againIf that's the case (too lazy to confirm, sorry) and fetcher jobs start erroring out, the queue raises exponentially. Hopefully not?
(DIR) Post #An0HLZuLUDpxCpwMMq by feld@friedcheese.us
2024-10-14T17:04:32.662164Z
0 likes, 0 repeats
@mint @i @olivia that redirects me to an account named beer_bastard. Same?
(DIR) Post #An0HLae4kCPtUfKse8 by mint@ryona.agency
2024-10-14T17:10:05.899973Z
0 likes, 1 repeats
@feld @i @olivia Yeah, misskey uses flakes in their AP actor IDs.
(DIR) Post #An0HY15KGhjcyA0W1Y by feld@friedcheese.us
2024-10-14T17:11:34.901614Z
1 likes, 0 repeats
@mint @i @olivia > 5. pleromer inserts RemoteFetcherWorker jobs for those posts these inserts were not set to be unique, but they are now
(DIR) Post #An0HegKVpkHQcrHyXA by i@declin.eu
2024-10-14T17:13:22.039115Z
0 likes, 0 repeats
@feld @olivia @mint would be nice to get a new release out then, people who don't run develop complained about the follow bug not being fixed yet too
(DIR) Post #An0Hlp71ZxiJtJNxVw by mint@ryona.agency
2024-10-14T17:14:49.997919Z
0 likes, 1 repeats
@feld @i @olivia Indeed, but that's more of last frontier measure. There would still be some friction left pertaining to checking whether such job exists and raising an exception if it does.
(DIR) Post #An0HnysoDrxmm1lUxc by olivia@pl.voltrina.net
2024-10-14T17:15:10.821404Z
0 likes, 0 repeats
is it safe to delete the stuck oban jobs while pleroma is running? i'd prefer to keep my instance up in case someone DM's me lol
(DIR) Post #An0HzjSnNVJWQSCW7k by i@declin.eu
2024-10-14T17:17:12.997518Z
1 likes, 0 repeats
@olivia yes it is and i've done it too many times without breaking anything, add a and queue = 'background' in there not to impact other things if you want to and let err rip
(DIR) Post #An0IDU5DMIHymKjEIK by feld@friedcheese.us
2024-10-14T17:17:21.352738Z
1 likes, 0 repeats
@mint @i @olivia you don't want to raise an exception on a duplicate job in Oban; that would break a lot of stuff needlessly. It just drops the job silently. It's not an error scenario that needs to raise / cause the process to abort.
(DIR) Post #Apw9BxfaCpuzFSVHAe by saagar@federated.saagarjha.com
2025-01-10T09:45:14.092235Z
2 likes, 0 repeats
@Vaghrad @i @feld @olivia @mint Also wanted to add in my appreciation; my server was also dying for a few days and this seems to have helped for now. Thanks!