Post ATfSgjhECduixJQlLk by cracky@rot.gives
(DIR) More posts by cracky@rot.gives
(DIR) Post #ATfSUBHSKl1ak5OCAK by errante@rot.gives
2023-03-16T08:35:59.456872Z
1 likes, 0 repeats
@pdo you know where pleroma stores notification feeds? trying to debug this issue which occasionally happens to my instance. im still able to post, local timeline functions, known network functions, but notifications and home timeline stop working. after i reciev about 20 or so notifications (enough to form a new 'page'), notifications start working again, but i cant scroll down below those 20 notifications, or past that point in my home timeline. the systemd pleroma logs dont show anything concerning
(DIR) Post #ATfSfVsy4699piL464 by errante@rot.gives
2023-03-16T08:38:03.033404Z
2 likes, 0 repeats
@p my theory is that its an issue where pleroma refuses to load any feed with a particular user in it. thats why local TL works, and why home TL and notifs dont work, and why known network starts working faster. of course, i cant tell who this user is...bcos, yknow. nothing loads
(DIR) Post #ATfSgjhECduixJQlLk by cracky@rot.gives
2023-03-16T08:38:18.779270Z
1 likes, 0 repeats
@errante @p did you mute something that youd otherwise be getting tons of notifs from
(DIR) Post #ATfSznvGiHT7Z95fKy by errante@rot.gives
2023-03-16T08:41:42.549668Z
1 likes, 0 repeats
@cracky @p uhhh.. i think im in those periodic fse hellthreads?
(DIR) Post #ATfT13m5GIT51FOQqm by errante@rot.gives
2023-03-16T08:41:59.041365Z
1 likes, 0 repeats
@cracky @p like the ones where the rum guy tags 30 people
(DIR) Post #ATfTbWgbFMWn6zDThw by iamtakingiteasy@eientei.org
2023-03-16T08:48:29.858831Z
1 likes, 0 repeats
@errante @p Notifications are just the filtered activities from activity table, you may need to enable debug logs to actually get any meaningful BE errors. Exact filter is constructed lib/pleroma/notification.ex and debug loglevel should also log the resulting query. Most likely one of notifications contains something whacky that cannot be rendered without error.
(DIR) Post #ATfTjt61oIeevr66G8 by errante@rot.gives
2023-03-16T08:50:02.227418Z
1 likes, 0 repeats
@iamtakingiteasy @p yea, thats my thought too. where do i enable BE error logging?
(DIR) Post #ATfTyu46URIcHUMD0y by iamtakingiteasy@eientei.org
2023-03-16T08:52:42.004156Z
2 likes, 0 repeats
@errante @p You can set console logger to debug in runtime, if you have admin FE / database configuration enabled, without any restarts. If not, you have to edit your prod.secret.exs to `config :logger, :console, level: :debug` and restart, be prepared to walls of text in either way.
(DIR) Post #ATgEebLP51pGGzGogy by p@freespeechextremist.com
2023-03-16T17:35:47.602184Z
2 likes, 0 repeats
@errante > do you know where pleroma stores notification feeds?If you mean the database, it's in the notifications table; the web endpoint, is /api/v1/notifications.> after i reciev about 20 or so notifications (enough to form a new 'page'), notifications start working againYou'll wanna look at that in the browser console. If it couldn't parse the JSON, usually it's because there was an error page and sometimes this is because your server is shitting itself. Sometimes it's a timeout, sometimes something has caused some weird error trying to render the notification (I think this used to happen with users that were deactivated), sometimes you've muted a giant thread that has several pages of your notifications in it.One thing you can do to avoid the notification endpoint timing out is to do a full vacuum. If you combine that with clearing out old ones (say likes/reposts/reactions where seen = 't' and (CURRENT_TIMESTAMP-inserted_at)>'1 month'::interval), it has a noticeable impact on the table. Could also be a timeout due to DB congestion.
(DIR) Post #ATgEhgUCQ3Ln5Zr2Q4 by ins0mniak@freespeechextremist.com
2023-03-16T17:36:21.030108Z
1 likes, 0 repeats
@p @errante Whoa man, you're a pirate now.
(DIR) Post #ATgElMdIAw66fnrWFM by p@freespeechextremist.com
2023-03-16T17:37:00.936165Z
3 likes, 0 repeats
@iamtakingiteasy @errante > Notifications are just the filtered activities from activity tableThere's a notifications table.
(DIR) Post #ATgEo4waOlLygIBIe0 by errante@rot.gives
2023-03-16T17:37:27.833185Z
1 likes, 0 repeats
@p does a vacuum fix DB congestion?its not only notifications that fail, but also the home timeline
(DIR) Post #ATgH0gsBPAYbhPGNiC by iamtakingiteasy@eientei.org
2023-03-16T18:02:06.417023Z
2 likes, 0 repeats
@p @errante Yes, but it is just a many-to-many relation between users and activities with classification for filtering. Considering that activities are inner-joined and all the symptoms described so far (some point at which both notifications and timeline endpoints are not returning valid json, not even the empty list), issue is most likely with some specific activity/related object contents. Debug logs should indicate error origin.
(DIR) Post #ATgLhxQy6g87KaQ90i by p@freespeechextremist.com
2023-03-16T18:54:50.107524Z
1 likes, 0 repeats
@ins0mniak @errante :venomsnake:
(DIR) Post #ATgMQw2G9YXY1bJICu by p@freespeechextremist.com
2023-03-16T19:02:57.823542Z
1 likes, 0 repeats
@errante Doing a vacuum doesn't fix the congestion, just alleviates it. (Updates and deletions create gaps in the data and the index. Sometimes this space gets recycled, sometimes not. Full vacuum eliminates the gaps and rebuilds the indexes, so if you have a lot of shaky data, it requires less disk I/O for any given query. Careful with it, though, as running it against the entire DB can require as much disk as the table you run it against; in Pleroma's case, doing this with activities or objects can take up a lot of space and Postgres's failure mode in that case is unhelpful: it doesn't remove the temporary files.)> its not only notifications that fail, but also the home timeline DB appropriately tuned?
(DIR) Post #ATgMSD8i2vdjMTw3jE by xue@collapsitarian.io
2023-03-16T19:03:10.868818Z
2 likes, 0 repeats
@ins0mniak @p @errante shard in head grows bigger every time P does something evil, for example doesnt leave tip or goes through crosswalk during red light
(DIR) Post #ATgMY8CdJVqLmvasIS by errante@rot.gives
2023-03-16T19:04:13.546368Z
1 likes, 0 repeats
@p done no tuning
(DIR) Post #ATgNBtSpX0SXez87Gq by p@freespeechextremist.com
2023-03-16T19:11:27.021824Z
1 likes, 0 repeats
@iamtakingiteasy @errante Yeah, notifications are that table; the contents in the bar on the side are a join of that with activities.> some point at which both notifications and timeline endpoints are not returning valid json, not even the empty listI'm willing to bet money that the invalid JSON is a 504 error page if it's affecting notifications *and* the public timeline. Malicious or glitchy data is unlikely, most of that's been hammered out of Pleroma. I listed a handful of causes elsewhere in the thread.
(DIR) Post #ATgNGjDH4UGrJc9vfs by p@freespeechextremist.com
2023-03-16T19:12:19.442376Z
4 likes, 0 repeats
@xue @ins0mniak @errante :venomsnakedemon2:
(DIR) Post #ATgNbVjkN3YLbJG36m by p@freespeechextremist.com
2023-03-16T19:16:04.805414Z
2 likes, 0 repeats
@errante Pleroma puts Postgres through its paces. You'll wanna do a little of that. This is a really good *starting* point, and does the job for most people: https://pgtune.leopard.in.ua/ .
(DIR) Post #ATgNqxCCYrwl6gvmpE by dcc@annihilation.social
2023-03-16T19:18:51.382023Z
2 likes, 0 repeats
@p @errante crazy to me some people have not tuned postgres
(DIR) Post #ATgO3A2ocHEcQnMTnE by p@freespeechextremist.com
2023-03-16T19:21:04.631401Z
4 likes, 1 repeats
@dcc @errante Postgres's defaults keep DBAs in business.