Post APLzgnuR38qZ7ndyHw by Twizzay@social.linux.pizza
(DIR) More posts by Twizzay@social.linux.pizza
(DIR) Post #APLrN7BcrtnhUGmjcO by selea@social.linux.pizza
2022-11-07T06:12:05Z
0 likes, 0 repeats
The queued jobs finally reached 0!Thank you for your patience!
(DIR) Post #APLru6q5BIGVxwRQ5w by selea@social.linux.pizza
2022-11-07T06:18:01Z
0 likes, 0 repeats
I think I have found out the root cause of the long queues, and I will be working on it during the day.Things should be improving
(DIR) Post #APLsWFUDTl13mzOqrA by kirenida@social.linux.pizza
2022-11-07T06:24:55Z
0 likes, 0 repeats
@selea was that the reason I couldn't upload photos to a post some 12 hours ago?
(DIR) Post #APLt9MByTsJhseW1b6 by stemid@mastodon.se
2022-11-07T06:32:00Z
0 likes, 0 repeats
@selea what was the root cause in your case?
(DIR) Post #APLtcWc2ohS6b4cNFI by selea@social.linux.pizza
2022-11-07T06:37:21Z
0 likes, 0 repeats
@stemid The root cause is increase in traffic :pBut the variable that made it worse is dead remote instances, which makes sidekiq timeout and retry
(DIR) Post #APLtwR7awTuXMGrzkG by JohanEmpa@mastodon.green
2022-11-07T06:40:22Z
0 likes, 0 repeats
@selea @stemid That's interesting
(DIR) Post #APLwi0HeEBGtvkNihU by stemid@mastodon.se
2022-11-07T07:11:48Z
0 likes, 0 repeats
@selea that was my suspicion too because sidekiq queues started growing as traffic started growing and I suspected a lot of instances were buckling, causing sidekiq jobs to timeout.but how did you fix that on your end? is it possible to decrease the timeout for dead instances or something?
(DIR) Post #APLxgUgoJBaFbizlYG by selea@social.linux.pizza
2022-11-07T07:22:53Z
0 likes, 0 repeats
@stemid The queue was at 0 this morning, for the first time in days. So I sent out a toot and checked the queue, saw it decrease and also saw the "retries" just exploding.I did notice that some instances, even big ones was throwing errors.
(DIR) Post #APLyKLGgkvuhyGmIOe by selea@social.linux.pizza
2022-11-07T07:30:04Z
0 likes, 0 repeats
@stemid Anyway, I'll gonna purge the dead instances (thoose I know for certain that they are perma-dead).No idea how do decrease that actually
(DIR) Post #APLzgnuR38qZ7ndyHw by Twizzay@social.linux.pizza
2022-11-07T07:45:13Z
0 likes, 0 repeats
@selea How are we doing on resources/donations?
(DIR) Post #APM0UyRy8JZuIPFfKi by selea@social.linux.pizza
2022-11-07T07:54:25Z
0 likes, 0 repeats
@Twizzay Resources looks fine for now, I expect things to be calmer now when people are at work.People have been generous the last days too
(DIR) Post #APMOLthGplC5qcc6rY by mooted@social.linux.pizza
2022-11-07T12:21:15Z
0 likes, 0 repeats
@selea thank you for running this instance!
(DIR) Post #APMvSrYkTaIpjVYa1o by gracion@social.linux.pizza
2022-11-07T18:32:26Z
0 likes, 0 repeats
@selea Great work thank you! I hope bumping up my tiny Liberapay helps defray a bit of new costs. I infer having the donation link on linux.pizza and not on social.linux.pizza is intentional.Thank you friendly admin!