Post 9xo7LvPV5x1y14311s by ferh@mastodon.host
(DIR) More posts by ferh@mastodon.host
(DIR) Post #9xnwwHq05Z6QVEZwmG by ferh@mastodon.host
2020-08-05T07:07:29Z
1 likes, 1 repeats
🛎️ Syncing of media files to object storage went too fast and rate limited the storage bucked during the night (CEST night), resulting in Mastodon's backlog processing/retry queue having a damn huge latency (4 hours right now).Stopped the sync (which was mostly complete) so rate limiting would stop and Mastodon could process more without retrying.Gonna add temporarily more workers for the queue so it processes the backlog more rapidly
(DIR) Post #9xnyApLcOcn2RMrhKa by trinsec@mastodon.host
2020-08-05T07:21:16Z
0 likes, 1 repeats
@ferh Thanks! I'm finally starting to get my first notification from out of this server (from 20 hours ago). ;)
(DIR) Post #9xnyVwfONbONDlBTO4 by ferh@mastodon.host
2020-08-05T07:25:08Z
1 likes, 1 repeats
interestingly even after stopping the sync we're still getting rate limited by object storage (while it shouldn't exceed 250 req/s, as there's just 50 workers)…
(DIR) Post #9xnzbhYZZT6839IqVE by ferh@mastodon.host
2020-08-05T07:36:38Z
0 likes, 1 repeats
@trinsec Weird (20 hours) because Sidekiq is saying (Latency: 4 hours). But cool. Not really solved tho, still getting a lot of rate limiting errors from S3. still on it..
(DIR) Post #9xo7LvPV5x1y14311s by ferh@mastodon.host
2020-08-05T08:46:23Z
1 likes, 1 repeats
New update: Scaleway Object Storage is just incredibly broken. Even without any requests it's still limiting or even returning error 400.
(DIR) Post #9xo7eS5iKKv2aQcwD2 by ferh@mastodon.host
2020-08-05T09:07:29Z
1 likes, 1 repeats
Amazing! Three hours after opening the ticket at Scaleway they responded. We never hit the rate limit, it was a technical issue on their side. Should be fixed now and the processing queue should process everything now
(DIR) Post #9xoELpWlfNXdRyLgI4 by trinsec@mastodon.host
2020-08-05T10:22:35Z
0 likes, 1 repeats
@ferh Cool! Looking forward to up-to-date feeds soon. :DAnd was that exclamation sarcastic? I can't quite figure out if 3 hours is a good or bad response time, heh.
(DIR) Post #9xoH7t0iDgdg3pOiPo by ferh@mastodon.host
2020-08-05T10:53:25Z
0 likes, 1 repeats
@trinsec yes quite sarcastic. I wish they were monitoring their services a bit more.... always annoying to stumble upon an issue on their side, while they don't notice it, and act after some hours...
(DIR) Post #9xoZs3koPEjxsR7wsC by trinsec@mastodon.host
2020-08-05T14:21:48Z
0 likes, 1 repeats
@ferh Getting stuff from 7 hours ago appearing now. I daresay it's an improvement over 20 hours. ;)
(DIR) Post #9xopPGIIImuhdc4uzw by ferh@mastodon.host
2020-08-05T17:16:23Z
0 likes, 1 repeats
@trinsec yeah :( the queue is going down very slowly. For some reason I can get Sidekiq to use all his threads nor all the DB connections.
(DIR) Post #9xozBqyU7wg4t73e3k by ferh@mastodon.host
2020-08-05T19:06:33Z
0 likes, 1 repeats
another update: the queue is still processing :(
(DIR) Post #9xp5CKoNnEeD9PwfZ2 by eqe@aleph.land
2020-08-05T16:36:19Z
0 likes, 0 repeats
@ferh @trinsec well their status page says it's operational so everything must be fine 🙄
(DIR) Post #9xp5CL3csYqBuh4qK8 by ferh@mastodon.host
2020-08-05T20:14:37Z
0 likes, 1 repeats
@eqe haha :-) It's not ODS I'm using, but "Object Storage". Either way -- ODS had a shit ton of issues since sept, and yeah, you can see it right ? :))) @trinsec
(DIR) Post #9xq0t22JAbj8l8xW52 by trinsec@mastodon.host
2020-08-06T07:00:46Z
0 likes, 1 repeats
@ferh My feeds seem to be up-to-date again mostly. Is stuff back to regular scheduling now?
(DIR) Post #9xq3QKvSA0J0oBlFx2 by ferh@mastodon.host
2020-08-06T07:29:45Z
0 likes, 1 repeats
@trinsec yes! \o/
(DIR) Post #9xq3gt9mepOrbkMBjU by trinsec@mastodon.host
2020-08-06T07:32:47Z
0 likes, 1 repeats
@ferh 🎉 (We so need Telegram stickers here!)