Post 790857 by roguelazer@mastodon.technology
 (DIR) More posts by roguelazer@mastodon.technology
 (DIR) Post #790658 by Gargron@mastodon.social
       2018-10-27T01:00:57Z
       
       0 likes, 1 repeats
       
       The one monitoring tool Mastodon desperately needs is something that would scream alarm if the Sidekiq queue is growing faster than it's being processedThis message is brought to you by 600k backlogged items on mastodon.social because of something I am still investigating (table lock-up?)
       
 (DIR) Post #790695 by likerOfPosts@mastodon.social
       2018-10-27T01:27:42Z
       
       0 likes, 0 repeats
       
       @Gargron hey we should make mastodon but red
       
 (DIR) Post #790812 by wclayf@mastodon.social
       2018-10-27T01:47:44Z
       
       0 likes, 0 repeats
       
       @Gargron But that's a great problem to have. Glad to see ActivityPub rockin' and rollin'. I'm currently adding it to "SubNerd" (sbnode.com). I bet the MongoDB in SubNerd can eat up 600K messages per nanosecond.....or not.
       
 (DIR) Post #790857 by roguelazer@mastodon.technology
       2018-10-27T01:53:42Z
       
       0 likes, 0 repeats
       
       @Gargron My usual recommendation is to store an enqueue timestamp in each job and then monitor the mean/median/p95 time in queue using statsd or some other standard tool. It's a more reliable metric than raw queue size or derivatives thereof.
       
 (DIR) Post #791022 by ben@mastodon.lubar.me
       2018-10-27T02:05:17Z
       
       0 likes, 0 repeats
       
       @Gargron I think the instance fedibash is on was getting hit pretty hard by "insult generator bot" talking to fedibash and neither of them understanding what was going on and just replying back and forth multiple times per second
       
 (DIR) Post #791359 by mperham@mastodon.xyz
       2018-10-27T02:26:15Z
       
       0 likes, 0 repeats
       
       @Gargron Watch backlog or latency? https://github.com/mperham/sidekiq/wiki/Monitoring#monitoring-queue-backlog