Post APOjjHVoKn87ThUlE0 by offby1@wandering.shop
(DIR) More posts by offby1@wandering.shop
(DIR) Post #APJYmTbINr1OnVFfV2 by tacertain@infosec.exchange
2022-11-06T02:43:03Z
0 likes, 3 repeats
If there's anybody out there running a large (*) Mastodon server and needs help/advice scaling, hit me up and I'll help if I can. Please contact on Twitter or LinkedIn as well as here. I'm not confident in my notification settings. Feel free to RT (I guess that's reToot) for reach.(*) I.e. if you're just struggling to get one running for you and your buddies, that's not what I mean. But if you're suffering from thousands of new users driving up load, I might be able to help. I've dealt with one or two scaling issues in my time...
(DIR) Post #APNM8vhrRCI7JurvBw by etoani@freeradical.zone
2022-11-07T23:28:12Z
0 likes, 0 repeats
@tacertain We were just talking about setting up one, for the learning. Have you tried? I've been favorably impressed how @tek manages freeradical.zone. I'm not nearly prepared to match their onprem raided hardware configuration.
(DIR) Post #APNM8wGbM2533f7fLE by tek@freeradical.zone
2022-11-07T23:31:42Z
0 likes, 0 repeats
@etoani @tacertain In fairness, I've been doing this for over 5 years now. You're seeing the end results of all the mistakes I made along the way, not all the goofy experiments I tried and failed at. 😀
(DIR) Post #APNN8xV4rtI8q0T1E0 by tacertain@infosec.exchange
2022-11-07T23:36:39Z
0 likes, 0 repeats
@etoani @tek I have not. @offby1 runs wandering.shop, and I've been chatting with him a little about his experience
(DIR) Post #APNN8xx5Blh2ErZNsO by offby1@wandering.shop
2022-11-07T23:39:07Z
0 likes, 0 repeats
@tacertain @etoani @tek ... and my situation is that I inherited both the shop's easy mode infra (it was never intended to be SUPER big) and a period of great growth in the Fediverse, as well as some serious cost constraints.... still, it's getting better. Running all of a 1000-user Mastodon instance on a shared-vCPU cloud host is probably not the direction I'd recommend you take it, but with some engineering you can get pretty far.
(DIR) Post #APNN8zhuf8kNgOUIVc by etoani@freeradical.zone
2022-11-07T23:41:44Z
0 likes, 0 repeats
@offby1 @tacertain @tek Having scaled social networks before.... I would blanche at the AWS cost. Federation has an upside, but will hit the wall if you are hosting some heavily subscribed users.
(DIR) Post #APNN904FK6bynew8Js by tek@freeradical.zone
2022-11-07T23:42:49Z
0 likes, 0 repeats
@etoani @offby1 @tacertain I'm currently at about $50/mo with 4,000 users, and some growth headroom. If I weren't able to self-host some of the hardware, it'd probably be closer to $100.
(DIR) Post #APNP0SAnep0pGp3RJY by offby1@wandering.shop
2022-11-08T00:03:43Z
0 likes, 0 repeats
@tek @etoani @tacertain I'm curious: what's your hosting provider?
(DIR) Post #APNP4C45UPUeTBhM3s by tek@freeradical.zone
2022-11-08T00:04:27Z
0 likes, 0 repeats
@offby1 @etoani @tacertain I use DigitalOcean for the VPC, and my living room for the physical stuff.
(DIR) Post #APOjjHVoKn87ThUlE0 by offby1@wandering.shop
2022-11-08T15:30:34Z
0 likes, 0 repeats
@tek @etoani @tacertain ah. I've been avoiding the idea of self-hosting on my residential internet. I imagine it'd be way cheaper, but... a bit scary to handle.
(DIR) Post #APOkcx7NLWhaRYbzOa by tek@freeradical.zone
2022-11-08T15:40:48Z
0 likes, 0 repeats
@offby1 @tacertain @etoani My ISP is fine with it, and the connection is generally excellent.
(DIR) Post #APOlzDHG2wcGszPe4W by tacertain@infosec.exchange
2022-11-08T15:55:56Z
0 likes, 0 repeats
@tek @etoani @offby1 How many subscribed posts per day does that translate to? I assume that's the main scaling factor?
(DIR) Post #APOmz3EJVEMBg7WQm8 by tek@freeradical.zone
2022-11-08T16:07:12Z
0 likes, 0 repeats
@offby1 @tacertain @etoani I hadn’t thought to count that until now, and here’s what I found: https://freeradical.zone/@tek/109309036456312761And yeah, that’s definitely the main scaling factor. It’s a byproduct of #users * $engagement, where engagement among very new users tends to be very high.
(DIR) Post #APOvSpF8ruBGcWcNns by etoani@freeradical.zone
2022-11-08T17:42:08Z
0 likes, 0 repeats
@tek @offby1 @tacertain I would expect the standard deviation of follows to be pretty big. But as the network grows the follow counts grow as well both for new and old users. If growth continues it will be interesting to see if cache pressures or federation overhead are the next scaling issue. I enjoyed the RPi solution to worker threads!
(DIR) Post #APOxMFTN9JDDqpGXhY by tacertain@infosec.exchange
2022-11-08T18:01:06Z
0 likes, 0 repeats
@etoani @tek @offby1 I would expect both follows and post frequency to follow a power law, which means that scaling will be dominated by a few users who follow a few users that post a *lot*.
(DIR) Post #APOxMG74lgyHpxqFaS by tek@freeradical.zone
2022-11-08T18:03:23Z
0 likes, 0 repeats
@tacertain @offby1 @etoani I think you’re right. The good news is that 2 people following the same person should take much more work than 1 person following them. If we get 100 new users who like you, say, then it’s not 100x the load since we’re already interacting with you.
(DIR) Post #APOxYLqLI1Dt5FmTdA by tacertain@infosec.exchange
2022-11-08T18:05:35Z
0 likes, 0 repeats
@tek @offby1 @etoani for sure (assuming you meant *not* much more work 🙂)
(DIR) Post #APOxih7t0z7xzHv2TA by etoani@freeradical.zone
2022-11-08T18:07:30Z
0 likes, 0 repeats
@tek @tacertain @offby1 So really its not that we all follow a lot of accounts, as that we follow a lot of discrete accounts. Post frequency is only really a problem if anyone is listening I would guess, since that hits federation overhead.
(DIR) Post #APOxvCF79MOty8GFpg by tek@freeradical.zone
2022-11-08T18:09:40Z
0 likes, 0 repeats
@etoani @tacertain @offby1 To a first approximation, yeah.
(DIR) Post #APOxws9MpDbAEQN8JU by tek@freeradical.zone
2022-11-08T18:09:56Z
0 likes, 0 repeats
@tacertain @offby1 @etoani Oops! Yep.
(DIR) Post #APOy6FIg7fGadYCGzQ by offby1@wandering.shop
2022-11-08T18:11:33Z
0 likes, 0 repeats
@tek @etoani @tacertain I've assumed that the actual load is tied to how many federated servers you follow; I hoped that there would be some consolidation of updates at the server level.
(DIR) Post #APOyCzu1ljViuQt0Eq by tek@freeradical.zone
2022-11-08T18:12:55Z
0 likes, 0 repeats
@tacertain @etoani @offby1 I don’t know enough of the details to answer that.
(DIR) Post #APOyXdXc6U8RdaFB0i by tacertain@infosec.exchange
2022-11-08T18:16:34Z
0 likes, 0 repeats
@tek @offby1 @etoani More generally, most users are not going to cost much to add, a few users are going to dominate your scaling needs. You could probably compute that relationship if you were curious. There are three ways you might look at it: 1/ For each user, sum up the number of incoming posts from all the subscriptions for that user, counting 1 per post. That shows you which users are consuming a lot, but it over counts impact because of sharing.2/ time instead of counting 1 per user, count 1 divided by the total number of users subscribed to that feed. This gives you a "shared cost" estimate of what each user is costing, but still over counts the marginal cost of each user.3/ Do the same as /2/, but only count the posts for which the denominator is 1 (i.e. posts where the subscribed user is the only subscriber on your host). This under counts your total cost, but tells you the marginal cost of each user.