Post APbalyCsJZ9XSbnFui by chendo@mastodon.social
 (DIR) More posts by chendo@mastodon.social
 (DIR) Post #APbalyCsJZ9XSbnFui by chendo@mastodon.social
       2022-11-14T06:42:48Z
       
       0 likes, 1 repeats
       
       Watching instance owners needing to react to the sudden influx of users has me reminiscing about Switter early days.In March 2018, we (Assembly Four) deployed Mastodon to a VPS with 1.5GB of RAM and launched https://switter.at/ in response to FOSTA/SESTA that was pushing sex workers off platforms.Word spread like wildfire, and I had to double resources 2 hours after the announcement. And had to double it 2 hours after that. We had to scale out in the first week.#devops #mastodon 1/x
       
 (DIR) Post #APbam0YDbBPijNnKSW by chendo@mastodon.social
       2022-11-14T06:53:09Z
       
       0 likes, 0 repeats
       
       I had originally containerised Mastodon and ran it on Dokku, as I figured that was the easiest option at the time. However, Dokku back then did not have the ability to scale out. Considering scaling up was going to hit limits real soon, I figured the easiest option was to split stateful stuff on to its own machine, and then installed Dokku on 10 VMs, and deployed Mastodon to all of them.git pushing to 10 VMs using iTerm's simultaneous input was a huge hack but it worked  ¯\_(ツ)_/¯ 2/x
       
 (DIR) Post #APbam1xmLdSD6wlFwG by chendo@mastodon.social
       2022-11-14T06:55:54Z
       
       0 likes, 0 repeats
       
       Our stats for 1 week and 1 month in, so you can attempt to plan ahead:One week in:* 13k users* 71k toots* 1.03TB of bandwidth* 21.5M requests* 57k emails sentOne month in:* 56k users* 491k toots* 12TB of bandwidth* 148M requests* 340k emails sentOur biggest costs (ignoring time spent) were compute and email delivery.3/x
       
 (DIR) Post #APbam3Nh4lmHVbtSyG by chendo@mastodon.social
       2022-11-14T06:58:14Z
       
       0 likes, 0 repeats
       
       Then we got kicked off Cloudflare with zero warning. We had originally trusted them because they fought for free speech and took forever to kick off neo nazis off their platform, so we assumed we'd be safe. See https://www.theverge.com/2018/4/19/17256370/switter-cloudflare-sex-workers-banned for more info.Our infra costs doubled that day. CDNs are expensive. We ended up switching to a dedicated server to save on costs, which was good enough at the time.Ultimately, the overhead of running 10 VMs running Dokku got too much4/x
       
 (DIR) Post #APbam4sDWlmu8ZBMBc by chendo@mastodon.social
       2022-11-14T07:03:42Z
       
       0 likes, 0 repeats
       
       I taught myself enough Kubernetes (I've had previous experience with orchestration when we used DCOS/Mesosphere when I was at LIFX), and with the help of Rancher, spun up a cluster and moved our workloads to that. Adding capacity became comparatively trivial, and we gained the ability to use horizontal autoscalers for both our web and sidekiq processors.However, k8s is a complex beast, and then you need to figure out how to operate k8s as well as Mastodon.5/x
       
 (DIR) Post #APbam6OrqrV0s7SwiW by chendo@mastodon.social
       2022-11-14T07:09:03Z
       
       0 likes, 0 repeats
       
       Unfortunately, after being kicked off Cloudflare (🖕), we were subjected to DDOS attacks and had to deal with that ourselves. https://Deflect.ca by eQualitie looked great, however they did not support websockets back then.Ultimately, we ended up hacking together our own global edge with an-inhouse L7 WAF capable of dropping hostile IPs across our edge.Hopefully you'll never have to get this far. Deflect.ca handles websockets now I believe, so I'd recommend them.6/x
       
 (DIR) Post #APbam7z3xm2vmfPMm0 by chendo@mastodon.social
       2022-11-14T07:14:48Z
       
       0 likes, 0 repeats
       
       2 months in, Switter had 72k users, and averaged 420GB of traffic per day. A year later, it had 243k users.When we had to shut Switter down due to shit laws being passed in Australia, it was:* the 3rd biggest Mastodon instance in the fediverse* 435k users* 9.1M toots* 8.6M replies* 16.6M reblogs* 4.9M favouritesThe hardest problems IMO were moderation, dealing with spam, and managing our own mail server. Everything else was relatively simple by comparison.7/x
       
 (DIR) Post #APbam9fzfdyt26VAKO by chendo@mastodon.social
       2022-11-14T07:17:14Z
       
       0 likes, 0 repeats
       
       Instance owners: be wary of what your legal liability is regarding the content on your instance. GDPR etc etc.Users: be wary of your instance's legal liabilities and assume they can read everything you post, even if it's private. Assume the instance you're on may not be around forever.Anyway, that's my rant over. Questions?For more info and context, see https://medium.com/assembly-four/my-six-week-rollercoaster-ride-172eb58ba80e and the memorial at https://switter.at