[HN Gopher] The Hardest Scaling Issue
___________________________________________________________________
The Hardest Scaling Issue
Author : zdw
Score : 56 points
Date : 2023-01-27 22:59 UTC (1 days ago)
(HTM) web link (blog.codeberg.org)
(TXT) w3m dump (blog.codeberg.org)
| rigelbm wrote:
| Isn't there a wide ecosystem of options between "I have to
| physically go to DC to configure new server" and "Storing
| everything in EvilCorp Public Cloud"? There are several hosting
| providers out there that offer anything from bare metal servers,
| to VPS, to OpenStack, to Kubernetes and beyond. Going into a DC
| physically to configure a new server sounds like the hardest
| option possible short of them managing the DC themselves.
|
| I don't feel like the author gives enough rationale for the
| choices made. Making the decision so binary (Hard Way vs
| EvilCorp), it becomes easier to justify "Hard Way". I think if
| author accepts that there could be other options, maybe that will
| help with the problem stated.
| bee_rider wrote:
| It takes 8 paragraphs to work up to:
|
| > The hardest scaling issue is: scaling human power.
|
| I just point this out because the author makes a couple head-
| fakes at technical issues, haha.
| leowoo91 wrote:
| That could be the title even tbh.
| nequo wrote:
| Is it optimal for Codeberg to host on their own hardware? They
| write that it puts immense pressure on them: We
| transitioned from a basic stack (ext4 filesystem, Postfix, not
| much more) to one including so many shiny new things: Starting
| with the duty of maintaining own hardware (which is a journey on
| its own), we introduced LXC containers, Ansible, BTRFS, ZFS, Ceph
| and more. In the past months, I found myself reading a
| lot of documentation ... Still, there was no time left to dig
| deep into Ansible and ZFS, and by now both tools have even been
| mostly dropped, to reduce the stress on our team members.
|
| They write that self-hosting also leads head-on to the trade-off
| of what to spend scarce donor money on: For
| donation-powered non-profits, this uses donations that could have
| served better, and leads to burn-out in the team.
|
| The argument against cloud hosting is that the data then resides
| with a big hosting company: No matter which
| instance you join or if you host yourself, the data will reside
| on one of the few big cloud provider's systems.
|
| But Codeberg's mission is to host public repositories of open
| source projects. Why is it a problem if the data sits on someone
| else's hardware? Wouldn't the mission be better served by
| focusing on "Codeberg the software" instead of "Codeberg the
| hardware?"
| latchkey wrote:
| We run petabytes of storage, and growing.
|
| We dropped Ceph (which is too complex with a minefield of
| issues, and few people have deep experience in it) and just use
| ZFS (which tons of people have experience in and performs
| flawlessly).
|
| We also use Ansible since it automates and codifies our
| deployments. I can't imagine dropping it... in favor of what?
| Manually typing install commands on servers?
|
| It is good they have lifted the covers on their service in this
| blog post. It screams "stay away".
| arter4 wrote:
| Out of curiosity, are you running ZFS across datacenters or
| do you have a single DC?
|
| My understanding is, ZFS doesn't inherently provide
| clustering, so each server has its own pool. If you have
| multiple DCs, how do you handle disaster recovery? Do you use
| async replication?
| latchkey wrote:
| We have multiple DC's although the data isn't replicated
| across DC's. We are a Filecoin storage provider. Users of
| that network choose where they want to store things and are
| effectively responsible for geo data replication, then
| using the protocol we can prove to them that their data is
| on our machines.
| phphphphp wrote:
| > We run petabytes of storage, and growing.
|
| That can't be right -- can it? They have less than 50,000
| users. For petabytes of storage to be necessary, their
| average per-user storage would need to be _dozens of
| gigabytes_. I 've been using GitHub for more than a decade
| and the sum total of every repository I've published is less
| than that. I found a few repositories on Codeberg that are
| using gigabytes of storage but the majority are normal sized
| Git repositories.
| latchkey wrote:
| I'm talking about my own business.
| scraptor wrote:
| If codeberg's mission only to host public repos with no further
| qualifiers they could just use github. The whole point is to
| not use services provided by untrustworthy and ideologically
| misaligned megacorps.
| nequo wrote:
| I sympathize with that. I do see a point in-between those two
| extremes though.
|
| I don't want to give GitHub free advertising by hosting my
| code with them nor do I want to feed their ML models. The
| latter is not completely avoidable since someone could mirror
| my repo on GitHub or they could just scrape it from Codeberg.
| But the former is successfully avoided by using Codeberg,
| even if Codeberg is hosted in the cloud.
| jacooper wrote:
| 100%, you can't always reach your goal in the ideal way, so
| don't put up with unnecessary things that hinder your progress
| towards you end goal.
| prhrb wrote:
| Just host in the cloud and scale it with a few clicks
| theideaofcoffee wrote:
| I don't know if this was missing a '/s' but... it's never
| "just" (a four letter word in operations) and never "a few".
| cbrozefsky wrote:
| Just read the article before commenting with "just do X"
| MoOmer wrote:
| Presumably cost/performance are of concern; along with probably
| wanting to use CephFS directly over an NFS.
| sekh60 wrote:
| What's wrong with using CephFS over NFS? cephadm can
| automatically deploy NFS ganesha containers to serve out
| NFSv4, it's all automated.
| debarshri wrote:
| Question is should they even take onus of hosting the service.
| Design wise, it could be selfhost the repositories and discover
| by a service managed by codeberg. That would save lot of donors
| money. I'm sure the storage cost alone runs into few thousands
| every month. I'll argue if it is not-for-profit org, does it
| justify this cost. To me it feels like at one moment, they have
| to make a hard call to switch the model.
___________________________________________________________________
(page generated 2023-01-28 23:01 UTC)