[HN Gopher] Designing Our Serverless Engine: From Kubernetes to ...
___________________________________________________________________
Designing Our Serverless Engine: From Kubernetes to Nomad,
Firecracker, and Kuma
Author : eric_khun
Score : 107 points
Date : 2021-07-07 14:59 UTC (8 hours ago)
(HTM) web link (www.koyeb.com)
(TXT) w3m dump (www.koyeb.com)
| JanMa wrote:
| Nice post, thanks for sharing it.
|
| From personal experience I can only agree with their choice to
| pick Nomad. At the place where I work we have been running Nomad
| as our main container orchestrator for around 2.5 years now. It's
| rock solid, very easy to set-up and maintain and overall not too
| complex to understand in depth.
| jimaek wrote:
| Great post and a great tech stack.
|
| As far as I know Fly.io also went the same path of using the
| Nomad stack and tools.
|
| We did something similar at appfleet.com and also decided against
| Kubernetes. We opted to write a lightweight manager of
| Firecracker VMs.
| rockyluke wrote:
| Thanks for sharing! That's currently an uncommon change but I
| must admit you explained it well.
|
| Quick question for the team: did you consider or try Consul as a
| service mesh ?
| yann_eu wrote:
| Hi!
|
| Yann, co-founder at Koyeb, here.
|
| Yes, we considered Consul as a service mesh but found that it
| was too strongly coupled with the network and task layer of
| Nomad. We were looking for something highly customizable which
| wouldn't get into the way.
|
| Also, the multi-tenancy features are paid features, which might
| have been difficult to sustain economically for us.
| tkiolp4 wrote:
| I'm waiting the day someone discovers a simpler alternative to
| k8s. It will be like that day in which we realized that http
| verbs >>> Corba/rmi for web services.
| lloydatkinson wrote:
| I'm hoping that this is the start of a trend away from the overly
| complex, overly engineered, big ball of mud that is Kubernetes
| towards simpler and easier deployment/hosting/container runtimes
| that are easier for smaller and large teams alike to manage.
|
| Don't get me started on how "devops" has become a meme. I want to
| deploy services and features. I wouldn't want to spend my full
| time dealing with Kubernetes and containers and all manner of
| complexity.
|
| I think cloud providers have much better models generally (e.g.
| Azure App Service). Setup pipeline, deploy, done. Containers are
| an optional thing.
|
| What I'm saying is, I don't think it's fair on anyone that
| essentially the only choices are 1) run servers and VM's yourself
| and manage deploying to them 2) kubernetes 3) cloud providers.
|
| There needs to be an option between them.
| dmitriid wrote:
| > I'm hoping that this is the start of a trend away from the
| overly complex, overly engineered, big ball of mud that is
| Kubernetes towards simpler and easier
| deployment/hosting/container runtimes that are easier for
| smaller and large teams alike to manage.
|
| But _is it_ easier to manage? _Is it_ simpler?
|
| So they went from kubernetes to
|
| - Control Pane running in
|
| - Container running on
|
| - Firecracker with Kuma running on
|
| - Nomad with CoreOS running on
|
| - Some servers
|
| No idea what half of those words are, how they all are
| ocnfigured, interact with each other, and fail in spectacular
| ways.
| kfk wrote:
| Kubernetes is complex ok but I disagree on cloud providers
| offering better alternatives, especially Azure. Azure is bug
| ridden and support depends on what type of price tier you are
| in. The thing with Cloud providers is when their services go
| down or change you are on the hook for supporting your app, not
| them. At least Kubernetes (or Nomad) lets you mange your own
| dumpster fire around which you can build safer deployment
| workflows.
| mountainriver wrote:
| There have been a ton of in between options, they just rarely
| work well for anything outside of a basic use case.
|
| Kubernetes is complex because the reality of what you're
| dealing with is complex.
|
| Kubernetes is way better than how we used to do things, not a
| perfect system but there are a lot of good reasons why it's
| built the way it is.
| seer wrote:
| Not sure about the other clouds, but GCP is just atrocious for
| this with both of their offerings.
|
| App Engine, seems to be no longer developed, and is considered
| "old" technology, it works, but its hard to do more enterprisy
| stuff, like virtual private cloud (VPC) with it. For example
| you can configure it to be able to access a VPC, but you cannot
| put it "inside" the VPC so others services can interact with
| it.
|
| Cloudrun seems to be the new hot thing, and it works too, seems
| to be made using kubernetes itself, so promises future
| development and integration. However there is a tiny flaw in
| the system, berried in a fine print in an obscure doc page.
| Cloudrun severely throttles cpus on any containers that are not
| currently processing an http request. So you cannot build
| anything that is long running - kafka consumers, batch
| processes etc. What you are supposed to do is delve head first
| into the whole of GCPs ecosystem and embrace their queues,
| message brokers, and event sourcing systems (as they're all
| http request based), woe to the ones that have to interact with
| tech thats outside of it.
|
| /rant
| Thaxll wrote:
| There is nothing complicated to deploy a single deployment on
| Kubernetes, people that say so either don't use Kuberentes or
| are just lying.
|
| https://kubernetes.io/docs/concepts/workloads/controllers/de...
|
| What's complicated is to operate a Kubernetes cluster, nothing
| forces you to do that, major cloud provider all do it pretty
| well.
|
| As for OP post lot of things don't make sense, I think the only
| valid reasons is about isolation which k8s is not very good at
| for now ( multi tenancy ). The rest is interesting... let them
| build there own solution to maintain and operate and see how it
| goes for them in 2 years.
|
| Things that I found strange:
|
| - Complexity "understanding how Kubernetes works and why it was
| implemented this way is hard" So instead they're building their
| own tools? If you have the knowledge to build a similar
| platform, extanding and understanding k8s is easy.
|
| - Global and multi-zone deployments "Implementing multi-zone
| with Kubernetes requires deploying a full cluster per zone,
| with a dedicated control plane for each data center." good luck
| with a single control plane controling all of you regions, one
| little problem will bring down all your servers. You should
| always have isolated regions. Nothing prevent you from building
| some sort of orchestration around multi region deployments.
|
| - Scalability: Kubernetes is known to have limits in terms of
| the number of nodes in a cluster. Yes there are limits, but
| looking at your startup I bet that 5000 nodes for a single
| cluster is more than what you need in a single region.
| https://kubernetes.io/docs/setup/best-practices/cluster-larg...
| dilyevsky wrote:
| Ah yes still waiting on that infinitely scalable, simple and
| cheap option too.
|
| Edit: forgot reliable
| tantalor wrote:
| > Kubelet uses between 10% and 25% of RAM ... We're more around
| 100MB with our new architecture
|
| These figures are not comparable.
| dilyevsky wrote:
| Highly questionable figures. Our kubelets have a 2% memory
| limit which I've never seen hit as it's a hugely over-
| provisioned.
| pm90 wrote:
| > Global and multi-zone deployments: User workloads on Koyeb need
| to be able to run in multiple zones. Kubernetes doesn't support
| multi-zone out of the box. Implementing multi-zone with
| Kubernetes requires deploying a full cluster per zone, with a
| dedicated control plane for each data center.
|
| I'm having trouble understanding this. K8s worker nodes should be
| deployable across zones, regions etc. You can label the nodes
| with the zone id and use taints/tolerations to ensure workloads
| are deployed in specific zones/regions (if that's what you want).
| bproven wrote:
| yeah i am not sure if they mean "region" (as in AWS region)
| instead of zone. if that is the case then I agree with their
| assessment.
| dilyevsky wrote:
| Yeah pretty sure it's that bc k8s 100% supports multi-az on
| every major cloud provider out of the box. It also supports
| federation via kubefed but it needs a separate control plane
| in each region.
|
| In theory nothing should be stopping you from deploying your
| etcd/master nodes in different regions but you'd probably
| need to tweak cloud resource provider to handle that and if
| one of regions is partitioned away from quorum those master
| become unavailable
| rileymichael wrote:
| You certainly can deploy workers across regions, however the
| latency to the control plane makes it quite unpleasant.
| staticassertion wrote:
| I'm curious - when a user deploys their new code to your product,
| does that kick off a new Nomad job, or is that managed
| internally, kicking off a koyeb-managed Firecracker with Kuma for
| service discovery?
| ngrilly wrote:
| Seems similar to fly.io, but fly.io seems better: uses WireGuard
| instead of service mesh, supports custom domains, supports any
| TCP or UDP service, provides volumes, etc. But that's great
| seeing more options in that market!
___________________________________________________________________
(page generated 2021-07-07 23:01 UTC)