[HN Gopher] Kubernetes is hard
       ___________________________________________________________________
        
       Kubernetes is hard
        
       Author : suralind
       Score  : 103 points
       Date   : 2023-03-27 20:19 UTC (2 hours ago)
        
 (HTM) web link (rcwz.pl)
 (TXT) w3m dump (rcwz.pl)
        
       | [deleted]
        
       | 0xbadcafebee wrote:
       | > Kubernetes is complex and I think they are partially right
       | 
       | Kubernetes is a distributed centralized operating system which
       | itself depends on a distributed decentralized database, and has a
       | varying network topology, permissions system, plugins, scheduler,
       | storage, and much more, depending on how & where it was built,
       | and runs applications as independent containerized environments
       | (often deeply dependent on Linux kernel features) which can all
       | have their own base operating systems. All of which must be
       | maintained, upgraded, patched, and secured, separately, and
       | frequently.
       | 
       | Kubernetes is literally the most complex single system that
       | almost anyone in the world will ever use. It is the Katamari
       | Damacy of the cloud.
       | 
       | > It allows dev teams to not worry about all these things; all
       | they must do is to write a simple YAML file.
       | 
       |  _cackles, then sobs_
       | 
       | > More importantly, teams no longer need to ask DevOps/infra
       | folks to add DNS entry and create a Load Balancer just to expose
       | a service.
       | 
       |  _more sobbing_
       | 
       | > Should you use Kubernetes?
       | 
       | Should you change a tire with a crowbar, a can of WD40, and a
       | lighter? Given an alternative, the alternative is usually better,
       | but sometimes you don't have an alternative.
        
       | paxys wrote:
       | Kuberenetes has been a total failure at defining a simple devops
       | workflow, but I don't think that is due to any deficiencies in
       | the product. The basic premise behind its common use case -
       | automating away the SRE/ops role at a company - is flawed in
       | itself. Companies that blindly make the switch are painfully
       | finding out that the job of a system operator wasn't just to
       | follow instruction checklists but apply reasoning and logic to
       | solve problems, similar to that of any software engineer. And
       | that's not something you can replace with Kubernetes or any other
       | such tool.
       | 
       | On the other hand, there's still a lot of value in having a
       | standard configuration and operating language for a large
       | distributed system. It doesn't have to be easy to understand or
       | use. Even if you still have to hire the same number of SREs, you
       | can at least filter on Kubernetes experience rather than having
       | them onboard to your custom stack. And on the other side, your
       | ops skills and years of experience are now going to be a lot more
       | transferrable if you want to move on from the company.
        
       | overgard wrote:
       | I agree with the point that production is hard. There's so many
       | things you just don't think about as a developer that end up
       | being important. Log storage, certificate renewal, etc.
       | 
       | I think how "hard" kubernetes is depends on how deep you go. If
       | you're building a cluster from scratch, on your own hardware,
       | setting up the control plane yourself etc. it's very very hard.
       | On the other hand, if you're using a hosted service like EKS and
       | you can hand off the hardware and control plane management to
       | someone else, IMO it's actually very easy to use; I actually find
       | it a lot easier than working with the constellation of services
       | amazon has to offer for instance.
       | 
       | I do think there are parts of it where "best practices" are still
       | being worked out though, like managing YAML files. There's also
       | definitely some rough edges. Like, Helm charts are great... to
       | use. They're an absolute nightmare to write, and there's all
       | sorts of delightful corner cases like not being able to reliably
       | upgrade things that use StatefulSet (last I used anyway). It's
       | not perfect, but honestly if you learn the core concepts and use
       | a hosted service you can get a lot out of it.
        
         | ActorNightly wrote:
         | I disagree that production is hard in itself, I think the way
         | people approach production makes it needlessly more difficult.
         | It seems that when launching a service, everyone seems to try
         | to get the entire stack in one shot, which is difficult.
         | 
         | A much better approach is "1. Make it work 2. Make it good 3.
         | Make it fast". Your initial prototypes need to define the core
         | functionality, and then you incrementally build stuff on top of
         | that.
        
           | yourapostasy wrote:
           | When I see developers use k8s as "programmable infra
           | replacing on-prem functionality as a baseline", some time
           | after step 3 is, "make it as cheap as on-prem".
           | Unfortunately, many of the "move fast and break things" tenor
           | that got the services built in k8s in the first place, baked
           | in decisions that will cost as much to re-engineer into an
           | elastic cloud-savvy form that also saves money compared to
           | on-prem.
           | 
           | This is often from organizational or project management mis-
           | coordination than technical choices the developers are fully
           | aware of. Like a system's owners not prioritizing putting
           | events of interest into a message queue and insisting upon
           | polling for the information on one end. And an offshore
           | operations team saying learning how to support AWS
           | Lambda/Azure Automation/GCP Cloud Functions is an added cost
           | scope to the contract. So the developer is backed into a
           | corner to run cron/SystemD-based polling off a small EC2.
           | Thousands of decisions like this add up fast, and take a
           | major executive committee commitment to root out tech debt
           | and prevent new tech debt from accumulating to unsustainable
           | levels to prevent cloud initiatives from sinking beneath a
           | mire of expensive small decisions in pursuit of new
           | accomplishments. It is a tough balancing act.
        
           | lcw wrote:
           | > 1. Make it work 2. Make it good 3. Make it fast
           | 
           | Anecdotally, everytime someone tells me Kubernetes is
           | overkill and then follows your approach. In a year they end
           | up building the capabilities that come out of the box with
           | Kubernetes, and of course they aren't as well thought out.
           | Because they were done adhoc and as needed.
        
           | YetAnotherNick wrote:
           | > 1. Make it work 2. Make it good 3. Make it fast
           | 
           | Disagree. The first two objective clashes very much with each
           | other. I have worked in teams that did that. And generally if
           | anything non trivial is done without planning to just "make
           | it work", it would be more often than not, ridden with many
           | design issues which doesn't allow it to be good.
        
           | Veuxdo wrote:
           | Security as well. Now step 0 is "Make it secure". When using
           | microservices connected via the internet, you have no choice.
        
             | madeofpalk wrote:
             | > When using microservices
             | 
             | 1. Make it work
        
         | sshine wrote:
         | > If you're building a cluster from scratch, on your own
         | hardware, setting up the control plane yourself etc. it's very
         | very hard.
         | 
         | I'd like to make a distinction:
         | 
         | If you do cluster deployment from scratch, Kubernetes actually
         | gets easier.
         | 
         | Because you get good at it. What it also gets is more time-
         | consuming.
         | 
         | If you aim to replicate all of AWS, then unrealistically so.
         | 
         | The art is knowing when to stop. For a lot of people that's
         | even before Kubernetes.
         | 
         | For others, you can run all your apps in a cluster, except
         | persistent storage.
        
           | kazen44 wrote:
           | At the end of the day, getting a stable production
           | environment is simply a tradeoff between the amount of
           | complexity you need to make your infrastructure do what you
           | want, and reducing complexity because it removes failure
           | points from the production environment.
           | 
           | K8s is nice and all, but if all you really need can be solved
           | by 2 VM's and a way to announce an anycast address (or use a
           | loadbalancer if that is not an option), why would i add all
           | that complexity?
        
             | sshine wrote:
             | I totally agree.
             | 
             | I've gotten away with docker-compose for container
             | orchestration for the last 16 months.
        
             | b33j0r wrote:
             | For reasons of experience, all I ever want from a system is
             | that it's reproducible.
             | 
             | I had a vanity website running k8s in a managed cloud. I
             | thought I was backed up by my provider and original ansible
             | deployment, which was of course developed iteratively.
             | 
             | I originally did this mostly to do a practice deployment,
             | and get the workflow.
             | 
             | A few years later, it went down and I didn't notice for a
             | few weeks. It was too unimportant to invest in monitoring,
             | and not worth it to do over. Redeploying gave me lots of
             | confusing errors (note: I also do this stuff for work)
             | 
             | Frankly, I was surprised that the provider doing updates
             | would make my years-stable site fall over. I haven't tried
             | that one again for funsies, yet. It's the vendor
             | specificity that was my d'oh!
        
       | endisneigh wrote:
       | Cost aside, I wonder how far you can get with something like a
       | managed newsql database (Spanner, CockroachDB, Vitess, etc.) and
       | serverless.
       | 
       | Most providers at this point offer ephemeral containers or
       | serverless functions.
       | 
       | Does a product focused, non infra startup even need k8s? In my
       | honest opinion people should be using Cloud Run. It's by far
       | Google's best cloud product.
       | 
       | Anyway, going back to the article - k8s is hard if you're doing
       | hard things. It's pretty trivial to do easy things using k8s,
       | which only leads to the question - why not use the cloud
       | equivalents of all the "easy"things? Monitoring, logging,
       | pub/sub, etc. basically all of these things have cloud
       | equivalents as services.
       | 
       | The question is, cost aside, why use k8s? Of course, if you are
       | cost constrained you might do bare metal, or a cheaper
       | collocation, or maybe even a cheap cloud like DigitalOcean.
       | Regardless, you will bear the cost one way or another.
       | 
       | If it were really so easy to use k8s to productionize services to
       | then offer as a SaaS, everyone would do it. Therefore I assert,
       | unless those things are your service, you should use the cloud
       | services. Don't use cloud vms, use _cloud services_ , and
       | preserve your sanity. After all, if you're not willing to pay
       | someone else to be oncall, that implies the arbitrage isn't
       | really there enough to drive the cost down enough for you to pay,
       | which might imply it isn't worth your time either (infra
       | companies aside).
        
         | mirekrusin wrote:
         | Cloud services are shit unless I can run them locally when
         | developing and testing.
        
         | rs999gti wrote:
         | > In my honest opinion people should be using Cloud Run. It's
         | by far Google's best cloud product.
         | 
         | Is this the same thing as running containers in Azure App
         | Services?
        
           | endisneigh wrote:
           | It's similar, tho I'd say that is more like App Engine. The
           | azure equivalent is probably Azure container instances
        
         | davnicwil wrote:
         | Or just app engine honestly.
         | 
         | Works with docker containers so you can run the same simple
         | stack locally as in prod. No need for more exotic serverless
         | architectures.
         | 
         | Generous free tier, too!
         | 
         | Have only good things to say about it for quickly firing up a
         | product.
        
           | asciimike wrote:
           | > Does a product focused, non infra startup even need k8s? In
           | my honest opinion people should be using Cloud Run. It's by
           | far Google's best cloud product.
           | 
           | > Or just app engine honestly.
           | 
           | As a former App Engine PM and PM of Cloud Run, this warms my
           | heart to hear--I'm glad folks love to use it :)
           | 
           | It's been a few years since I've worked on these products,
           | but they were designed to be composed and used by each other.
           | Cloud Run provides an autoscaling container runtime with
           | Knative compatible API; Cloud Build (and Container Registry)
           | + Cloud Run = App Engine, App Engine + CloudEvents = Cloud
           | Functions.
           | 
           | With a Knative compatible API, you can theoretically take
           | those primitives and move them to your own K8s cluster,
           | managed (GKE/EKS/AKS) or self-managed, giving folks a
           | tremendous amount of flexibility down the line if/when they
           | need it (hint: the majority of customers didn't, and the
           | fully managed services were great).
        
           | brobdingnagians wrote:
           | Similar with fly.io, I have an application running on there
           | and pleasantly surprised they don't even charge if the amount
           | is under $5/month. I've been very happy with how easy it is
           | to deploy and with the hosted Postgres. I'm using the JVM and
           | works well; I originally played around with Elixir and was
           | especially impressed with their feature set for it.
        
         | solatic wrote:
         | This. Greenfield products should be serverless by default. By
         | the time you have sustained traffic to the point where you can
         | run the numbers and think that you could save money by
         | switching off serverless, that's a Good Problem To Have, one
         | for which you'll have investors giving you money to hire DevOps
         | to take care of the servers.
        
           | joshribakoff wrote:
           | Disagree
           | 
           | I tried to use lambda. Cold startup really is awful. You have
           | to deal with running db migrations in step functions or find
           | other solutions. The aurora serverless also does not scale to
           | zero. Once you get traffic you overload RDS and need to pay
           | for and setup a RDS proxy, and dont get me started on the
           | pointless endeavor of trying to keep your lambdas warm. Sort
           | of defeats the point. Serverless is not actually serverless
           | and ends up costing more for less performance and more
           | complexity
           | 
           | Its way simpler and cheaper to start with a single VPS single
           | point of failure, then over time graduate to running docker
           | compose or a single node k3 cluster on that VPS. And then
           | eventually scale out to more nodes...
        
             | evantbyrne wrote:
             | Serverless does not necessarily mean lambda. It could be
             | just about anything that runs containers for you. AWS ECS
             | has an offering called Fargate that I've been happy with
             | for our hosting. You are right though that the compute
             | costs are typically more than renting a traditional VPS.
             | There is definitely a tradeoff between labor and compute
             | costs.
        
             | vanjajaja1 wrote:
             | The point of serverless isn't to run 0 servers in your
             | downtime, its to abstract away everything related to
             | running hardware. I have an app that is built on a runtime
             | (jdk, node, whatever) and I shouldn't have to deal with
             | anything below that layer.
        
         | suralind wrote:
         | I think you could push that setup far. I'm not familiar with
         | GCP or Cloud Run, but it probably integrates nicely with other
         | services GCP offers (for debugging, etc.).
         | 
         | I'd be curious to read if anybody has that setup and what scale
         | they have.
        
         | suralind wrote:
         | Regarding the second part, I totally agree, either use cloud or
         | don't. For some reason, most companies want to be cloud-
         | agnostic and so they stay away from things that are too
         | difficult to migrate between cloud providers.
        
       | honkycat wrote:
       | I am consistently confused by all of the talk about how "hard"
       | Kubernetes is.
       | 
       | We spin up EKS. We install the newrelic and datadog log ingestion
       | pods onto it, provided in a nice "helm" format.
       | 
       | We install a few other resources via helm, like external secrets,
       | and external dns, and a few others.
       | 
       | Kubernetes EKS runs like a champ. My company saves 100k/mo by
       | dynamically scaling our cloud services, all of which are running
       | on Kubernetes, to more efficiently use compute classes.
       | 
       | My company has over 50 million unique users monthly. We have
       | massive scale. Kubernetes just works for us and we only have 2
       | people maintaining it.
       | 
       | What we gain is a unified platform with a consistent API for
       | developing our services. And if we wanted to migrate elsewhere,
       | it is one less thing to worry about.
       | 
       | -\\_(tsu)_/-
       | 
       | Feels like some kind of hipster instinct to dislike the "cool new
       | thing"... even though k8 has been around for years now and has
       | been battle tested to the bone.
        
         | Turbots wrote:
         | Your company "saves" over 100k/month paying WAY too much for
         | EKS, which is extremely expensive.
         | 
         | If you're at any decent scale (looks like you are), then switch
         | to GKE, or switch to on-prem and buy some hardware + a
         | Kubernetes distro like Mirantis/Openshift/Tanzu.
         | 
         | Heck, go run k3s on Hetzner and you won't have that much more
         | work, but save literally millions at the scale you're talking
         | about.
        
           | unity1001 wrote:
           | > Heck, go run k3s on Hetzner
           | 
           | The moment Hetzner starts providing managed DBs or managed
           | kubernetes, they will wipe the floor with all the big 3...
        
         | drdaeman wrote:
         | So, what do you do when one of your pods suddenly cannot
         | connect to another, even though both nodes seem to be passing
         | healthchecks and stuff?
         | 
         | Spinning up a K8s cluster "in the cloud" is easy and everyone
         | jumps on that happy "look how simple it all is" bandwagon,
         | forgetting than it's just the beginning of a very long journey.
         | There are millions of blog articles of varying quality that
         | explain how easy it is, because it's very simple to spam search
         | engines retelling a story how to click a couple buttons or do
         | some basic Terraform/CloudFormation/whatever.
         | 
         | And here's what they don't tell you - maintaining this
         | machinery is still _your_ business, because all you get is a
         | bunch of provisioned node machines and a cookiecutter to spin
         | it up. Plus a bunch of scripts to handle most common scenarios
         | (scaling, upgrading, some basic troubleshooting, etc). The rest
         | is either on your or tech support (if you pay extra for it).
         | And if you have a sysadmin-for-hire contract anyway, then its
         | them who should have an opinion what 's easy and what's hard.
         | Contracting other people is always relatively easy - compared
         | to what they do.
        
       | rglover wrote:
       | Kubernetes is hard because it's over-complicated and poorly
       | designed. A lot of people don't want to hear that because it was
       | created by The Almighty Google and people have made _oodles_ of
       | money being k8s gurus.
       | 
       | After wasting two years chasing config files, constant
       | deprecations, and a swamp of third-party dependencies that were
       | supposedly "blessed" (all of which led to unnecessary downtime
       | and stress), I swapped it all out with a HAProxy load balancer
       | server in front of some vanilla instances and a few scripts to
       | handle auto-scaling. Since then: I've had zero downtime and
       | scaling is region-specific and chill (and could work up to an
       | infinite number of instances). It just works.
       | 
       | The punchline: just because it's popular, doesn't mean it's the
       | best way to do it.
        
       | birdyrooster wrote:
       | It's not hard.
        
         | menacingly wrote:
         | something that considerably helped my communication was to
         | transition from "what this intelligent person said is patently
         | wrong" to "in what sense is this intelligent person correct".
         | 
         | It saves me and everyone else a lot of time, because if the
         | obvious black-and-white response were needed I probably
         | wouldn't be having the conversation in the first place.
        
       | gautamdivgi wrote:
       | You need a decent sized team to run an on premise k8s infra. If
       | you're in the cloud use a managed k8s.
       | 
       | It's not for everyone in that I agree with the point the author
       | makes. But if you have multiple teams doing app development k8s
       | can be really nice. We do data movement and ML services. AKS has
       | proven great for our use.
        
       | menacingly wrote:
       | Perhaps put more simply, operating in production has a lot of
       | intrinsic complexity that will probably surface in the tooling,
       | and if you constantly reinvent to "fix" the complexity you'll
       | eventually end up putting up it back.
       | 
       | That's how you end up with the modern javascript tooling
       | hellscape where it looks like no one was around to tell a bright
       | young developer "no"
        
       | jasoneckert wrote:
       | While I understand where the author is coming from, my opinion of
       | Kubernetes (and production deployment in general) isn't that it
       | is hard per se, but that it involves many components.
       | 
       | I liken it to Lego. Each component separately isn't hard to work
       | with, and once you figure out how to connect it to other
       | components, you can do it 100 times easily. And like Lego, a
       | typical Kubernetes environment may consist of several dozen or
       | hundred pieces.
       | 
       | So, I wouldn't describe Kubernetes as hard - I would describe it
       | as large (i.e., comprised of multiple interconnected components).
       | And by being large, there is a fair amount of time and effort
       | necessary to learn it and maintain it, which may make it seem
       | hard. But in the end, it's just Lego.
        
       | tptacek wrote:
       | 37signals is not like the typical large-scale startup. They have
       | an extremely small team (around 30 people?), and just a couple of
       | products.
       | 
       | Large-scale startups use dynamic-scheduled cloud services in part
       | to reduce coupling between teams. Every service --- and there are
       | dozens --- is scheduled independently, and new teams can get spun
       | up to roll out new services without too much intervention from
       | other teams.
       | 
       | When you've got a couple products that have been in maintenance
       | mode for 10+ years and then 2 just two primary products, both of
       | which are on the same stack, and you can predict your workloads
       | way out into the future (because you charge money for your
       | services, don't do viral go-to-markets, and don't have public
       | services), there simply isn't much of a win to dynamic
       | scheduling. You can, in fact, just have a yaml file somewhere
       | with all your hosts in it, and write some shell-grade tooling to
       | roll new versions of your apps out.
       | 
       | A lot of the reflexive pushback to not using k8s seemed like it
       | came from people that either didn't understand that 37signals was
       | doing something closer to static scheduling, or that don't
       | understand that most of what makes k8s complicated is dynamic
       | scheduling.
        
         | anonzzzies wrote:
         | > what makes k8s complicated is dynamic scheduling.
         | 
         | ... Which almost no startup or otherwise ever will need.
         | Creating complex stuff for workloads you will never ever have.
         | You hope to have them, but that's called premature
         | optimisation. And then you still mostly likely fall in the
         | bracket of a company that will never need it.
        
         | solatic wrote:
         | > Every service --- and there are dozens...
         | 
         | Most startups that I see trying to go with microservices too
         | early do so while keeping a shared database between the
         | services, so they're not really microservices, but a
         | distributed monolith. This turns into a massive, massive pain.
         | 
         | Doing microservices well means building out templates for CI/CD
         | pipelines, templates for observability, figuring out how best
         | to share (or not share) common credentials like third-party API
         | keys, setting up a service mesh, setting up services like
         | Backstage and Buf... which inevitably requires hiring dedicated
         | engineers at enormous cost.
         | 
         | If you can set up a monolith, why would you switch to
         | microservices? So that you can have a smaller container image
         | and faster autoscaling? So the developers on the second and
         | third teams you hire can wait a little less time for CI tests
         | to run?
         | 
         | It's a pretty big mistake to adopt microservices too early.
        
           | kdhfkdbjdh wrote:
           | or too late.
           | 
           | you just described all big corp attempts at giving aws money.
           | 
           | keep the database and all the bugs and leaky abstractions and
           | spaghetti code... but split and duplicate code in a dozen
           | repos... done. microservices.
        
         | majewsky wrote:
         | > Large-scale startups use dynamic-scheduled cloud services in
         | part to reduce coupling between teams.
         | 
         | This is the crux. It's Conway's Law in action. The promise of
         | Kubernetes to a large org is that you can split the baremetal
         | and OS layer into one team that manages everything up to
         | Kubernetes, and then that's the common interface where all
         | other teams deploy their applications into. And besides the
         | team separation, the value is that you have a central layer to
         | put automatic policy enforcement, engineering doctrines, and so
         | forth.
        
           | kazen44 wrote:
           | while i agree with you that kubernetes allows you to split
           | the stack down to individual "specialisations", i don't see
           | how this is different with kubernetes compared to
           | "traditional" Virtual machines?
           | 
           | Usually the differentiator is either the container or VM.
           | Everything beneath that gets managed by some kind of
           | operations team (or teams if you are really large or
           | specialized), while the stuff running ontop of the
           | infrastructure is usually done by developers/application
           | maintainers.
        
       | amir734jj wrote:
       | I would not use k8s unless we are convinced it will benefit us in
       | the long run (think about constant effort that needs to put in to
       | get things running). k8s is not magic. I would just stick with
       | docker-compose or digital ocean for small startup. OR rent a VM
       | on Azure OR if you really really need k8s use a managed k8s.
        
         | brody_hamer wrote:
         | Docker swarm is a great option too, for production
         | environments. It's like the production-ready big brother to
         | docker-compose (with better health checks and deployment
         | rollout options). And it has much less of a learning curve than
         | k8s.
        
       | scubbo wrote:
       | > [K8s] allows dev teams to not worry about all these things; all
       | they must do is to write a simple YAML file. More importantly,
       | teams no longer need to ask DevOps/infra folks to add DNS entry
       | and create a Load Balancer just to expose a service. They can do
       | it on their own, in a declarative manner, if you have an operator
       | do to it.
       | 
       | Yeah, as opposed to Cloudformation or Terraform, where
       | you...uhhh...
       | 
       | Don't get me wrong, it requires work to set up your corporate
       | infrastructure in your Favourite Cloud Provider(tm) to make those
       | things available for developers to manage. But it takes work in
       | k8s too - even the author says "if you have an operator to do
       | it". Kubernetes is great for what it's great for, but these are
       | terrible arguments in favour of it.
        
         | suralind wrote:
         | That's true, but I'd argue that TF is not as powerful as k8s.
         | You could combine it with auto scaling services offered by
         | cloud providers, but then you need to understand plugins and
         | with k8s it often is a single value. For example, you can "add"
         | Prometheus scraper just by adding labels. You won't have that
         | with TF.
        
       | thomasfromcdnjs wrote:
       | Most implementations are just slower, because they have not been
       | optimised what so ever, and optimising is not even that easy for
       | a senior.
        
       | tkiolp4 wrote:
       | I think the post title should be called "Production is hard" (as
       | the author talks about later on). Pick up any technology out
       | there: from Python, to C++, to K8s, to Linux... Do the analogous
       | "Hello world" using such technologies and run the program on your
       | laptop. Easy. You congratulate yourself and move on.
       | 
       | Production is another story. Suddenly your program that wasn't
       | checking for errors, breaks. The memory that you didn't manage
       | properly becomes now a problem. Your algorithm doesn't cut it
       | anymore. Etc.
        
         | morelisp wrote:
         | > Suddenly your program that wasn't checking for errors,
         | breaks. The memory that you didn't manage properly becomes now
         | a problem.
         | 
         | Yeah, nobody deployed anything and ran it for months, even
         | years, before Kubernetes.
        
           | 988747 wrote:
           | Yes, they did, but did it really take less effort than
           | running it on Kubernetes?
        
             | deathanatos wrote:
             | People _think_ it took less effort.
             | 
             | Right up until you needed to do one of the very many things
             | k8s implements.
             | 
             | For example, in multiple previous employers, we had
             | cronjobs: you just set up a cronjob on the server, I mean,
             | really, how hard is that to do?
             | 
             | And that server was a single point of failure: we can't
             | just spin up a second server running crond, obviously, as
             | then the job runs twice. Something would need to provide
             | some sort of locking, then the job would need to take
             | advantage of that, we'd need the job to be idempotent ...
             | all of which, except the last, k8s does out of the box.
             | (And it mostly forces your hand on the last.)
             | 
             | Need to reboot for security patches? _We just didn 't do
             | that,_ unless it was something like Heartbleed where it was
             | like "okay we have to". k8s permits me to evict workloads
             | while obeying PDB -- in previous orgs, "PDBs" (hell, we
             | didn't even _have a word to describe the concept_ ) were
             | just tribal knowledge known only by those of us who SRE'd
             | enough stuff to know how each service worked, and what you
             | needed to know to stop/restart it, and then do that times
             | waaay too many VMs. With k8s, a daemonset can just handle
             | things generically, and automatically.
             | 
             | Need to deploy? Pre-k8s, that was just bespoke scripts,
             | e.g., in something like Ansible. If a replica failed to
             | start after deployment, did the script cease deployment?
             | Not the first time it brought everything down, it didn't:
             | it had to grow that feature by learning the hard way.
             | (Although I suppose you can decide that you don't _need_
             | that readiness check in k8s, but it 's at least a hell of a
             | lot easier to get off the ground with.)
             | 
             | Need a _new VM_? What are the chances that the current one
             | actually matches the Ansible, and wasn 't snowflaked? (All
             | it takes is _one_ dev, and _one_ point in time, doing _one_
             | custom command!)
             | 
             | The list of operational things that k8s supports that are
             | common amongst "I need to serve this, in production" things
             | goes on.
             | 
             | The worse part of k8s thus far has been Azure's half-aaS'd
             | version of it. I've been pretty satisfied with GKE, but
             | I've only recently gotten to know it and I've not pushed it
             | quite as hard as AKS yet. So we'll see.
        
             | slyall wrote:
             | No it didn't. You ended up with each site doing things
             | differently. You'd go somewhere and they would have a
             | magical program with a cute name written by a founder that
             | distributed traffic, scheduled jobs and did autoscaling. It
             | would have weird quirks and nobody understood it.
             | 
             | Or you wouldn't have it at all. You'd have a nice simple
             | infra and no autoscaling and deploys would be hard and
             | involve manually copying files.
        
             | ranger207 wrote:
             | Yes, until you've scaled enough that it wasn't. If you're
             | deploying a dev or staging server or even prod for your
             | first few thousand users then you can get by with a handful
             | of servers and stuff. But a lot of stuff that works well on
             | one or three servers starts working less well on a dozen
             | servers, and it's around that point that the up-front
             | difficulty of k8s starts to pay off with the lower long-
             | term difficulty
        
             | lowercased wrote:
             | Yes.
        
             | morelisp wrote:
             | Yes.
        
             | nosequel wrote:
             | Yes. Much.
        
             | jamwt wrote:
             | Yes, way less effort
        
             | nailer wrote:
             | Yes.
        
             | athoscouto wrote:
             | I'm going to diverge from sibling comments: it depends. As
             | the article points out, k8s may really simplify deploys for
             | devs, while giving the autonomy. But it isn't always worth
             | it.
        
             | majormajor wrote:
             | Definitely if they weren't running very complex services or
             | their business tolerated an occasional maintenance window.
             | 
             | The particular way Kubernetes can bite you is that it makes
             | it much easy to _start_ with far more complex setups - not
             | necessarily much harder than to start with a simple setup!
             | - but then you have to maintain and operate those complex
             | setups forever, even if you don 't need them that much.
             | 
             | If you're growing your team and your services, having a
             | much bigger, more complicated toolbox for people to reach
             | into on day 1 gives you way more chance of building
             | expensive painful tech debt.
        
               | suralind wrote:
               | I think it may appear so because Kubernetes promotes good
               | practices. Do logging, do metrics, do traces. That list
               | quickly grows and while these are good practices, there's
               | a real cost to implement them. But I wouldn't agree that
               | Kubernetes means building tech debt - on the contrary, if
               | you see the tech debt, k8s makes it easier to get rid of
               | it, but that of course takes time and if you don't do it
               | regularly that tech debt is only gonna grow.
        
               | majormajor wrote:
               | I just rarely see people tackling greenfield problems
               | with the discipline to choose to do Kubernetes without
               | also choosing to do "distributed" in a broader,
               | complexity-multiplying way.
               | 
               | If not for Kubernetes (and particularly Kube + cloud
               | offerings) I really doubt they'd do all the setup
               | necessary to get a bunch of distributed systems/services
               | running with such horizontal sprawl.
        
             | winrid wrote:
             | Yeah... at small to medium scale anyway.
        
         | bdougherty wrote:
         | Production is only as hard as you make it.
        
           | madeofpalk wrote:
           | Production is as easy as you (and your users) will tolerate.
        
         | papruapap wrote:
         | Well then you could the resume the whole article in the title
         | because there isnt anything else in it.
        
         | znpy wrote:
         | Quoting Bryan Cantrill: production is war.
        
           | wingmanjd wrote:
           | I'm a simple man...I see a Bryan Cantrill quote and I upvote.
        
           | srejk wrote:
           | And war is hell.
        
       ___________________________________________________________________
       (page generated 2023-03-27 23:00 UTC)