[HN Gopher] How I think about Kubernetes
___________________________________________________________________
How I think about Kubernetes
Author : todsacerdoti
Score : 47 points
Date : 2025-12-26 20:44 UTC (2 hours ago)
(HTM) web link (garnaudov.com)
(TXT) w3m dump (garnaudov.com)
| NewJazz wrote:
| Love the HN title mod here lol
| stavros wrote:
| HN removes a "how" if the post starts with it, sometimes making
| it hilarious.
| dkdcio wrote:
| I've seen this a few times now, what's the context/background
| on why this is done?
| NewJazz wrote:
| I think maybe they check for "How to [...]" and other
| variations? "How I broke TLS 1.3" -> "I broke TLS 1.3"
| dkdcio wrote:
| yeah but why?
| zem wrote:
| yeah, really wish they would fix that one!
| frisovv wrote:
| Tbh the missing how is probably why I followed the link. And
| I appreciated the post, so net positive outcome here.
| DonHopkins wrote:
| Otherwise you might confuse it with the HN mod tomhow.
|
| Same reason they remove "dang" if the post starts with it,
| like the discussion about "Dang! Who ate the middle out of
| the daddy longlegs".
|
| https://ifunny.co/picture/dang-who-ate-the-middle-out-of-
| the...
| websiteapi wrote:
| I always wonder if things can be simpler. When you think of a
| really simple DB you think of SQLite. What's the really simple
| K8s? Even doing a single node deployment these days seems
| complicate with Prometheus, Grafana, etc. etc. docker/podman
| compose up with quadlets and all of this stuff just seems so eh.
|
| I really like the idea of something like Firebase, but it never
| seems to work out or just move the complexity to the vendor,
| which is fine, but I like knowing I can roll my own.
| eyeris wrote:
| Big question is which feature subset you want to replicate.
|
| Kubernetes means everything to everyone. At its core, I think
| it's being able to read/write distributed state (which doesn't
| need to be etcd) and being able for all the components
| (especially container hosts) to follow said state. But the
| ecosystem has expanded significantly beyond that.
| jauntywundrkind wrote:
| IMO this is what keeps people from building systems that might
| challenge kubernetes. Everyone wants to say Kuberentes is too
| complex, so we built something that does much less. I respect
| that! But I think it usually fails to grok what Kubernetes is
| and why it's such an interesting and vital rallying point,
| that's so thoroughly captured our systems-making. Let's look at
| the premise:
|
| > _That's why I like to think of Kubernetes as a runtime for
| declarative infrastructure with a type system._
|
| You can go build a simple way to deploy containers or ship
| apps: but you are missing what I think allows Kubernetes to be
| such a big tent, thats a core useful platform for so many.
| Kubernetes works the same _for all types_ , for everything you
| want to manage. It's the same desired state management +
| autonomic systems patterns, whatever you are doing. An
| extensible platform with a very simple common core.
|
| There are other takes and other tries, but managing _desired_
| state for _any_ kind of type is a huge win that allows many
| people to find their own uses for kube, that is absolutely the
| cornerstone to it 's popularity.
|
| If you do want less, the one project I'd point to that is
| kubernetes without the kubernetes complexity is KCP. It's just
| the control plane. It doesn't do anything at all. This to me is
| much simpler. It's not finding a narrowly defined use case to
| focus on, it's distilling out the general system into it's
| simplest parts. Rebuilding a good simple bespoke app container
| launching platform around KCP would be doable, and maintain the
| overarching principles that make Kube actually interesting.
|
| I seriously think there is something deeply rotten with our
| striving for simplicity. I know we've all been burned, and
| there's so often we want to throw up our hands, and I get it.
| But the way out is through. I'd rather dance the dance & try to
| scout for better further futures, than reject & try to walk
| back.
| zsoltkacsandi wrote:
| Everything in infrastructure is a set of trade-offs that work
| in both directions.
|
| If you want better monitoring, metrics, availability,
| orchestration, logging, and so on, you pay for it with time,
| money, and complexity.
|
| If you can't justify that cost, you're free to use simpler
| tools.
|
| Just because everyone sets up a Kubernetes / Prometheus / ELK
| stack to host a web app that would happily run on a single VPS
| doesn't mean you need to do the same, or that nowadays this is
| the baseline for running something.
| vbezhenar wrote:
| Of course things can be simpler.
|
| Remove abstractions like CNI, CRI, just make these things
| built-in.
|
| Remove unnecessary things like Ingress, etc, you can always
| just deploy nginx or whatever reverse proxy directly. Also
| probably remove persistent volumes, they add a lot of
| complexity.
|
| Use some automatically working database, not separate etcd
| installation.
|
| Get rid of control plane. Every node should be both control
| plane and worker node. Or may be 3 worker nodes should be
| control plane, whatever, deployer should not think about it.
|
| Add stuff that everyone needs. Centralised log storage,
| centralised metric scrapping and storage, some simple web UI,
| central authentication. It's reimplemented in every Kubernetes
| cluster.
|
| The problem is that it won't be serious enough and people will
| choose Kubernetes over simpler solutions.
| NewJazz wrote:
| Some people want their k8s logs to be centralized with non
| k8s logs. Standardizing log storage seems like a challenging
| problem. Perhaps they could add built in _log shipping_. But
| even then, the transfer format needs to be specified.
|
| Adding an idp is pretty standard in k8s... What do you want
| to actually do different?
| bigstrat2003 wrote:
| > What's the really simple K8s?
|
| It's k3s. You drop a single binary onto the node, run it, and
| you have a fully functional one-node k8s cluster.
| whytevuhuni wrote:
| > What's the really simple K8s?
|
| I think K8s couples two concepts: the declarative-style cluster
| management, and infrastructure + container orchestration. Keep
| CRDs, remove everything else, and implement the business-
| specific stuff on top of the CRD-only layer.
|
| This would give something like DBus, except cluster-wide, with
| declarative features. Then, container orchestration would be an
| application you install on top of that.
|
| Edit: I see a sibling mentioned KCP. I've never heard of it
| before, but I think that's probably exactly what I'd like.
| KronisLV wrote:
| In ascending order of functionality and how much complexity you
| _need_ : - Docker Compose running on a single
| server - Docker Swarm cluster (typically multiple nodes,
| can be one) - Hashicorp Nomad or K3s or other light
| Kubernetes distros
| zsoltkacsandi wrote:
| > Thinking of Kubernetes as a runtime for declarative
| infrastructure instead of a mere orchestrator results in very
| practical approaches to operate your cluster.
|
| Unpopular opinion, but the source of most of the problems I've
| seen with infrastructures using Kubernetes came from exactly this
| kind of approach.
|
| Problems usually come when we use tools to solve things that they
| weren't made for. That is why - in my opinion - it is super
| important to treat a container orchestrator a container
| orchestrator.
| szundi wrote:
| It would have helped if you tell us why you don't like this
| approach.
| zsoltkacsandi wrote:
| It's right there:
|
| > the source of most of the problems I've seen with
| infrastructures using Kubernetes came from exactly this kind
| of approach
|
| But some more concrete stories:
|
| Once, while I was on call, I got paged because a Kubernetes
| node was running out of disk space. The root cause was the
| logging pipeline. Normally, debugging a "no space left on
| device" issue in a logging pipeline is fairly
| straightforward, if the tools are used as intended. This
| time, they weren't.
|
| The entire pipeline was managed by a custom-built logging
| operator, designed to let teams describe logging pipelines
| declaratively. The problem? The resource definitions alone
| were around 20,000 lines of YAML. In the middle of the night,
| I had to reverse-engineer how the operator translated that
| declarative configuration into an actual pipeline. It took
| three days and multiple SREs to fully understand and fix the
| issue. Without such a declarative magic it takes usually 1
| hour to solve such an issue.
|
| Another example: external-dns. It's commonly used to manage
| DNS declaratively in Kubernetes. We had multiple clusters
| using Route 53 in the same AWS account. Route 53 has a global
| API request limit per account. When two or more clusters
| tried to reconcile DNS records at the same time, one would
| hit the quota. The others would partially fail, drift out of
| sync, and trigger retries - creating one of the messiest
| cross-cluster race conditions I've ever dealt with.
|
| And I have plenty more stories like these.
| NewJazz wrote:
| I feel like the author has a good grasp of the Kubernetes
| design... What about the approach is problematic? And why don't
| you think that is how Kubernetes was designed to be used?
| zsoltkacsandi wrote:
| I wrote some personal stories below in this thread as a
| response to another user.
| k8ssskhltl wrote:
| But then you need two different provisioning tools, one for
| infra in k8s, and one for infra outside k8s. Or perhaps using
| non-native tools or wrappers.
| zsoltkacsandi wrote:
| > But then you need two different provisioning tools, one for
| infra in k8s, and one for infra outside k8s.
|
| Yes, and 99% of the companies do this. It is quite common to
| use Terraform/AWS CDK/Pulumi/etc to provision the
| infrastructure, and ArgoCD/Helm/etc to manage the resources
| on Kubernetes. There is nothing wrong with it.
| btown wrote:
| One approach if "dang it, someone/I needed to use kubectl during
| the outage, how do we get gitops/poor-mans-gitops back in place
| to match reality" is, either agentically-looping or artisanally-
| looping, to try simple gitops configurations (or diffs to current
| gitops configurations) until a dry-run diff with your live
| configuration results in no changes.
|
| For instance, with Helm, I've had success using Helmfile's diffs
| (which in turn use https://github.com/databus23/helm-diff) to do
| this.
|
| There's more of a spectrum between these than you think, in a way
| that can be agile for small teams without dedicated investment in
| gitops. Even with the messes that can occur, I'd take it over the
| Heroku CLI any day.
| blackjack_ wrote:
| Yes, there is a term for a system that handles a declarative
| state of infrastructure and does reconciliation versus current
| state; a control plane. We have been talking about control planes
| in devops/ SRE for a number of years now! Welcome to the
| conversation.
| anymouse123456 wrote:
| The allure of declarative approaches to complex problem solving
| has finally been worn down to nothing for me and Kubernetes was
| the last straw, nearly 10 years ago.
|
| The mental gymnastics required to express oneself in yaml, rather
| than, say, literally anything else, invariably generates a horror
| show of extremely verbose boilerplate, duplication, bloat, delays
| and pain.
|
| If you're not Google, please for the love of god, please consider
| just launching a monolith and database on a Linux box (or two) in
| the corner and see how beautifully simple life can be.
|
| They'll hum along quietly serving many thousands of actual
| customers and likely cost less to purchase than a single month
| (or at worst, quarter) of today's cloud-based muggings.
|
| When you pay, you'll pay for bandwidth and that's real value that
| also happens to make your work environment more efficient.
| themgt wrote:
| _If you 're not Google, please for the love of god, please
| consider just launching a monolith and database on a Linux box
| (or two) in the corner and see how beautifully simple life can
| be._
|
| You can literally get a Linux box (or two) in the corner and
| run: curl -sfL https://get.k3s.io | sh -
| cat <<EOF | kubectl apply -f - ...(json/yaml here)
| EOF
|
| How am I installing a monolith and a database on this Linux box
| without Kubernetes? Be specific, just show the commands for me
| to run. Kubernetes that will work for ~anything. HNers spend
| more tokens complaining about the complexity than it takes to
| setup.
|
| _The mental gymnastics required to express oneself in yaml,
| rather than, say, literally anything else_
|
| Like, brainfuck? Like bash? Like Terraform HCL puppet chef
| ansible pile-o-scripts? The effort required to output your
| desired infrastructure's definition as JSON shouldn't really be
| _that_ gargantuan. You express yourself in anything else but it
| can 't be dumped to JSON?
| paddw wrote:
| > Thinking of Kubernetes as a runtime for declarative
| infrastructure instead of a mere orchestrator results in very
| practical approaches to operate your cluster.
|
| This is a pretty good definition.
|
| I think part of the challenge is the evolution of K8s over time
| sometimes makes it feel less like a coherent runtime and more
| like a pile of glue amalgamated from several different components
| all stuck together. That and you will have to be aware of how
| those abstractions stick together with the abstractions from your
| cloud provider, etc...
| tbrownaw wrote:
| It's an application server for multi-part containerized
| applications, like Tomcat is an application server for
| applications that can be turned into .war files.
___________________________________________________________________
(page generated 2025-12-26 23:00 UTC)