[HN Gopher] A skeptic's first contact with Kubernetes
___________________________________________________________________
A skeptic's first contact with Kubernetes
Author : todsacerdoti
Score : 103 points
Date : 2024-07-28 14:00 UTC (4 days ago)
(HTM) web link (blog.davidv.dev)
(TXT) w3m dump (blog.davidv.dev)
| jauntywundrkind wrote:
| > _Why are the storage and networking implementations "out of
| tree" (CNI / CSI)? Given the above question, why is there
| explicit support for Cloud providers? eg: LoadBalancer supports
| AWS/GCP/Azure/.._
|
| Kubernetes has been pruning out vendor-specific code for a while
| now, moving it out of tree. The upcoming 1.31 release will drop a
| lot of existing, already deprecated support for AWS & others from
| Kubernetes proper.
| https://github.com/kubernetes/enhancements/blob/master/keps/...
|
| There's some plan to make this non-dosruptive to users but I
| haven followed it closely (I don't use these providers anyhow).
|
| > _Why are we generating a structured language (YAML), with a
| computer, by manually adding spaces to make the syntax valid?
| There should be no intermediate text-template representation like
| this one._
|
| Helm is indeed a wild world. It's also worth noting that
| Kubernetes is also pushing towards neutrality here; Helm has
| never been an official tool, but Kustomzie is builtin to kubectl
| & is being removed.
| https://github.com/orgs/kubernetes/projects/183/views/1?filt...
|
| There's a variety of smart awesome options out there. First place
| I worked at that went to kube used jsonnet (which alas went
| unmaintained). Folks love CUE and Dhall and others. But to my
| knowledge there's no massive bases of packaged software like
| exists for Helm. Two examples,
| https://github.com/bitnami/charts/tree/main/bitnami
| https://github.com/onedr0p/home-ops . It'd be lovely to see more
| works outside Helm.
|
| Thanks sysdig for your 1.31 write up,
| https://sysdig.com/blog/whats-new-kubernetes-1-31/
| deivid wrote:
| > Kubernetes has been pruning out vendor-specific code for a
| while now, moving it out of tree.
|
| I wasn't aware, but it makes sense, and explains the current
| "confusing" state.
| jauntywundrkind wrote:
| Thanks for the write-up, and especially for storytelling your
| ability & willingness to wade in & find out! Proper hacker
| spirit!
|
| You did such a great job offering an informative high level
| view (honing in on the control loop feels like seizing upon a
| critical insight), & iterating through bits nicely. You were
| tactful & clear in raising some ongoing qualms, which indeed
| seem largely correct & frustrating, even to those who started
| as believers. Thanks thanks, a recommendable article indeed.
| bigstrat2003 wrote:
| Off topic - thank you for not having a super narrow text
| layout for your site. It seems like every other website these
| days has an incredibly narrow text width (I've seen as small
| as 600px which is so annoying to read). It was like a breath
| of fresh air to go to your site and have the text be a
| reasonable width without me having to fiddle with page
| styles.
| mtndew4brkfst wrote:
| _Kustomize is builtin to kubectl & is being removed_
|
| Hadn't heard this until now, I'm a rather happy user after
| being jaded about Helm 2/3. Do you happen to know if this
| removal is sentimentally closer to "we don't want to keep
| maintaining kustomize" or to "we don't want to keep embedding
| kustomize, please use its binary directly"?
| JojoFatsani wrote:
| The helm yaml thing really is annoying. Unfortunately it feels
| like helm is too firmly embedded to unseat at this point.
| IceWreck wrote:
| I've switched to kustomize for my homelab. Its built into
| kubectl (no extra tool needed) and its not really templating.
|
| It doesn't fully replace helm for 3rd party tools or packaging
| but its a solid alternative for less complicated setups.
| worldsayshi wrote:
| Funny then that kustomize apparently is being removed,
| according to another comment:
|
| https://news.ycombinator.com/item?id=41093797
| dv_dt wrote:
| My annoyance with helm is that it's possible for a helm install
| to fail, with no top level install status, yet have failed
| chart resources where you need to go digging around and
| manually delete before another install can be applied. I mean
| helm has one key job..
| azaras wrote:
| I agree with:
|
| > My opinion is that a large part of Kubernetes' value is derived
| from just two concepts
|
| I agree with the first one, "control loops," but the second one
| is "API-based," not "Services."
| hbogert wrote:
| His take on text interpolation is very right. I'm a SWE turned
| SRE because as a developer I really enjoyed using K8s. But as a
| full-time SRE where I work just means YAML juggling. It's mind
| numbing that everybody is okay with this, this really is our
| domain's assembly era, albeit with whitespace, colons, dashes and
| brackets.
|
| I've found solace in CUE which I just run locally to catch all
| the small errors everybody makes on a daily basis. Putting the
| CUE validation in our pipeline is too confronting for others, yet
| they're constantly making up best practices adhoc during reviews
| which could've easily been codified with CUE (or some other
| serious config language).
| galdosdi wrote:
| > Putting the CUE validation in our pipeline is too confronting
| for others
|
| Sad. Can you get away with boiling the frog by getting it
| initially added but configured to check very, very little?
| Maybe some specific rare category of mistake that doesn't
| happen enough to annoy your colleagues when it catches it but
| is implicated in a recent outage management still remembers?
|
| Then slowly adding rules over time as feasible (exploiting each
| outage as an opportunity to enlist stakeholder support for
| adding more rules that would have caught that particular
| misconfiguration event)
|
| Sometimes I think figuring out how to stage and phase gradually
| making the changes you need to improve the system in light of
| social inertia against it is the most complex part of corporate
| enterprise software work. I definitely remember a time I wrote
| a whole perl-based wheel-reinventing crappy puppet knockoff
| just for a set of nagios plugins, entirely not for technical
| reasons, but for the political reason that this way we would
| control it instead of the other department which refused to get
| with the program and let us do staged canary rollouts. It was
| wrong technically if you assume frictionless corporate
| politics, but it was the right and only practical way to
| achieve the goal of ending the drumbeat of customer outages.
| yourapostasy wrote:
| _> ...just means YAML juggling._
|
| Perhaps I'm missing something, but even the "YAML juggling"
| still seems to be in an immature stage. As an example, a
| problem I've been wrestling with recently where I could not
| find a solution with yq or other YAML wrangling tooling is the
| following.
|
| Say I have a YAML fragment like this:
|
| rules: - apiGroups: - rbac.authorization.k8s.io resources:
| -clusterroles verbs: - get - apiGroups: -
| rbac.authorization.k8s.io resources: -clusterroles verbs: -
| list
|
| I want to condense it to this: rules: - apiGroups: -
| rbac.authorization.k8s.io resources: -clusterroles verbs: - get
| - list
|
| More generally, I want to find the smallest most compact
| representation possible between the apiGroups, resources and
| verbs terms, combining everywhere possible. In the example, I
| combined the verbs, but the same combining can take place for
| the apiGroups and resources terms. This comes in really handy
| when cleaning up a litter trail of these constructs built from
| figuring out what resources to add to enable more complex Helm
| Chart installs (the iterative nature of that exercise is
| another rant in itself) in reasonably restrictive environments.
|
| I resorted to writing a script to solve just this problem, but
| I keep thinking I must have missed some solution to this
| everyone else seems to have solved but me. Apparently the
| solution eludes my local K8S experts as well, because they
| condense it by hand, which I refuse to accept as the proper
| solution.
| doctorpangloss wrote:
| > which I just run locally to catch all the small errors
| everybody makes on a daily basis
|
| What small errors?
| hosh wrote:
| The better way is to use manifest generators with a Turing-
| complete language. I wrote such a tool in Ruby.
|
| The problem is that it is written in Ruby. It's great for shops
| that already have Ruby expertise. Wherever I have implemented
| this, it's worked out great. I have my doubts about it in other
| shops.
|
| The community went a different way -- starting with templated
| JSON (not even JSON transforms). It was the least common
| denominator.
|
| This is an issue with the ecosystem rather than design flaws of
| Kubernetes itself.
| candiddevmike wrote:
| Ksonnet being deprecated was one of the worst things to happen
| with the ecosystem IMO. Some kind of Jsonnet tool should have
| been integrated with kubectl like kustomize.
|
| Shameless plug for Etcha, a configuration management tool (that
| works with kube!) built around Jsonnet manifests:
| https://etcha.dev
| FridgeSeal wrote:
| > I've found solace in CUE
|
| We rebuilt our Kustomization layers to be built via cue, so
| that we could theoretically work with less mistakes. Oh my god,
| never again.
|
| Incredibly confusing and horrible errors. Writing new modules
| was painful and unclear, trying to compose them together was
| even worse. I still get flashbacks to that "Invalid bytes"
| error.
| zbentley wrote:
| I think it's worth re-examining the widespread opposition to
| writing configs in an interpreted "real" programming
| language, rather than a string templating language (if you're
| looking to split hairs, in this taxonomy PHP and JSX are real
| languages, but they're close to the line. Python is
| prototypically a real language. Awk, Jinja, CUE, and Bash/ZSH
| string expansion are not. If the word "real" bothers you,
| pick another one).
|
| Like, people are correct that it is not great to allow
| (restricted at runtime or not) arbitrary code to define
| configs. I think "our config language is Python"-type
| statements are concerning in a whole lot of contexts, and for
| good reasons.
|
| But holy shit at what cost. The landscape formed by the
| extremity of that no-real-scripting-languages-allowed
| sentiment is _bleak_. We 've all dealt with it: the
| proliferation of handfuls of arcane tools that have to be
| present at specific versions in order to _render a string_.
| The triple-templated hellscapes written in Helm
| /Jinja/CUE/Jsonnet/DTL/ytt/JSON-patch/Dhall. Quoting hell.
| Indentation hell. Brace hell. The endless, endless bikesheds
| over which data language to render to (which are never really
| resolved and always just end up with JSON-interpolated-in-
| YAML-interpolated-in-TOML and transformed to the "right"
| format at render time). The flavor-of-the-month XKCD#927
| creation of each iteration of a data+templating language that
| will solve all of the problems, but for real this time.
|
| This really sucks. Not because its tedious; tedium is not
| necessarily a bad thing, with apologies to the automate-all-
| the-things crowd. But because it's _expensive_ : in time,
| training, bugs, incident MTTR, and so on.
|
| I dunno. Maybe we should accept that past some complexity
| threshold, you should just write your configs in a general
| purpose scripting language. Not because it's a _good_
| practice, but because our best attempts at an alternative are
| so much worse that it 's probably the _best_ practice to just
| do it in fucking Python. Starting from there and adding
| restrictions (i.e. Starlark) can 't possibly be worse than
| the currently blessed way.
| andrewaylett wrote:
| I don't object to the desired state being represented in
| YAML, I object to generating that YAML using something
| that's not a "real" programming language.
|
| I don't particularly care if you generate your data
| structures from schemas, or generate your schemas from data
| structures, but I do care if you don't generate your output
| YAML using anything that's not type-safe.
|
| If you're generating YAML to deploy, you can do that during
| build. If you're needing to combine configuration to
| generate new YAML at runtime, that's what controllers are
| for.
| andix wrote:
| I don't think that the k8s yamls/jsons are bad. It's just bad
| that we write and edit them as text files
|
| My take on the k8s yaml mess is, that we are lacking a modern
| replacement for Helm. The yaml templating on text file level is
| just pure crazyness. I think we would need an approach that is
| more like React/JSX or even better TSX (typescript JSX). Some
| possibility to get away from simple templating and more into
| structured descriptions, with typing and control flows.
|
| I think there are some approaches into this direction (like the
| k8s Terraform module, or the defunct ksonnet), but none of them
| got it perfectly right yet.
| mcorbin wrote:
| Nice article ;)
|
| > The number of rules may be large, which may be problematic for
| certain traffic forwarding implementations (iptables linearly
| evaluates every rule)
|
| Kube proxy also supports ipvs out of the box, and some CNI (like
| Cilium) can also replace kube proxy and rely on eBPF.
|
| > When a Pod is moved to another node, the traffic will be
| forwarded twice until the old DNS entry expires Not sure to
| understand this one. On a standard setup what happens is:
|
| - Pod A is running on a node, receiving traffic from a service
|
| - The pod is stopped by kubelet (that send a SIGTERM to it).
|
| - The pod should gracefully shutdown. During the shutdown phase,
| only _existing_ connections are forwarded to the stopping pod,
| new ones will be already forwarded elsewhere.
|
| - If the pod stops before the terminationGracePeriodSeconds
| duration (default 30s), everything is fine. Else, the pod is
| killed by kubelet. So it's developers that should make sure pods
| handle signals correctly.
|
| "Misbehaving clients (eg: ones that do not re-resolve DNS before
| reconnecting) will continue to work" => the services IP is stable
| so clients don't need to re-resolve.
|
| > Why does it not matter if the state is unstable? If I'm
| operating a cluster that can't settle, I'd like to know
| immediately!
|
| Kubernetes exposes a lot of metrics, on the control plane
| components or kubelet, usually using the Prometheus format. Look
| for example at the metrics exposed by kube state metrics:
| https://github.com/kubernetes/kube-state-metrics/tree/main/d...
|
| With controllers metrics + kube state metrics about most
| Kubernetes resources, you can easily build alerts when a resource
| fails to reconcile.
|
| > Basically, Horizontal Pod Autoscaler but with sensors which are
| not just "CPU"
|
| Take a look at KEDA, it's exactly this: https://keda.sh/ It
| "extends" the autoscaler capabilities. If you're running
| Prometheus you can for example scale on any metric that is stored
| in Prometheus (and so exposed by your application/infrastructure
| components: queue depth, latency, request rate...).
|
| Kubernetes was built to be extended like this. Same for your
| question "Why are the storage and networking implementations "out
| of tree" (CNI / CSI)?", to my experience support is very good
| today on various cloud providers or on premise infra components.
| Look at Karpenter for example, it's IMO a revolution in the
| Kubernetes node management world.
| politelemon wrote:
| This was a useful read and somewhat gels with my experiences.
|
| Looking at the statement at the beginning
|
| > and it only requires you to package your workload as a Docker
| image, which seems like a reasonable price to pay.
|
| is no longer true as you continue down the path, since it
| _actually_ requires you to do a lot more than you 'd think.
| deivid wrote:
| I've not yet gone down that path, in which other ways does your
| workload need to adapt? I understand that you may want to get
| more value out of logs, and include side-cars, you may want
| inter-pod encryption and include service meshes, etc; but
| that'd be something that requires extra complexity in any setup
| movedx wrote:
| OIDC integration, RBAC, data persistence/replication across
| nodes, observability, mTLS.
|
| And yes, you're right, all these things are complex in any
| situation, except when you simply use a load balancer and two
| servers. There are companies estimated to be worth close to
| US$2B using with sub 30 servers using a service you may have
| heard of: Stack Overflow
| (https://nickcraver.com/blog/2016/03/29/stack-overflow-the-
| ha... && https://nickcraver.com/blog/2016/05/03/stack-
| overflow-how-we...).
|
| (As an aside: K8s does not free you from your Cloud vendor -
| after floating around my city working on various "platforms",
| all locked in due to IAM, for example.)
| INTPenis wrote:
| Reminds me of myself, and probably many others on here. I've
| always been a skeptic of novelty. Just like with cryptocoins I
| was there on the ground floor, I remember when Docker was
| launching. I remember trying to understand what it was, and I
| remember someone comparing it to MSI for Linux.
|
| Unlike cryptocoins I didn't miss out on being a millionaire with
| containers. :(
|
| I just avoided containers until 2019! So to me it was first
| containers, and then kubernetes.
|
| That way I was already sold on the whole container image concept
| and namespaces in your OS, I had used FreeBSD jails in the early
| 2000s.
|
| So when I understood k8s I realized it's literally just a
| container orchestrator. It might seem complicated but that's all
| to do with being able to run containers on a range of nodes
| instead of just one. And of course having an API to tie it all
| together.
|
| Whether you project needs that or not is something you should
| definitely explore in depth before you set out and try to use it.
| Personally I prefer container hosts over k8s for most startup
| projects. I look forward to Talos' new podman package and being
| able to deploy a minimal podman container host with Talos, no k8s
| necessary.
| nrr wrote:
| "Whether you project needs that or not is something you should
| definitely explore in depth before you set out and try to use
| it." Unfortunately, as someone who's written these kinds of
| orchestrators a few times already (going back to jails on
| FreeBSD 4.8 and Xen domUs on NetBSD 2.0) and has a nose for
| where they're a good idea, I find this is overwhelmingly not
| the prevailing wisdom. I desperately wish it were.
|
| Somewhere, the industry went all-in on Kubernetes, and I have
| zero desire to touch most of the resulting software stacks. We
| sold folks hard on complexity being necessary, and Kubernetes
| makes it very tempting to go ham with it.
| cyberax wrote:
| My problem with K8s: the network abstraction layer just feels
| _wrong_.
|
| It's an attempt to replicate the old model of "hard exterior,
| gooey interior" model of corporate networks.
|
| I would very much prefer if K8s used public routable IPv6 for
| traffic delivery, and then simply provided an authenticated
| overlay on top of it.
| kasey_junk wrote:
| Something like? https://john-millikin.com/stateless-kubernetes-
| overlay-netwo...
| hitpointdrew wrote:
| > My problem with K8s: the network abstraction layer just feels
| _wrong_.
|
| > I would very much prefer if K8s used public routable IPv6 for
| traffic delivery
|
| shudder... nothing could feel more wrong to me than public
| routable IPv6, yuck.
| icedchai wrote:
| I've felt similarly. Possibly because I was online pretty
| early, pre-NAT... there was public IPv4 everywhere.
| Atreiden wrote:
| Great writeup on the core fundamentals, saved this to share with
| engineers who are new to k8s and need a quick primer.
|
| Re: This piece -
|
| > Given the Controller pattern, why isn't there support for
| "Cloud Native" architectures?
|
| > I would like to have a ReplicaSet which scales the replicas
| based on some simple calculation for queue depth (eg: queue depth
| / 16 = # replicas)
|
| > Defining interfaces for these types of events (queue depth,
| open connections, response latency) would be great
|
| > Basically, Horizontal Pod Autoscaler but with sensors which are
| not just "CPU"
|
| HPAs are actually still what you want here - you can configure
| HPAs to scale automatically based on custom metrics. If you run
| Prometheus (or a similar collector), you can define the metric
| you want (e.g. queue-depth) and the autoscaler will make scaling
| decisions with these in mind.
|
| Resources:
|
| https://kubernetes.io/docs/tasks/run-application/horizontal-...
|
| https://learnk8s.io/autoscaling-apps-kubernetes
| entropyie wrote:
| See KEDA and Karpenter for advanced k8s acaling
| zikduruqe wrote:
| ^ this right here. We used KEDA to query DynamoDB to look at
| a queue depth we wrote to a table. If number was X, then we
| would scale on it. Was pretty slick.
| deivid wrote:
| Using custom/external metrics is exactly what I was looking
| for, thanks!
|
| I think my misunderstanding comes from the distinction between
| "first party" sensors (CPU scaler) and "third party" sensors
| (via "external metrics").
|
| Is there a reason for this distinction? Will the CPU scaler
| eventually be pushed out of k8s as well?
| richardwhiuk wrote:
| Kubelet is already managing the CPU resource I guess from
| scheduling?
|
| I don't think there's any real movement on moving CPU scaler
| out of HPA.
| hosh wrote:
| This is a good start. It misses an important thing about
| Kubernetes that is often missed: extensibility. Each and every
| thing within Kubernetes can be swapped out for something else,
| including the scheduler. There are Custom Resource Definitions
| that supports these extensions and operators.
|
| For example, there is no built-in autoscaler for nodes, but
| someone wrote one and you can add one in there. It uses a
| constraint solver for determining whether to expand or shrink
| node groups. If you want to use something else, you can find
| something or write it and install it.
|
| Another example, you don't have to use kube-proxy. There are
| other ways to manage inter-node networking.
|
| To address some of the questions:
|
| > Why does it not matter if the state is unstable? If I'm
| operating a cluster that can't settle, I'd like to know
| immediately!
|
| I'd point to the Cynefine framework to help make sense of this.
| Kubernetes is a tool that helps manage things in the _complex_
| domain, rather than the _complicated_ domain. Unlike complicated
| systems, complex systems and complex adaptive systems may never
| reach a defined settled state.
|
| > Basically, Horizontal Pod Autoscaler but with sensors which are
| not just "CPU"
|
| That's already available. In addition to scaling on built-in
| metrics such as cpu and mem, there are ways to create custom
| metrics, including queue depth. You can do this because
| Kubernetes is extensible
|
| > Why are the storage and networking implementations "out of
| tree" (CNI / CSI)?
|
| It used to be in-tree, until the number of third party cloud and
| in-house storage and networking providers became unwieldy. This
| goes along with that fundemental characteristic of the Kubernetes
| design -- extensibility. AWS owns the Elastic Block Storage CSI,
| and GCP owns its CSI for its storage device. CNI allowed for the
| various service meshes, including exciting new ones such as the
| one based on eBFP.
|
| The Cynefine framework again, helps sheds some light on this: the
| best way to respond to things in the complex domain is to try a
| lot of different approaches. Even if one approach doesn't look
| like it works now, some future state may make that previously
| impractical approach to work well.
|
| > Given the above question, why is there explicit support for
| Cloud providers?
|
| The current versions of Kubernetes pushes those implementations
| to CNI and operators. So for example, in order to make use of AWS
| ELB/ALB for the Ingress object, you have to additionally install
| the AWS-specific driver. If you are using AWS's EKS service, this
| is managed as an EKS addon. Under the hood, these drivers are,
| guess what, pods managed by replicasets managed by deployments
| that listens to the Kubernetes API servers for changes to the
| Ingress resource.
|
| Not everyone uses it. On one of the Kubernetes sites I worked on,
| we used Traefik inside Kubernetes and its custom resource
| definition, IngressRoute. Everytime you create an Ingress, the
| AWS driver will create a completely new Load Balancer, which of
| course, drives up cost for very little gain.
| deivid wrote:
| Thanks for the detailed answer
|
| > extensibility
|
| This is something that irks me right now, but at my current
| knowledge level (~0) it didn't feel right to even mention:
|
| If Kubernetes pushes everything out (CSI, CNI, Cloud provider
| integration, LoadBalancer, ...), doesn't it become "just a
| control loop runner"?
|
| I'm sure there's value in that, but I can imagine that most
| people running clusters would make different choices for
| various options, landing you with _every cluster being unique_,
| which feels wrong somehow.
| hosh wrote:
| That is correct. Every cluster is unique. What we have
| instead are a set of standard design patterns that are
| broadly (but not universally) applicable.
|
| Even the core components can be swapped out so that
| Kubernetes can be adapted. For example, there are several
| teams that wrote kubelet replacements. The built-in scheduler
| for kubernetes is only responsible for pod placement by
| contacting the kubelet on a node and changing its desired
| state. The kubelet is responsible for actual placement.
|
| That meant people have created kubelets that could run wasm
| workloads as pods. This was how native Windows workloads were
| implemented. I remember correctly, there are also kublets
| that could run unikernels, microvms, etc. The design pattern
| of label selectors remains the same, so that same concept can
| be used to place those specialized workloads.
|
| I also learned something from an interview I once had ...
| their platform team came to the conclusion that every site is
| unique because every company will have its own specific needs
| and unique combination of technology. They (deliberately)
| don't use Kubernetes. They even crafted their interview
| process for that. It's a timed session to attempt to do
| something deceptively simple, that turned out to be an ops
| nightmare. There are no internet resources describing how to
| do that. No one is expected to finish it within the time.
| Instead, they are looking at how someone work through the
| problem as well as their emotional regulation while under
| time pressure.
|
| Finally, I'd like to draw attention to a fallacy that has
| plauged modernity, and it is very applicable to computing
| design. This is from the field of political science, from a
| book written by James C. Scott called "Seeing Like a State".
| There is a blog post about the key concept -- legibility --
| https://www.ribbonfarm.com/2010/07/26/a-big-little-idea-
| call...
|
| The pursuit of legibility has lead us to imposing a overly
| simplistic view onto something complex instead of really
| understanding the complexity. In the case of governments,
| this imposition uses authoritarian powers, and results of
| fragile solutions. In the case of computing infrastructure,
| we create abstractions and are surprised when it leaks.
| skywhopper wrote:
| Why is he mad at Kubernetes about Helm? Yes, Helm is a mess. Yes,
| lots of people use Helm to deploy things to Kubernetes, but
| critiques of Helm are not valid critiques of Kubernetes.
| andix wrote:
| As long as there is no popular replacement for Helm, it will
| stay tightly coupled with Kubernetes. Not in a technical way,
| but in the way that you're always going to use it together with
| k8s. For now Helm is the only sane way to package a complex k8s
| deployment in a reusable way. Kustomize is great too, but quite
| limited in many ways.
| sigwinch28 wrote:
| If we replace "YAML" with "JSON" and then talk about naive text-
| based templating, it seems wild.
|
| That's because it is. Then we go back to YAML and add whitespace
| sensitivity and suddenly it's the state-of-the-art for declaring
| infrastructure.
| mark242 wrote:
| Here's what I would like from a Kubernetes-like system.
|
| I have a collection of machines.
|
| I have a very simple file that defines A) what I want to run, B)
| how the pieces communicate to each other, and C) how I want it to
| scale.
|
| Make that work without me having to think about any of the
| underlying infrastructure. (If this sounds suspiciously similar
| to Heroku, there you go)
| jfengel wrote:
| I think I'd be even happier if it had reasonable defaults for
| C. At least as a novice, I probably don't know how it wants to
| scale, and should be able to figure it out better than me.
| hosh wrote:
| There are tools that can do that, and are not as complex as
| Kubernetes
|
| I once interviewed at a place where their notion of "scaling"
| is to turn up the knob on Heroku. They also pay for it. They
| did not have anyone on their team who knew what questions to
| ask. They had grown to the point where they had outgrown their
| approach, and yet, have never developed the in-house
| capability.
|
| I mentioned the Cynefine framework elsewhere, and I'll mention
| it again. Infrastructure and how it supports application,
| users, stakeholders, is a _complex_ system.
|
| A lot of people treat this as if it were in the _complicated_ ,
| or even _clear_ domain. For certain class of use-cases, you can
| get away with this. But a company itself is a complex adaptive
| system, whose needs changes as it matures and scales ... or it
| doesn 't, and it collapses.
|
| The Cynefine framework describes four different domains, and
| the way you make decisions are different for each of them. You
| start getting in trouble when you attempt to use the decision-
| making process for one domain to a different domain. This often
| happens when the situation has evolved without the decision-
| makers noticing. It is easier for experts to see how clear
| domains changes to complicated domains, but it is more
| difficult to see when complicated domains change to complex
| domains.
|
| Kubernetes is a sophisticated, adaptive, extensible tool for
| the complex domain. If it seems overly complex, it is because
| the domain is complex, and is not necessary if you are working
| from the complicated or clear domain.
| movedx wrote:
| No idea why you're being downvoted. Makes sense to me.
| AcerbicZero wrote:
| This was a solid write up; I've been using K8s (intermittently)
| for like, 5 years now, and I still spend an inordinate amount of
| time looking things up and trying to convert the nonsense naming
| conventions used to something understandable. I can think of 20
| or so projects that would have run great on K8s, and I can think
| of 0 projects that were running on K8s, which worked well.
|
| Eventually, seeing the wrong tool used for the wrong job time and
| time again I came around to seeing K8s as the latest iteration of
| time sharing on a mainframe, but this time with YAML, and lots of
| extra steps.
| cybrexalpha wrote:
| > Why are we generating a structured language (YAML), with a
| computer, by manually adding spaces to make the syntax valid?
|
| Yep, it sucks. It's not like nobody has tried to do better, but
| nothing else has the adoption of Helm. Ultimately text is, as
| always, universal.
|
| If you want a fun fact: the communication between kubectl and the
| kube-api-server is actually in JSON, not YAML.
| movedx wrote:
| > If you want a fun fact: the communication between kubectl and
| the kube-api-server is actually in JSON, not YAML.
|
| YAML is a super set of JSON. All JSON is valid YAML. No one
| uses YAML over the wire as you cannot guarantee the conversion
| from YAML to JSON as YAML is a super set and make contain
| things like anchors, which the JSON parser cannot handle.
|
| That's why :)
| zbentley wrote:
| Nobody tell the JSON::XS CPAN maintainer you said that:
| https://metacpan.org/pod/JSON::XS#JSON-and-YAML
|
| Or John Millikin: https://john-millikin.com/json-is-not-a-
| yaml-subset
| FridgeSeal wrote:
| Solid write up, but small nitpick with the diagram at the start:
|
| It displays a pod containing multiple containers (this is fine
| and normal) but then highlights some of those containers to be
| different services.
|
| Unless you guys are running wildly different setups, or we're
| talking about sidecar and init containers which is a whole rabbit
| hole unto itself, I put different services in different pods.
| globular-toast wrote:
| Also the same diagram shows nodes being lower in the hierarchy
| than namespaces. Namespaces and nodes are orthogonal.
| jongjong wrote:
| Kubernetes provides a relatively simple abstraction to represent
| a cluster of machines as a single entity. It's just about as
| simple as it can be IMO.
|
| Its simplicity leverages the idea that each part of your software
| should be fully responsible for its own lifecycle and can handle
| and recover from all scenarios that can impact it.
|
| For example, if your app service happens to launch before your
| database service, then it should be able to seamlessly handle
| this situation and keep trying to reconnect until the database
| service is started. This characteristic is generally desirable in
| any operating environment but it is critical for Kubernetes. Its
| orchestration model is declarative, so it doesn't make much fuss
| over launch order... Yet it works well and simply if each of your
| services have this failure tolerance and self-healing
| characteristic.
___________________________________________________________________
(page generated 2024-08-01 23:00 UTC)