[HN Gopher] Scaling Kubernetes to Thousands of CRDs
___________________________________________________________________
Scaling Kubernetes to Thousands of CRDs
Author : hasheddan
Score : 107 points
Date : 2022-08-15 16:11 UTC (6 hours ago)
(HTM) web link (blog.upbound.io)
(TXT) w3m dump (blog.upbound.io)
| tchasnyk wrote:
| Is there a convenient way to limit the number of CRDs that
| Crossplane provider installs? For instance, someone wants to
| expose XRD only for one specific AWS service and completely
| ignore other AWS services
| hasheddan wrote:
| Crossplane maintainer here -- this has come up a number of
| times in the community, but ultimately we would prefer for the
| system to be able to handle an arbitrary number of CRDs,
| whether they are being used or not (as evidenced by the work
| outlined in this post). Moving to filtering CRDs that a
| provider installs can lead to a confusing experience where the
| presence of a given package does not mean the same thing across
| clusters, which also complicates the dependency model
| Crossplane packages implement. We view Composition + RBAC as
| the mechanisms to define what a given entity can access (i.e.
| the API line).
| dham wrote:
| cheriot wrote:
| How is that related to "Scaling Kubernetes to Thousands of
| CRDs"?
| [deleted]
| 5t4nl3y wrote:
| yeAh HoW iS tHiS ReLatEd!?
| candiddevmike wrote:
| Hearing the phrase "thousands of CRDs" can do funny things to
| people. It certainly makes me feel uneasy.
| cheriot wrote:
| I'll take the K8s API model over aws/gcp/azure/etc specific
| tools. I hope it expands to cover everything I need.
| vbezhenar wrote:
| As an alternative, you can use terraform to manage
| different infrastructures and run terraform from
| Kubernetes. Terraform is standard-de facto for those
| things and lots of high quality providers already exist.
| Also its language is quite expressive.
| tchasnyk wrote:
| Thank you for an excellent post on Crossplane usage in real life!
| This piece of technology is just starting its way to production
| systems, but it's definitely a great way to manage application
| infrastructure.
|
| As far as I understand, use cases for Crossplane - it supposed to
| be used together with CD systems, such as ArgoCD. ArgoCD on its
| own add quite a lot of CRDs to cluster, so the problem
| highlighted in the article is very relevant to heavy used
| clusters
| negz wrote:
| Post author here - thanks for the kind words!
|
| I wouldn't say Crossplane is _supposed_ to be used together
| with CD systems (that's optional) but that's certainly a common
| and practical use case.
| jontro wrote:
| Knit pick, the screen shot in the beginning of the post shows a
| postgresql server transformed into an mysql server.
| negz wrote:
| Haha I noticed that also shortly after I made this graphic.
| I've used it in a few talks and posts now and always worried
| someone would notice. Good eye.
| moondev wrote:
| Awesome post. It's very cool to see how Crossplane is pushing the
| boundaries of CRDs, as well as sharing their insights with the
| community and upstream.
|
| From the post it seems like interacting with Crossplane Managed
| Resources is done via a simplified "userspace" CRD/controller and
| not directly. Is direct interaction a Crossplane anti-pattern? It
| seems like both approaches have their own advantages.
| negz wrote:
| We certainly design Crossplane with the intent that end-users
| will interact with a "Composite Resource" API, not use MRs
| directly. Little bit like end-users using a Terraform module
| curated by their platform team rather than writing resources
| directly.
| andix wrote:
| Do I understand it correctly, that the kubernetes API is only
| used as a configuration management tool? So just that part, and
| not as container orchestration tool?
| negz wrote:
| Correct. We take the control plane of Kubernetes and extend it
| so that it can be used to configure/orchestrate anything, not
| just containers.
| CleverLikeAnOx wrote:
| This is great work! Even with just a few dozen CRDs kubectl can
| get a little slow on versions prior to these changes.
|
| I am excited about crossplane because handling infrastructure via
| a reconciliation loop addresses most of my problems with
| terraform. Namely, managing/sharing a state file, importing
| resources that already exist, and ensuring that reality doesn't
| drift from terraform state.
| theptip wrote:
| I do like this direction. What's the current state of the art
| for rendering the diffs? E.g. in Terraform your `terraform
| plan` will show whether the change requires a delete/create or
| just an edit.
|
| Naively, if I just look at the raw CRD diff, that information
| won't be apparent.
| negz wrote:
| We've thought about a client-side tool to diff desired from
| most-recently-observed actual state, similar to tf plan.
| Nothing built yet though.
|
| FWIW though we never automatically delete-and-recreate in
| Terraform fashion. Our thinking is that once Kubernetes
| supports it (or once we build an admission control webhook
| for it) we'd like to explicitly mark immutable fields as
| such, which would require you to explicitly delete-then-
| recreate. Little bit less declarative at the expense of being
| a lot less surprising given the constantly reconciling nature
| of our system.
___________________________________________________________________
(page generated 2022-08-15 23:01 UTC)