https://ahmet.im/blog/controller-pitfalls/ /home/ahmetb * Blog * About me * Tweets * GitHub * Talks So you wanna write Kubernetes controllers? What they don't tell you about developing scalable and reliable controllers Published on 22 January 2025 (est. 17 min read) kubernetes controller development Any company using Kubernetes eventually starts looking into developing their custom controllers. After all, what's not to like about being able to provision resources with declarative configuration: Control loops are fun, and Kubebuilder makes it extremely easy to get started with writing Kubernetes controllers. Next thing you know, customers in production are relying on the buggy controller you developed without understanding how to design idiomatic APIs and building reliable controllers. Low barrier to entry combined with good intentions and the "illusion of working implementation^1" is not a recipe for success while developing production-grade controllers. I've seen the real-world consequences of controllers developed without adequate understanding of Kubernetes and the controller machinery at multiple large companies. We went back to the drawing board and rewritten nascent controller implementations a few times to observe which mistakes people new to controller development make. 1. Design CRDs like Kubernetes APIs 2. Single-responsibility controllers 3. Reconcile() method shape 4. Report status and conditions 5. Learn to use observedGeneration 6. Understand the cached clients 7. Fast and offline reconciliation 8. Reconcile return values 9. Workqueue/resync mechanics 10. Expectations pattern 11. Conclusion Design CRDs like Kubernetes APIs It takes less than 5 minutes to write a Go struct and generate a Kubernetes CustomResourceDefinition (CRD) from it thanks to controller-gen. Then it takes several months to migrate from this poorly designed API to a better v2 design while the old API is being used in production. Don't do that to yourself. If you're serious about developing long-lasting production grade controllers, you have to deeply understand the API Conventions that Kubernetes uses to design its builtin APIs. Then, you need to study the builtin APIs, and think about things like "why is this field here", "why is this field not a boolean", "why is this a list of objects and not a string array". Only when you're able to reason about the builtin Kubernetes APIs and their design principles, you'll be able to design a long-lasting custom resource API. Beginners not grasping these API conventions often make these mistakes: 1. They don't understand the difference between status and spec and who should be updating each field (more about this later ). 2. They don't understand how to embed a child object within a parent object (e.g. how Deployment.spec.template becomes a Pod) so they end up re-creating child object properties in the parent object, usually with a worse organized structure. 3. They don't understand field semantics well (e.g. zero values, defaulting, validation) and end up with fields that are not set, or set to wrong values accepted into the API. I covered this topic in my CRD generation pitfalls article. If the behavior of the API is not clear when a field is not set, you've already failed. API conventions guide covers this topic fairly well. If you study the builtin Kubernetes APIs extensively, you'll find out things like spec field is not a "must have"^2, and not all APIs offer a status field. I would go as far as to say that you should also study custom APIs of projects like Knative, Istio and other popular controllers to develop a better understanding of organizing fields, and how to reuse some core types Kubernetes already offers (like ControllerRevision, PodSpecTemplate). Single-responsibility controllers Time and time again we find engineers adding new unrelated responsibilities to existing controllers because it seems like a good place their thing can be shoved into. Kubernetes core controllers don't have this problem for a reason. One of the main Kubernetes design principles is that controllers have clear inputs and outputs --and they do a well-defined job. For example, the Job controller watches Job objects and creates Pods, which is a clear mental model to reason about. Similarly, each API is designed to offer a well defined functionality. A controller's output can be an input to another controller. This is all what the UNIX philosophy suggests for a well reasoned system. I recommend studying the common controller shapes (great talk by Daniel Smith, one of the architects of kube-apiserver) and the core Kubernetes controllers. You'll notice that each core controller in Kubernetes core has a very clear job and inputs/outputs that can be explained in a small diagram. If your controller isn't like this, you're probably misarchitecting either your controller, or your CRDs. If you architect your APIs and controllers correctly, your controllers will run in harmony as if they're integrating with Kubernetes core APIs or an off-the-shelf operator. When you controller design doesn't quite feel right, or has too many inputs/outputs, does too much, or in general doesn't feel right, you're probably doing it unidiomatically. I struggled with this a lot myself, especially while developing controllers that manage external resources that have a non-declarative configuration paradigm. Reconcile() method shape Assuming you use kubebuilder (which uses controller-runtime like almost everyone else to develop a controller, and you implement the Reconcile() method that controller-runtime invokes every time one of your inputs change. This is where your controller does its magic, and since it's possible to implement this method in any way, most beginners dump their spaghetti here. Therefore, large projects like Knative define their own common controller shapes where every controller runs the same set steps in the same order. By developing a common controller shape/framework, you create a "guardrail" so that other engineers don't deviate and introduce bugs in the reconciliation flow easily. Sadly, controller-runtime is not opinionated about this topic. Your best bet is to read other controllers (like Cluster API to learn the idioms and master the reconciliation flow. There are also new projects like Apollo SDK by Reddit that claims to offer finite-state machines for controller-runtime reconcilers. Over time, we found that almost all our controllers have a similar reconciliation flow. Here's a pseudo-code of how our controllers look like: func (m *FooController) Reconcile(..., req ctrl.Request) (ctrl.Result, error) { log := ctrl.LoggerFrom(ctx) obj := new(apiv1.Foo) // 1. Fetch the resource from cache: r.Client.Get(ctx, req.NamespacedName, obj) // 2. Finalize object if it's being deleted // 3. Add finalizers + r.Client.Update() if missing orig := foo.DeepCopy() // take a copy of the object since we'll update it below foo.InitializeConditions() // set all conditions to "Unknown", if missing // 4. Reconcile the resource and its child resources (the magic happens here). // Calculate the conditions and other "status" fields along the way. reconcileErr := r.reconcileKind(ctx, kcp) // magic happens here! // 5. if (orig.Status != foo.Status): client.Status.Patch(obj) } Most notably, you'll see here that we always initialize conditions and always update status inside reconcileKind even if the reconciliation fails. I recommend enforcing a similar common shape for controllers developed at your company (you can use custom kubebuilder plugins during scaffolding, but you can't really enforce that either). Report status and conditions I practically never seen a beginner engineer create a CRD that has properly designed status fields (if one exists, at all). Kubernetes API conventions discuss this at length, so I'll keep it brief. If an API object is reconciled by a controller, the resource should expose its status in status fields. For example, there's no ConfigMap controller so ConfigMap doesn't have a status field. At LinkedIn, our custom API objects have a status.conditions field, similar to the Kubernetes core or Knative conditions, and we use something similar to Knative condition set manager that provides high-level accessor methods to set the conditions, and sort them etc. This helps us define and report conditions for API objects in a high-level way in the reconciler code: func (r *FooReconciler) reconcileKind(obj *Foo) errror { // Create/configure a Bar object, wait for it to get Ready if err := r.reconcileBar(obj); if err != nil { obj.MarkBarNotReady("couldn't configure the Bar resource: %w", err) return fmt.Errorf("failed to reconcile Bar: %w", err) } obj.MarkBarReady() } Every time we mark a condition, the condition manager recalculates the top-level Ready condition, which all our objects have as Kubernetes API conventions suggest. Other controllers and humans consume this top-level condition to understand how the objects are doing (plus you get to use kubectl cond on your objects). Learn to use observedGeneration Something notable is that all our conditions have an observedGeneration field. You'll even see some popular community CRDs (like ArgoCD Application) do not offer this field. Essentially, this field tells us whether the condition is calculated based on the last configuration of the object --or whether we're looking at a scale status information because the controller hasn't gotten to reconciling the object after the update. For example, observing a Ready condition set to True alone means nothing (other than at some point in the past it was true). The condition offers a meaningful status info if and only if the cond.observedGeneration == metadata.generation. Real-world story: A controller we had in production didn't have the notion of observedGeneration, so its callers would update the object's spec and immediately check its Ready condition. This condition would almost always be stale, as the controller hadn't reconciled the object yet. So the callers interpreted an app rollout as completed, even though it hadn't even started yet (and sometimes actually failed, but that failure was never noticed). Understand the cached clients controller-runtime, by default, gives you a client to the Kubernetes API which serves the reads from an in-memory cache (as it uses shared informers from client-go under the covers). This is mostly fine, as controllers are designed to operate on stale data --but it is detrimental if you didn't know this was the case since you might be writing buggy controllers due to this (more on this later in "expectations" section). When you perform a write (which directly hits the API server), its results may not be immediately visible in the cached client. For example, when you delete an object, it may still show up in the list result in a subsequent reconciliation to your surprise. The lesser-known behavior of controller-runtime most beginners don't realize is that controller-runtime establishes new informers on-the-fly. Normally, when you specify explicit event sources while building your controller (e.g. in builder.{For,Owns,Watches}), the informers are started and caches are started during startup. However, if you try to make queries with client.{Get,List} on resources that you haven't declared upfront in your controller setup, controller-runtime will initialize an informer on-the-fly and block on warming up its cache. This leads to issues like: * Controller-runtime starting a watch for a resource type and start caching all its objects in memory (even if you were trying to query only one resource), potentially leading to the process running out of memory. * Unpredictable reconciliation times while the informer cache is syncing, during which your worker goroutine will be blocked from reconciling other resources. That's why I recommend setting ReaderFailOnMissingInformer: true and disabling this behavior so you're fully aware of what kinds your controller is maintaining watches/caches on. Otherwise, controller-runtime doesn't provide any observability on what informers it's maintaining in the process. controller-runtime offers a lot of other cache knobs, such as entirely disabling the cache on certain types, dropping some fields from the in-memory cache, or limit the cache to certain namespaces. I recommend studying them to better understand how you can customize the cache behavior. Fast and offline reconciliation Reconciling an object that is alredy up-to-date (i.e. goal state == current state) should be really fast and offline --meaning it should not make any API calls (to external APIs or writes to Kubernetes API). That's why controllers use cached clients to serve reads from the cache to determine the state of the world. I've seen many real-world controllers making API calls to external systems, or make status updates to Kubernetes API (even when nothing has changed) every time Reconcile() got invoked. This is an anti-pattern, and a really bad idea for writing scalable and reliable controllers: 1. They bombarded the external APIs with unnecessary calls during controller startup (or full resyncs, or when they had bugs causing infinite requeue loops) 2. When the external API was down, reconciliation would fail even though nothing has changed in the object. Depending on the implementation, this can block the next steps in the reconciliation flow even though those steps don't depend on this external API call. 3. Logic that takes long to execute in a reconciliation loop will hog the worker goroutine, and cause workqueue depth to increase, and reduce the throughput/responsiveness of the controller as the worker goroutine is occupied with the slow task. Let's go through a concrete example: Assume you have an S3Bucket controller that creates and manages S3 buckets using AWS S3 API. If you make a query to S3 API on every reconciliation, you're doing it wrong. Instead, you should store the result of the S3 API calls you made, in a field like status.observedGeneration, to reflect what's the last generation of the object that was successfully conveyed to S3 API. If this field has 0 value, the controller knows it needs to make a "Create Bucket" call to S3 API. When a client updates the S3Bucket custom resource, its metadata.generation will no longer match its stored status.observedGeneration, so the controller knows it needs to make a "Update Bucket" call to S3 API, and only upon success it will update the status.observedGeneration field. This way, you avoid making calls to the external S3 API when the object is already up-to-date.^3 Reconcile return values Your Reconcile() function signature returns ctrl.Result +error values. Usually beginners don't have a solid grasp on what values to return from Reconcile(). You should know that your Reconcile() function is invoked every time your event sources declared in builder.{For,Owns,Watches} changes. If you know that, my general advice while returning reconciliation values: 1. If you have errors during reconciliation, return the error; not Requeue: true. Controller-runtime will requeue for you. 2. Use Requeue: true only when there's no error but something you started is still in progress, and you want to check its status with the default backoff logic. 3. Use RequeueAfter: