[HN Gopher] For the love of god, stop using CPU limits on Kubern...
___________________________________________________________________
For the love of god, stop using CPU limits on Kubernetes (updated)
Author : ciceryadam
Score : 34 points
Date : 2022-09-27 14:54 UTC (8 hours ago)
(HTM) web link (home.robusta.dev)
(TXT) w3m dump (home.robusta.dev)
| rahen wrote:
| We use CPU limits at work for the simple reason we can't
| autoscale deployments without having them set. An HPA will deploy
| a new pod each time the CPU limit has been reached for more than
| 30 seconds.
|
| The whole point is to scale out, not up.
| ekimekim wrote:
| The argument against this is consistency. Without a limit set,
| you are only guarenteed up to your request's worth of cpu, but
| you will often be allowed to have more. This can create a false
| sense of security, as your application is working fine (even
| though it occasionally exceeds its request). Until one day, when
| a neighbor happens to get thirsty, and your application suddenly
| breaks. Limits front-load the brokenness so that it happens
| immediately instead of randomly.
| zufallsheld wrote:
| > Until one day, when a neighbor happens to get thirsty, and
| your application suddenly breaks. Limits front-load the
| brokenness so that it happens immediately instead of randomly.
|
| Can you expand on this? Why should the application break? It
| still has its guaranteed cpu requests. If it breaks with the
| defined requests, shouldn't it always break?
| kobalsky wrote:
| Not op, but it would break because your CPU request wasn't
| high enough from the start and the problem was hidden because
| you had CPU to spare in the node. Once you don't have CPU to
| spare, the app breaks.
|
| EDIT: the vertical pod autoscaler helps with this as it will
| adjust requirements on pods to make sure they don't
| over/under allocate.
| sascha_sl wrote:
| I've had this happen in the real world.
|
| In this case it was a GraphQL implementation that gradually
| grew in size, complexity and scope. The team maintaining it
| never adjusted the initial resource requirements within about
| 2 years because it has never been a problem - until a
| different (much larger by total CPU allocated) service
| started consuming all the bursting it could get and the
| GraphQL service just stopped processing requests. Before any
| autoscaling could kick in, it went into CrashLoopBackoff with
| failing liveness.
|
| You can and should catch such cases early with monitoring,
| but our platform team was extremely tiny, especially for how
| many developers it served.
| stingraycharles wrote:
| It's a trade-off that needs to be considered on a case-by-case
| basis. This is one of those cases where "one rule fits all"
| doesn't work.
| skyde wrote:
| I don't agree on the recommendation for memory << Always set your
| memory requests equal to your limits >>
|
| you can layer high priority service and low priority service
| better if you use some buffer.
| clhodapp wrote:
| My current model for memory on k8s tends to agree with the
| article.
|
| Would it be possible for you to explain " _you can layer high
| priority service and low priority service better if you use
| some buffer_ " further?
| 988747 wrote:
| The reason to never use CPU limits is different than those stated
| in the article. In short: Linux kernel sucks. More specifically,
| the "Completely Fair Scheduler" (CFS) sucks at enforcing those
| limits. Setting any limit at all causes CFS to waste like half of
| CPU cycles on enforcing it, and only the other half is available
| for any useful work.
| iknownothow wrote:
| This advice comes from tunnel vision and makes perfect sense if
| you know that you have exactly two pods running at any given
| time. But if you have exactly two pods, then why bother use k8s?
| IIRC one of the major selling points of K8s was on-demand scaling
| or auto scaling horizontally. Which means the number of pods you
| have in the cluster is dynamic.
|
| In the context of pods dynamically spinning up and spinning down,
| it's bad when a pod replica can't be allocated in the cluster
| "predictably" but there is nothing worse than when a new pod (new
| deployment) fails because "Marcus the pod" drank all the water
| and now I have to call DevOps and wait god knows how long before
| they spin up a new node to guarantee a spot for the new pod.
|
| Bin-packing is a already an np-hard problem. If you remove limits
| from CPU then you're adding probabilities into the mix. So, for
| the love of god, always use limits unless you have a very
| specific use case.
| birdyrooster wrote:
| I don't think this is a remotely compelling argument to never use
| limits.
___________________________________________________________________
(page generated 2022-09-27 23:02 UTC)