[HN Gopher] Architecture Patterns: The Circuit-Breaker
___________________________________________________________________
Architecture Patterns: The Circuit-Breaker
Author : kiyanwang
Score : 78 points
Date : 2023-11-20 07:30 UTC (1 days ago)
(HTM) web link (lab.scub.net)
(TXT) w3m dump (lab.scub.net)
| hinkley wrote:
| I'm starting to think circuit breakers are not long for this
| world.
|
| You get your front end reliably serving web pages to thousands of
| people and the bevy of search engine bots out there, and you
| start wondering why only the front end has to be robust like
| this. Why don't your services use the same set of tools to
| approach a similar level of reliability?
|
| In particular, if you have seven services or apps all calling the
| same service, a pretty common arrangement is one circuit per
| client-service pair, so you have five different clusters trying
| to track and react to the health of a sixth, when a load balancer
| in front of that service could be handling it instead.
|
| They aren't robustness in depth. They're a coping mechanism for
| not having it.
| pluto_modadic wrote:
| circuit breakers /can/ make sense, if done rationally. kinda
| like that line in fallbacks vs failovers where "if the second
| system is so reliable why bother with the first?"
|
| https://aws.amazon.com/builders-library/avoiding-fallback-in...
|
| (fallback: e.g., using direct DB instead of cache, failover:
| second cache/API)
| hinkley wrote:
| > if the second system is so reliable why bother with the
| first.
|
| Hmm. That's a difficult one to unpack.
|
| Sometimes services need to be reliable in different
| dimensions. And that first service just might not be able to
| scale high enough to do some more esoteric operation on the
| data. Though I could see someone arguing that from the user's
| perspective the first and second system are trying to pretend
| to be the same thing.
|
| I think the services I have the hardest time shooting are the
| ones that exist to prevent priority inversion. I maintain a
| batch processing system I have a hard time justifying in some
| ways. I get more value out of it as a guineapig for new tools
| and techniques we may want to apply more broadly, than I get
| from the actual task it performs, but I certainly don't want
| the core system trying to handle it either. It'd be a lot
| more noise. So I want to kill it, and I want to keep it.
|
| Some tasks have escalating priority over time, and trying to
| mix them with tasks that don't can be problematic.
| sgarland wrote:
| I've also seen people reinventing the wheel (poorly), seemingly
| to avoid existing and proven technology, like writing a bespoke
| DB load balancer / health checker in Node using Redis and
| Dynamo, instead of just standing up HAProxy.
|
| "My JavaScript will surely be more performant and less buggy
| than production-grade C that's existed for decades."
| baq wrote:
| I guess it's more 'I don't need another layer in my stack
| which I don't know how to configure; I just need this one
| feature, how hard can it be'.
| hinkley wrote:
| I have my fingers in some of our reloadable configuration
| code and one of the places I see it used a lot is basically a
| bespoke service discovery system. It's so stupid, and it adds
| a lot of overhead. There are at least three competing
| solutions (architectures, not vendors) for this problem and
| two of them were old hat before this code was ever written.
|
| Stop it.
| rbanffy wrote:
| I came to think of circuit breakers as a crude tool that has a
| very limited set of uses, much smaller than what it's used for.
| It might work as a last resort, but it should never be deployed
| as your first way to deal with failures.
|
| If at all possible, APIs should communicate server health status
| back to their clients, and clients should have ways to deal with
| partial outages of supporting services and systems.
|
| Also, that sequence diagram should have been a state diagram.
| Don't people teach proper UML in schools these days?
| bsaul wrote:
| I had the same problem understanding the state diagram but
| didn't even realize he used the wrong type of UML diagram :))
| brodouevencode wrote:
| UML is taught, sometimes, but once you get into the industry it
| is a coin flip as to whether or not it is properly used - if it
| is used at all.
| jerf wrote:
| I would think the first resort in most of the non-flooding-
| related cases[1] is backpressure. If you've got too many
| requests coming in, be sure that internally you start parking
| the excess incoming requests in some low-resource-usage
| configuration (e.g., simply noting the incoming TCP connection
| but then not even read()ing from it until you are ready), and
| let the system push back naturally by the way its latency
| increases. This obviously naturally pairs with both client-side
| and server-side timeouts, giving both a vote in how long
| they're willing to wait.
|
| This is generally what you want. Actively breaking the
| connections if I think I'm overloaded is only something I'd add
| if there is some additional constraint on the system that
| suggests it's a good idea due to some specific local concerns.
| Circuit breaking produces a much sharper (for lack of a better
| term) performance graph, where instead of degrading gracefully
| like backpressure will, you suddenly start getting active
| failures.
|
| I've built a lot of backpressure in my systems; I'm yet to need
| a circuit breaker. YMMV, of course, but I see some people
| referring to it as a basic tool and I would disagree; the basic
| tool is backpressure.
|
| [1]: Flooding is its own thing. In general, to weather DOS
| attacks but still have the service functional requires another
| system or likely _set_ of systems in front of the service that
| can handle the DOS issues. I say this just to point out that
| circuit breakers aren 't really for this case, because no
| system can on its own handle a full-on DOS flood, no matter
| what code you write, when the DOS flood may literally be coming
| in at a speed where it is faster than you can even reject
| connections, let alone do any work.
| vault_ wrote:
| Yeah, I don't think circuit breakers are really the
| appropriate choice in most of the situations the article is
| describing. Rate limiting and backpressure seem like better
| options most of the time.
|
| The way I see it, circuit breakers are safety devices.
| They're for when you need to keep a system in a safe control
| region and are wiling to sacrifice some reliability in order
| to achieve that. e.g. preventing customers from accidentally
| turning your globally distributed whatever into a DDOS
| platform or limiting the blast radius when infrastructure
| automation decides it should delete everything.
| rbanffy wrote:
| Oh yes! I used response times as a proxy for backend health a
| couple times. Still, it's better if the service tells you
| there is some degradation in a way the client can understand
| and handle. Sometimes, only some of the requests need to be
| slowed down or paused, leading to some limited functionality
| but not a complete loss of service.
| jdwyah wrote:
| When people think of circuit breakers, I think most people think
| of the automated things like hystrix / istio circuit breaking.
|
| I feel like those have their place, but in practice are often
| overkill and they are complex enough that it's hard to keep them
| loaded in my brain when I rarely think about them. What I find to
| be more broadly useful is the ability to change things like
| timeout and retry on the fly. Or other more manual kill switches
| via dynamic configuration.
|
| When something is on fire and my service is getting overloaded by
| another service that is crushing it with retries, I can then make
| a surgical change. Not automatic, but still a good MTTR. I think
| this is one of the better ways to use what we're building at
| Prefab.
| brodouevencode wrote:
| Agreed. Configurability (which is a proxy of circuit-breaking),
| especially at the border, is often overlooked when we build
| systems. If it's not configurable it probably (but not always)
| have some "smarts" to deal with problems.
___________________________________________________________________
(page generated 2023-11-21 23:04 UTC)