[HN Gopher] Minimizing correlated failures in distributed systems
       ___________________________________________________________________
        
       Minimizing correlated failures in distributed systems
        
       Author : kristianpaul
       Score  : 19 points
       Date   : 2022-05-25 21:28 UTC (1 days ago)
        
 (HTM) web link (aws.amazon.com)
 (TXT) w3m dump (aws.amazon.com)
        
       | 0xbadcafebee wrote:
       | I think the biggest takeaway here is _system-wide operations are
       | bad for system-wide reliability_.
       | 
       | Also watch out for snowballing. External dependent systems should
       | not rely on your _entire_ system, but a portion of it. Building
       | your system into regions and AZs is a cheap hack to enable you to
       | perform operations on a portion of your system rather than the
       | whole. If external dependent systems also depend on just a
       | portion of your system, you avoid total system collapse and
       | snowballing.
        
         | mjb wrote:
         | > I think the biggest takeaway here is system-wide operations
         | are bad for system-wide reliability.
         | 
         | Yes!
         | 
         | System-wide operations, whether they are human-driven
         | operations ("ssh onto that box"), control-plane operations
         | ("remove all the failed servers"), DI operations ("deploy the
         | new code"), or even basic algorithmic things like replication
         | ("put the same state onto all the servers") are the top causes
         | of correlation that I've seen in the wild. Whether or not this
         | matters to you depends a lot on what you're building, and how
         | often you can tolerate failures. But if you're building
         | something that needs high availability, durability, integrity,
         | etc it's worth paying a huge amount of attention to the things
         | that can introduce correlation in your systems.
         | 
         | If you're interested in reading more beyond what Joe (the OP)
         | talks about some methods of avoiding those in the article (he's
         | a colleague of mine at AWS):
         | 
         | * Our "Millions of Tiny Databases" paper goes into a lot of
         | detail on another AWS take on reducing correlated failure (http
         | s://www.usenix.org/conference/nsdi20/presentation/brooke...). *
         | Some AWS folks from the S3 team also touch on correlation in
         | this talk: https://www.youtube.com/watch?v=DzRyrvUF-C0&t=2410s
         | * I've written in the past about the role of software
         | deployments in correlated failure
         | (https://brooker.co.za/blog/2022/01/31/deployments.html), and
         | about how to think about the role of redundancy
         | (https://brooker.co.za/blog/2021/04/14/redundancy.html).
        
       ___________________________________________________________________
       (page generated 2022-05-26 23:01 UTC)