[HN Gopher] Outage Stories: The copy and paste outage
       ___________________________________________________________________
        
       Outage Stories: The copy and paste outage
        
       Author : kevin_nisbet
       Score  : 42 points
       Date   : 2022-08-13 22:25 UTC (1 days ago)
        
 (HTM) web link (wdkwwdk.com)
 (TXT) w3m dump (wdkwwdk.com)
        
       | mananaysiempre wrote:
       | The way I read this, the redundancy mechanism depends on a
       | process actually being dead when monitoring thinks it's dead. But
       | that's really not the way you're supposed to do this, is it?
       | Maybe it's just... resting. At the very least it seems like the
       | allegedly dead process should be shot in the head before
       | activating the backup one (although that's likely quite difficult
       | to do reliably even if the mental model is simple).
        
         | kevin_nisbet wrote:
         | You're correct, it was a bad design. The vendor that built this
         | system had a number of issues in this realm that demonstrated a
         | lack of experience in building for the reliability targets of
         | telecom. This was just one of the easier stories to tell.
         | 
         | STONITH is a good design option, but I think this
         | implementation would've still struggled. The way they were
         | tracking processes was fragile, so the shoot the process lost
         | track of which pid it should signal. And this was a sort of
         | clustered system, so if you shoot the node you take out other
         | services (think of murdering a kubernetes node to terminate one
         | process, you're going to inflict some spillover.)
        
       | user00012-ab wrote:
       | unrelated to whatever content was on that page, but for some
       | reason I just don't trust websites that fade the text in; I also
       | dislike pages like blogger.com that have a "loading page" intro
       | to every site.
       | 
       | Just feels dirty or something.
        
       | bombcar wrote:
       | Often "zombie" systems/processes/hardware are much harder to
       | handle resiliently than actually exploded/dead. You need end-to-
       | end testing of the system to detect that even though everything
       | "seems" right, something is wrong (and potentially even to decide
       | that "box X is testing correct but not working, kill it".
        
       | justinsaccount wrote:
       | > And the way this system detected a process failure was to
       | search the process list for a matching process. If we run the
       | command super-ha-proxy..., and then it's in the process list, the
       | process is working. If it's not in the process list, it must have
       | crashed, exited, etc. and we need to launch a new instance.
       | 
       | > The supervisor script just starts detecting the old process,
       | and has no idea that the additional instance has been launched
       | 
       | The number of times I've seen outages and weird flaky problems
       | from some shitty script trying (and failing) to parse ps
       | output...
       | 
       | daemontools, runit, s6, systemd, etc all solve this problem
       | properly.
        
       | tpankaj wrote:
       | This is why production services should have health checks that
       | actually verify whether the service is doing its job in the
       | system, not whether it's merely running the process. A process
       | can be completely broken while still technically running, and
       | vice versa (at least with janky ps-parsing scripts).
        
       | latchkey wrote:
       | It is interesting how the article talks about having to provide
       | excellent uptime and then goes on to describe how the system was
       | designed to be brittle. This is less about a copy/paste error and
       | more about system design.
        
       ___________________________________________________________________
       (page generated 2022-08-14 23:01 UTC)