[HN Gopher] Cloudflare incident on October 30, 2023
___________________________________________________________________
Cloudflare incident on October 30, 2023
Author : todsacerdoti
Score : 65 points
Date : 2023-11-01 16:43 UTC (6 hours ago)
(HTM) web link (blog.cloudflare.com)
(TXT) w3m dump (blog.cloudflare.com)
| ChrisArchitect wrote:
| The Ask thread about it while it was occurring:
| https://news.ycombinator.com/item?id=38074906
| geocrasher wrote:
| Heh. They needed Workers KV running to get Workers KV running.
| ryanjshaw wrote:
| They say they needed to use a manual/break glass process to get
| things going again. Kinda scary to think Cloudflare could brick
| itself.
| swatcoder wrote:
| > Kinda scary to think Cloudflare could brick itself.
|
| Never forget that essentially all interesting systems _may_
| brick themselves and _will_ at least catastrophically fail
| every now and then. For all the sales pitches about "the
| cloud" providing resiliency, it's fundamentally centralizes
| an ungodly share of the internet onto a handful of failure
| points.
|
| What had been a frothy background noise of independent
| hosting and data center failures has become a global blackout
| risk, where worldwide operations break down and failures
| cascade through interdependent systems.
|
| On a business level, it's hard to care about that, as you're
| not going to be held to blame if everything else goes into
| crisis at the same time as you, but on a societal level, it's
| kinda f---'d.
|
| It's like moving all the agriculture onto the flood plain.
| You get maximum abundance most of the time, but when it fails
| in those inevitable floods, you at least better have kept
| some stockpiles stored uphill somewhere.
| starttoaster wrote:
| They don't say what the manual/break glass process was
| exactly. But I likened it to something like a `helm rollback`
| for a kubernetes helm release. Manual/break-glass doesn't
| have to mean a team of engineers are logged into VM instances
| `vi` editing config files back to a previous state and
| `scp`-ing older binaries over, it just probably means there
| wasn't an automated build/deploy tool process to perform it.
|
| > Kinda scary to think Cloudflare could brick itself.
|
| I mean.. They are providers of software that iterate on their
| product suite in a SaaS environment. I think this sentiment
| holds for pretty much any SaaS company. And I guess if
| everything is scary then nothing is really all that scary. Do
| you disagree?
| ryanjshaw wrote:
| I meant "brick" as distinct from "break", in the sense that
| their apparent circular dependencies could result in an
| unrecoverable failure scenario. Purely speculation of
| course, it's probably not a real risk.
|
| That being said, it's a scary thought because they are
| different from other SaaS in that the run a significant
| portion of the internet.
| wbl wrote:
| You know how cell sites have gas generators and the gas line
| uses cell to communicate status? I've always wondered about
| that one.
|
| Plus would a diesel seller be able to take cash if a data
| center needed it to start up again. There's plenty more where
| that thoughts from
| thedaly wrote:
| I don't know about cell towers, but all AT&T central
| offices have generators with massive fuel tanks that can
| run even if the gas supply is cut off. I would be surprised
| if the backup for cell towers didn't have similar systems.
| remram wrote:
| I worry a lot about those things. I am sending this to my
| colleagues.
|
| It's too easy to set up new sysop tools on the shiny new
| cluster/filesystem/cloud that makes everything easy. However
| how do you fix that shiny but complex thing if your tools are
| down?
| nickenumbers wrote:
| Incident report was solid.
| LoganDark wrote:
| This is an incredibly detailed and honest incident report. Good
| on CloudFlare for getting things running again so quickly, and
| identifying so many problem areas to improve.
| cowthulhu wrote:
| I'd be interested to know more about the "break glass" mechanism.
| Anyone know of any blog posts from CF on this topic?
| xeromal wrote:
| I assume when they said that they mean a hacky, manual, editing
| config files means of redirecting traffic.
| vrosas wrote:
| > make build && ENV=prod make deploy
| waythenewsgoes wrote:
| Likely involves their most senior devs bypassing normal
| deployment procedures / access controls. In this case, to be
| able to edit their live production environment instance instead
| relying on Workers KV / other impacted internal services (which
| a lot of their infrastructure relies on) Which would then
| enable a proper rollback. The aim being to restore service as
| quickly as possible to minimize the customer impact.
| CodeWriter23 wrote:
| > Workers KV is our globally distributed key-value store.
|
| IOW our single point of failure failed
| JohnMakin wrote:
| A continuing theme in these broad outages that keep happening is
| that these company's are dogfooding their own services to provide
| the same service to their customers.
|
| I'm thinking of the AWS lambda outage taking out the control
| plane as a recent example.
|
| I'm not sure or saying they're _wrong_ for doing this, I just
| keep noticing this same pattern of "One of our products broke,
| now all of our tooling is failing." This seems.. wrong somehow?
| depr wrote:
| I'm not as concerned about a large part of internet traffic going
| through Cloudflare as many here are, but I think when a service
| like CF becomes more and more popular, it has to grow, and there
| is the irresistible impulse to do more complex things. Inevitably
| this complexity leads to outages/issues.
___________________________________________________________________
(page generated 2023-11-01 23:02 UTC)