[HN Gopher] More than DNS: Learnings from the 14 hour AWS outage
___________________________________________________________________
More than DNS: Learnings from the 14 hour AWS outage
Author : birdculture
Score : 48 points
Date : 2025-10-27 15:56 UTC (2 days ago)
(HTM) web link (thundergolfer.com)
(TXT) w3m dump (thundergolfer.com)
| ggm wrote:
| Aside from not dogfooding, what would have reduced the impact?
| Because "don't have a bug" is .. well it's the difference between
| desire and reality.
|
| Not dogfooding is the software and hardware equivalent of the
| electricity network "black start" -you never want to be there,
| but somewhere in the system you need a honda petrol generator,
| which is enough to excite the electromagnets on a bigger
| generator, which you spin up to start the turbine spinning until
| the steam takes up load and the real generator is able to get
| volts onto the wire.
|
| Pumped Hydro is inside the machine. It's often held out as the
| black-start mechanism because it's gravity, there's less to go
| wrong, but if we are in 'line up the holes in the cheese grater'
| space, you can always have 'want of a nail' issues with any
| mechanism. The honda generator can have a hole in the petrol
| tank, the turbine at the pumped hydro can be out for maintenance.
| nijave wrote:
| A cap on region size could have helped. Region isolation didn't
| fail here so splitting us-east-1 into 3,4,5 would have been a
| smaller impact
|
| Having such a gobstoppingly massive singular region seems to be
| working against AWS
| elchananHaas wrote:
| DynamoDB is working on going cellular which should help. Some
| parts are already cellular, and others like DNS are in
| progress. https://docs.aws.amazon.com/wellarchitected/latest/
| reducing-...
| pas wrote:
| there's already some virtualization going on. (I heard that
| what people see as us-east-1a might be us-east-1c for others
| to spread the load. though obviously it's still too big.)
| JCM9 wrote:
| Legitimate question on if the talent exodus from AWS is starting
| to take its toll. I'm talking about all the senior long-turned
| folks jumping ship for greener pastures, not the layoffs this
| week which mostly didn't touch AWS (folks saying that will happen
| in future rounds).
|
| The fact that there was an outage is not unexpected... it
| happens... but all the stumbling and length to get things under
| control was concerning.
| j45 wrote:
| Hope not.. Smooth tech that runs is like the Maytag man.
|
| Tech departments running around with their hair on fire /
| always looking busy isn't one that always builds trust.
| zorpner wrote:
| Corey Quinn wrote an interesting article addressing that
| question:
| https://www.theregister.com/2025/10/20/aws_outage_amazon_bra...
|
| Some good information in the comments as well.
| Nextgrid wrote:
| If you average it out over the last decade do we really have
| more outages now than before? Any complex system with lots of
| moving parts is bound to fail every so often.
| thundergolfer wrote:
| It's the length of the outage that's striking. AWS us-east-1
| has had a few serious outages in the last ~decade, but IIRC
| none took near 14 hours to resolve.
|
| The horrible us-east-1 S3 outage of 2017[1] was around 5
| hours.
|
| 1. https://aws.amazon.com/message/41926/
| Nextgrid wrote:
| Couldn't this be explained by natural growth of the amount
| of cloud resources/data under management?
|
| The more you have, the faster the backlog grows in case of
| an outage, so you need longer to process it all once the
| system comes back online.
| JCM9 wrote:
| Not really. The issue was the time it took to correctly
| diagnose the issue and then the cascading failures that
| resulted triggering more lengthy troubleshooting. Rightly
| or wrongly it plays into the "the folks that knew best
| how all this works have left the building" vibes. Folks
| inside AWS say that's not entirely inaccurate.
| SketchySeaBeast wrote:
| I can't imagine a more uncomfortable place to try and
| troubleshoot all this than in a hotel lobby surrounded by a dozen
| coworkers.
| thundergolfer wrote:
| It wasn't too bad! The annoying bit was that the offsite
| schedule was delayed for hours for the other ~40 people not
| working on the issue.
| tptacek wrote:
| Good to see an analysis emphasizing the metastable failure mode
| in EC2, rather than getting bogged down by the DNS/Dynamo issue.
| The Dynamo issue, from their timeline, looks like it got fixed
| relatively quickly, unlike EC2, which needed a fairly elaborate
| SCRAM and recovery process that took many hours to execute.
|
| A faster, better-tested "restart all the droplet managers from a
| known reasonable state" process is probably more important than
| finding all the Dynamo race conditions.
___________________________________________________________________
(page generated 2025-10-29 23:00 UTC)