[HN Gopher] Multiple Digital Ocean services down
___________________________________________________________________
Multiple Digital Ocean services down
Author : inanothertime
Score : 111 points
Date : 2025-11-18 12:48 UTC (10 hours ago)
(HTM) web link (status.digitalocean.com)
(TXT) w3m dump (status.digitalocean.com)
| jsheard wrote:
| They don't name names but it's probably due to the ongoing
| Cloudflare explosion. I know the DigitalOcean Spaces CDN is just
| Cloudflare under the hood.
| matt-p wrote:
| Just spaces CDN, not spaces - you'd think they'd just turn the
| CDN off for a bit.
| potato3732842 wrote:
| You can't just "turn off CDN" on the modern internet. You'd
| instantly DDOS your customers' origins. They're not
| provisioned to handle it, and even if they were the size of
| the pipe going to them isn't. The modern internet is built
| around the expectation that everything is distributed via
| CDN. Some more "traditional" websites would probably be fine.
| matt-p wrote:
| If you are DO you could, you just decided not to bother.
| They control the origins it's spaces (s3), so they could
| absolutely spin up further gateways or a cache layer and
| then turn the CDN off.
| graemep wrote:
| Either you are wrong and they do not have the capacity to
| do that, or they have decided it is acceptable to be down
| because a major provider is down
|
| I imagine a cache layer cannot be that easy to spin up -
| otherwise why would they outsource it?
| matt-p wrote:
| You outsource it because clouflare have more locations
| than you so offer lower latency and can offer it at a
| cost that's cheaper or the same price as doing it
| yourself.
| graemep wrote:
| Which suggests its expensive enough for it to be unlikely
| they just have the capacity lying around to spin up.
| oasisbob wrote:
| Might be just me, but I can think of many origins under my
| control which could live without a (non-functional) CDN for
| a while.
|
| CDN is great for peak-load, latency reductions, and cost -
| but not all sites depend on it for scale 24/7
| tgma wrote:
| nit: that's more DoS (from a handful of DO LBs) than DDoS.
| TechRemarker wrote:
| Yes all sites showing the CloudFlare error due to the massive
| outage. Seems their outages are getting more frequent and taking
| down the internet in new ways each time.
| archerx wrote:
| I have two projects on DO using droplets and they are still
| running fine.
| hshdhdhj4444 wrote:
| It seems mostly a CludFlare related issue.
|
| My DOs are working fine as well.
| sgc wrote:
| Are you using their "reserved IPs"? I was thinking of
| starting to use them, but now I wonder if it is part of their
| load balancing stack under the hood.
| soheilpro wrote:
| Droplets are fine.
|
| > _This incident affects: API, App Platform (Global), Load
| Balancers (Global), and Spaces (Global)._
| giancarlostoro wrote:
| So yesterday Azure got hit hard, today CF and DO are down, bad
| week or something else?
| zx8080 wrote:
| Year-end promotion cycle is the worst time for end-users and
| the best one for engineers greedy for promotions.
| Lammy wrote:
| Don't blame individual engineers who want to do what will be
| rewarded instead of company performance policies that reward
| this type of behavior.
| fridder wrote:
| shoot, there are also end of year layoffs and reorgs to
| pump up those year end numbers
| red-iron-pine wrote:
| what engineers, mate? they AI now
|
| and they're doing just spectacular
| BubbleRings wrote:
| I would also like to know people's opinion on this.
| matt-p wrote:
| DDOS, but I don't really understand why in particular.
| giancarlostoro wrote:
| Having known people like this, its either flexing about who
| has the more powerful botnet or advertising who can do what.
| red-iron-pine wrote:
| NATO testing internal infra, or Russian hackers stepping it
| up after aggressive sabotage efforts in Eastern Europe?
| watermelon0 wrote:
| Azure DDoS event happened in October. Blog post about the
| attack was published yesterday, and was quickly picked up by
| news sites.
| showerst wrote:
| I use DO's load balancers in a couple of projects, and they don't
| list Cloudflare as an upstream dependency anywhere that I've
| seen. It's so frustrating to think you're clear of a service then
| find out that you're actually in their blast radius too through
| no fault of your own.
| wesammikhail wrote:
| slight off topic: I used DO LBs for a little while but found
| myself moving away from that toward a small droplet with
| haproxy or nginx setup. Worked much better for me personally!
| showerst wrote:
| The point of an LB for these projects is to get away from a
| single point of failure, and I find configuring HA and
| setting up the networking and everything to be a pain point.
|
| These are all low-traffic projects so it's more cost
| effective to just throw on the smallest LB than spend the
| time setting it up myself.
| grayhatter wrote:
| If they are small projects, why are they behind a load
| balancer to begin with?
| nickmonad wrote:
| Usually because of SSL termination. It's generally
| "easier" to just let DO manage getting the cert
| installed. Of course, there are tradeoffs.
| showerst wrote:
| I use the LB's for high availability rather than needing
| load balancing. The LB + 2 web back-ends + Managed DB
| means a project is resilient to a single server failing,
| for relatively low devops effort and around $75/mo.
| grayhatter wrote:
| Are both servers deployed from the exact same
| repo/scripts? Or are they meaningful different, and/or
| balanced across multiple data centers?
|
| Did your high availability system survive this outage?
| showerst wrote:
| I have a couple of instances of this same pattern for
| various things that have been running for 5+ years, none
| of them have suffered downtime caused by the
| infrastructure. I use ansible scripts for the web
| servers, and the DO API or dashboard to provision the
| Load Balancer and Database. You can get it all hooked up
| in a half hour, and it really doesn't take any
| maintenance other than setting up good practices for
| rotating the web servers out for updates.
|
| They wouldn't survive DO losing a DC, they're not so
| mission critical that it's worth the extra complexity to
| do that, and I don't recall DO losing a DC in the past 10
| years or so.
|
| They did stay up during this outage, which was apparently
| mostly concentrated on a different product called the
| 'global load balancer', which ironically is exactly the
| extra complexity I mentioned to in theory survive a DC
| outage.
|
| Keep in mind these are "important" in the sense that they
| justify $100/mo on infra and monitoring, but not "life
| critical" in that an outage is gonna kill somebody or
| cost millions of bucks an hour. Once your traffic gets
| past a certain threshold, DO's costs don't scale that
| well and you're better off on a large distributed self-
| managed setup on Hetzner or buying into a stack like AWS.
|
| To me their LB and DB products hit a real sweet spot --
| better reliability than one box, and meaningfully less
| work than setting up a cluster with floating IP and
| heartbeats and all that for a very minimal price
| difference.
| miken123 wrote:
| It is mentioned in their list of subprocessors:
| https://www.digitalocean.com/trust/subprocessors
| bbss wrote:
| Regional LBs do not have Cloudflare as an upstream dependency.
| coreylane wrote:
| I find stuff like this all the time, railway.com recently
| launched an object storage service, but it's simply a wrapper
| for wasabi buckets under the hood, and they don't mention this
| anywhere... not even the subprocessors page
| https://railway.com/legal/subprocessors - customers have no
| idea they are using wasabi storage buckets unless they dig
| around the dns records. so i have to do all this research to
| find upstream dependencies and go subscribe to
| status.wasabi.com alerts etc.
|
| dig b1.eu-central-1.storage.railway.app +short
|
| s3.eu-central-1.wasabisys.com.
|
| eu-central-1.wasabisys.com.
| drob518 wrote:
| Dominos falling into dominos falling into dominos...
| aforty wrote:
| Cloudflare outage.
| igtztorrero wrote:
| I knew it, DigitalOcean CDN is using Cloudflare behind the
| scenes. Why DO ?
| mrkramer wrote:
| Who is next?
| red-iron-pine wrote:
| my guesses would be look at who has a FedRAMP capable service
| first.
|
| maybe also GCP, hetzner, akamai
| foxyv wrote:
| Man, it really seems like the cloud providers are having some
| tough times lately. Azure, AWS, and Cloudflare! Is everything
| just secretly AWS?
___________________________________________________________________
(page generated 2025-11-18 23:02 UTC)