[HN Gopher] Cloudflare Global Network experiencing issues
___________________________________________________________________
Cloudflare Global Network experiencing issues
Author : imdsm
Score : 2268 points
Date : 2025-11-18 11:35 UTC (11 hours ago)
(HTM) web link (www.cloudflarestatus.com)
(TXT) w3m dump (www.cloudflarestatus.com)
| imdsm wrote:
| Related to Azure DDoS?
|
| https://news.ycombinator.com/item?id=45955900
| alexdns wrote:
| whats your chain of thought here ? a company that has nothing
| to do with azure is down because azure got ddosed 2 weeks ago ?
| Jordan-117 wrote:
| Maybe that any actor sophisticated enough to take down Azure
| might also target Cloudflare?
| hirako2000 wrote:
| Especially when the next happens to be a major DDOS
| mitigator.
| polisaez wrote:
| Maybe related to their scheduled maintenance?
| https://www.cloudflarestatus.com/
| chedabob wrote:
| I thought that as we're seeing issues with LON, but their
| Manchester POP is also down and that didn't have any
| maintenance this morning.
| pstation wrote:
| Ironically I can't even read the link in the article because
| cloudflare is down.
| watermelon0 wrote:
| Linked Microsoft blog article mentions that DDoS was in
| October.
| its_notjack wrote:
| Ironically, DownDetector seems to be down because it protects its
| site with Cloudflare Turnstile... which is also down!
| imdsm wrote:
| I noticed this too!
| thrdbndndn wrote:
| The report there for AWS also skyrocketed, but I guess it's
| probably false positives?
| Macha wrote:
| Even many non tech people have begun to associate Internet
| wide outages with "aws must be down" so I imagine many of
| them searching "is aws down" and for down detector, a hit is
| a down report, so it will report aws impacts even when the
| culprit is cloudflare in this case
| mailund wrote:
| interesting, maybe "AWS is down" will become the new "the
| server is down" that some non-tech people throw around when
| anything unexpected happen on their computer?
| imdsm wrote:
| Internet is down, I guess I'll just look out of the window for a
| bit
| break_the_bank wrote:
| Ands its back. It was a very short window to look out of the
| window.
| porridgeraisin wrote:
| Not too much window to see anyway
| Filligree wrote:
| It's the first time it's been down that I've seen a sensible
| error message.
|
| But I was supposed to be commuting, so I guess I'll do that.
| ndsipa_pomu wrote:
| I was trying to look up banana-based jokes
| (https://upjoke.com/banana-jokes) and discovered that London
| Cloudflare seems to be down.
|
| Then, I tried various down detecting sites and they didn't seem
| to work either - presumably due to Cloudflare.
| bagpuss wrote:
| it's a slippery slope alright
| hirako2000 wrote:
| Not affected using tunnels, CDNs.
|
| it's probably related to the recent ddos attacks they helped
| mitigating.
| pavo-etc wrote:
| Saw cloudflare go down before my very eyes on colonist.io in
| Australia
| itzjacki wrote:
| A colleague of mine just came bursting through my office door in
| a panic, thinking he brought our site down since this happened
| just as he made some changes to our Cloudflare config. He was
| pretty relieved to see this post.
| Freak_NL wrote:
| Is there a word for that feeling of relief when someone else
| fucked up after initially thinking it was you?
| StanAngeloff wrote:
| Schadenfreude
| simonklitj wrote:
| Not quite, that's more like taking pleasure in the
| misfortune of someone else. It's close, but the specific
| relief bit that it is not _your_ misfortune is not captured
| skottenborg wrote:
| Internettet er vist ikke sa stort :-)
| simonklitj wrote:
| Faetter! Hvor genialt at se dig her. :)
| gnfargbl wrote:
| Nah, that's delight in someone else's misfortune. This is
| delight that the misfortune wasn't yours, which is slightly
| different.
| StanAngeloff wrote:
| 4 years of German and I still don't quite "get" it :^)
| TY!
| tagyro wrote:
| We have a saying:
|
| You know how you measure eternity?
|
| When you finish learning German.
| namblooc wrote:
| Katastrophenverursachererleichterung
| LinXitoW wrote:
| Katastrophenverursacherverlagerungserleichterung
| namblooc wrote:
| Even better
| jpmonette wrote:
| phewphoria
| Freak_NL wrote:
| Yup, that works.
| Sholmesy wrote:
| Well, at least something good came out of this incident.
|
| Perfect.
| zzzeek wrote:
| it has to sound like a german word though
| puilp0502 wrote:
| Is there a word for a feeling that there's gotta be a
| German word for this niche feeling?
| patneedham wrote:
| Probably Deutschwortsehnsucht
| (https://www.iamexpat.de/education/education-news/german-
| word...)
| usrusr wrote:
| You mean like when the Wortzusammensetzungsverdacht just
| hits you? (yeah, I just made that up, that's the beauty)
| datenhorst wrote:
| puhphorie
| chrisweekly wrote:
| Entlastungvergnugen?
| Shugyousha wrote:
| Fremdverfehlungserleichterung?
| shortrounddev2 wrote:
| When I'm debugging something, I'm not usually looking for the
| solution to the problem; I'm looking for sufficient evidence
| that I didn't cause the problem. Once I have that, the
| velocity at which I work slows down
| sys_64738 wrote:
| My manager once asked if he could have a "quick word". I
| said "velocity".
| cromka wrote:
| vindication?
| jspash wrote:
| The problem is, I still get the wrong end of the stick when
| AWS or CF go down! Management doesn't care, understandably.
| They just want the money to keep coming in. It's hard to
| convince them that this is a pretty big problem. The only
| thing that will calm them down a bit is to tell them Twitter
| is also down. If that doesn't get them, I say ChatGPT is also
| down. Now NOBODY will get any work done! lol.
| graemep wrote:
| Who decided to go with AWS of CF? If its a management
| decision tell them you need the resources to have a
| fallback if they want their system to be more reliable than
| AWS or CF.
| adriand wrote:
| Haha yeah I just got off the phone and I said, look, either
| this gets fixed soon or there's going to be news headlines
| with photographs of giant queues of people milling around
| in airports.
| aurareturn wrote:
| Every time a major cloud goes down, management tells us why
| don't we have a backup service that we can switch to. Then
| I tell them that a bunch of services worth a lot more than
| us are also down. Do you really want to spend the insane
| amount of resources to make sure our service stays up when
| the global internet is down?
| ec109685 wrote:
| Having an alt to Cloudflare isn't preposterous.
| hylaride wrote:
| This is why you ALWAYS have a proposal ready. I literally
| had my ass saved by having tickets with
| reliability/redundancy work clearly laid out with comments
| by out of touch product/people managers deprioritizing the
| work after attempts to pull it off the backlog (in one
| infamous case for a notoriously poorly conceived and
| expensive failure of a project that haunted us again with
| lost opportunity cost).
|
| The hilarious part of the whole story is that the same PMs
| and product managers were (and I cannot overemphasize this
| enough) absolutely militant orthodox agile practitioners
| with jira.
| mcphage wrote:
| Maybe this isn't great, but I get a hint of that feeling when
| I'm on an airplane and hear a baby crying. For a number of
| years, if I heard a baby crying, it was probably my baby and
| I had to deal with it. But now my kids are past that phase,
| so when I hear the crying, after that initial jolt of panic I
| realize that it isn't my problem, and that does give me the
| warm fuzzies. Even though I _do_ feel bad for the baby and
| their parents.
| adriand wrote:
| Related situation: you're at a family gathering and
| everyone has young kids running around. You hear a thump,
| and then some kid starts screaming. Conversation stops and
| every parent keenly listens to the screams to try and
| figure out whose kid just got hurt, then some other parent
| jumps up - it's not your kid! #phewphoria
| RhysU wrote:
| You're not alone in this feeling. I occasionally smile when
| it's not my kid.
| hackeraccount wrote:
| This is one of the secret joys of being a parent.
| spamizbad wrote:
| What's funny is as I get older this feeling of relief turns
| more like a feeling of dread. The nice thing about problems
| that you cause is that you have considerable autonomy to fix
| them. Cloudflare goes down you're sitting and waiting for a 3
| party to fix something.
| mewpmewp2 wrote:
| Why is it dread? I always feel good when big players mess
| up, as it makes me feel better about my own mess ups in
| life previously.
| kasey_junk wrote:
| Because my customers don't (and shouldn't care) it's a
| third party. If I caused it there is a chance I can fix
| it.
| mewpmewp2 wrote:
| If customers expected third party downtime to not affect
| their thing then you shouldn't have picked a third party
| provider or spent extra resources on not having a single
| point of failure? If they were happy with choosing the
| third party with knowledge of depending on said third
| party provider, then it was an accepted risk.
| _factor wrote:
| So you would rather be incompetent than powerless? Choice
| of third party vendor on client facing services is still
| on you, so maybe you prefer your incompetence be more
| direct and tangible?
|
| Even still, you should have policies in place to mitigate
| such eventualities, that way you can focus the
| incompetence into systematic issues instead. The larger
| the company, the less acceptable these failures become.
| Lessons learned is a better excuse for a shake and break
| startup than an established player that can pay to be
| secure.
|
| At some point, the finger has to be pointed. Personally,
| I don't dread it pointing elsewhere. Just means I've done
| my due D and C.
| shufflerofrocks wrote:
| Your priority (in this comment atleast) is about the
| finger-pointing, while the parent's priority is wanting a
| fix to the issue at hand.
| twodave wrote:
| Can't speak for GP but ultimately I'd rather it be my
| fault or my company's fault so I have something I can
| directly do for my customers who can't use our software.
| The sense of dread isn't about failure but feeling
| empathy for others who might not make payroll on time or
| whatever because my service that they rely on is down.
| And the second order effects, like some employee of a
| customer being unable to make rent or be forced to take
| out a short term loan or whatever. The fallout from
| something like this can have an unexpected human cost at
| times. Thankfully it's Tuesday, not a critical payroll
| day for most employees.
| mewpmewp2 wrote:
| But why does this case specifically matter? What if their
| system was down due to their WiFi or other layers beyond
| your software? Would you feel the same as well?
|
| What about all the other systems and people suffering
| elsewhere in the World?
| twodave wrote:
| I don't understand what point you're trying to make. Are
| you suggesting that if I can't feel empathy for everybody
| at once, or in every one of their circumstances, that I
| should not feel anything at all for anyone? That's not
| how anything works. Life (or, as I believe, God) brings
| us into contact with all kinds of people experiencing
| different levels of joy and pain. It's natural to
| empathize with the people you're around, whatever they're
| feeling. Don't over-complicate it.
| sys_64738 wrote:
| When others cause problems then you can put your feet up
| and surf the web waiting for resolution. Oh, wait.
| stonecharioteer wrote:
| It's close enough to Schadenfreude but not really.
| bookofjoe wrote:
| The German word "schadenfreude" means taking pleasure in
| someone else's misfortune; enjoyment rather than relief.
| bryanrasmussen wrote:
| since schaden is damage and freude is joy, not sure what it
| should be - maybe Schadeleichtig hmm...
| tauchunfall wrote:
| >maybe Schadeleichtig
|
| Maybe "Erleichterung" (relief)? But as a German
| "Schadenserleichterung" (also: notice the "s" between
| both compound word parts) rather sounds like a reduction
| of damage (since "Erleichterung" also means mitigation or
| alleviation).
| Rooster61 wrote:
| Schadenfriend?
|
| You gain relief, but you don't exactly derive pleasure as
| it's someone you know that's getting the ass end of the deal
| hoistbypetard wrote:
| schadenfuckup
| NitpickLawyer wrote:
| You missed a great opportunity to dead-pan him with something
| like "No, Bob, not just our site, you brought down the entire
| Internet, look at this post!"
| sakisv wrote:
| Well, you can never be sure that he didn't:
|
| https://www.fastly.com/blog/summary-of-june-8-outage
| itzjacki wrote:
| Oh don't you worry. We are very much talking about the global
| outage as if he was the root cause. Like good colleagues :)
| rapnie wrote:
| Hmm, wait a minute.. maybe he _was_ the cause! (no,
| kidding. just upping the pressure as a good peer :)
| bryanrasmussen wrote:
| are we truly good if we don't start a class action suit
| against this hapless scapegoat?!
| conorcleary wrote:
| Just join the one we've started over in this cubicle!
| srmarm wrote:
| > May 12, we began a software deployment that introduced a
| bug that could be triggered by a specific customer
| configuration under specific circumstances.
|
| I'd love to know more about what those specific circumstances
| were!
| CableNinja wrote:
| Damn, imagine being the customer responsible for that, oof
| whstl wrote:
| I would be fine if it was my "fault", but I'm sure people
| in business would find a way to make me suffer.
|
| But on a personal level, this is like ordering something at
| a restaurant and the cook burning the kitchen because they
| forgot to take out your pizza out of the oven or something.
|
| I would be telling it to everyone over beers (but not my
| boss).
| sakisv wrote:
| I would be tempted to put it on my CV :D
| WJW wrote:
| What do you imagine would be the result if you brought down
| cloudflare with a legitimate config update (ie not
| specifically crafted to trigger known bugs) while not even
| working for them? If I were the customer "responsible" for
| this outage, I'd just be annoyed that their software is
| apparently so fragile.
| Bloomy22 wrote:
| I'm pretty sure I crashed Gmail using something weird in its
| filters. It was a few years ago. Every time I did something
| specific (I don't remember what), it would freeze and then
| display a 502 error for a while.
| nevf1 wrote:
| It's also what was the cause of the Azure Front Doors global
| outage two weeks ago - https://aka.ms/air/YKYN-BWZ
|
| "A specific sequence of customer configuration changes,
| performed across two different control plane build versions,
| resulted in incompatible customer configuration metadata
| being generated. These customer configuration changes
| themselves were valid and non-malicious - however they
| produced metadata that, when deployed to edge site servers,
| exposed a latent bug in the data plane. This incompatibility
| triggered a crash during asynchronous processing within the
| data plane service. This defect escaped detection due to a
| gap in our pre-production validation, since not all features
| are validated across different control plane build versions."
| belter wrote:
| Wait for the post mortem ... It is a technical possibility,
| race condition propagates one customer config to all nodes...
| :-)
| bamboozled wrote:
| How do we know your colleagues changes didn't take down
| Cloudflare though?
| itzjacki wrote:
| Good point. We should probably assume they did, until proven
| otherwise.
| puilp0502 wrote:
| Guilty until proven innocent.
| disconnection wrote:
| Good that he worked it out so quick. I recently spent a day
| debugging email problems on Railway PaaS, because they silently
| closed an SMTP port without telling anyone.
| nrhrjrjrjtntbt wrote:
| The company where this colleague works? Cloudflare.
| dcjdfvk wrote:
| Even pornhub is down becuase it uses clouflare.
| carlos_rpn wrote:
| Is Cloudflare being down the work of conservative hackers and
| the rest of the internet is just collateral damage?
| arbuge wrote:
| Tell him it's worse than he thinks. He obviously brought the
| entire Cloudflare system down.
| mlrtime wrote:
| You joke and I think its funny, but as a junior engineer I
| would be quite proud if some small change I made was able to
| take down the mighty Cloudflare.
| throwup238 wrote:
| If I were Cloudflare it would mean an immediate job offer
| well above market. That junior engineer is either a genius
| or so lucky that they must be bred by Pierson's Puppeteers
| or such a perfect manifestation of a human fuzzer that
| their skills must be utilized.
| ethmarks wrote:
| A Ringworld reference in the wild?
| zarathustreal wrote:
| Internet points demand obscure references these days. My
| system prompt has its own area code
| throwup238 wrote:
| I never thought I'd get the chance, but then my Claude
| Code on Web credits ran out and I had to find another way
| to entertain myself.
| UltraSane wrote:
| Even after 20 projects I have only used $60 of my $250
| anvuong wrote:
| This reminds of a friend I had in college. We were
| assigned to the same group coding an advanced calculator
| in C. This guy didn't know anything about programming (he
| was mostly focused on his side biz of selling collector
| sneakers), so we assigned him to do all the testing, his
| job was to come up with weird equations and weird but
| valid way to present them to the calculator. And this
| dude somehow managed to crash almost all of our
| iterations except the few last ones. Really put the joke
| about a programmer, a tester, and a customer walk into a
| bar into perspective.
| jrochkind1 wrote:
| I love that he ended up making a very valuable
| contribution despite not knowing how to program -- other
| groups would have just been mad at him, had him do
| nothing, or had him do programming and gotten mad when it
| was crap or not finished.
| aws_ls wrote:
| Well its easy to cause damage by messing up the `rm`
| command, esp with `-fr` options. So don't take it as a
| proxy for some great skill which is required to cause
| damage.
| ethmarks wrote:
| You could easily cause great damage to _your_ Cloudflare
| setup, but CF has measures to prevent random customers
| deleting stuff from taking down the entire service
| globally. Unless you have admin access to the entire CF
| system, you can 't really cause much damage with rm.
| zidad wrote:
| You might want to consider migrating to Azure Front Door if
| that's a feature you like:
| https://www.infoq.com/news/2025/11/azure-afd-control-
| plane-f...
| methyl wrote:
| I kind of did that back in the days when they released
| Worker KV, I tried to bulk upload a lot of data and it
| brought the whole service down, can confirm I was proud :D
| amalcon wrote:
| It's also not _exactly_ the least common way that this sort
| of huge multi-tenant service goes down. It 's only as rare
| as it is because more or less all of them have had such
| outages in the past and built generic defenses (e.g.
| automated testing of customer changes, gradual rollout,
| automatic rollback, there are others but those are the ones
| that don't require any further explanation).
| 0xblinq wrote:
| Do you guys work at Cloudflare? Do you mind reverting that
| change just in case?
| 0xblinq wrote:
| Plot twist: They work at Cloudflare
| ants_everywhere wrote:
| Chances are still good that somewhere within Cloudflare someone
| really did do a global configuration push that brought down the
| internet.
|
| When aliens study humans from this period, their book of fairy
| tales will include several where a terrible evil was triggered
| by a config push.
| sefke wrote:
| I woke up getting bombarded by multiple clients messages of
| sites not working, I shitted my pants because I've changed the
| config just yesterday. When I saw the status message
| "cloudflare down" I was so relieved.
| raxxorraxor wrote:
| Did your colleague perhaps change the Cloudflare config again
| right now? Seems to be down again.
| theoldgreybeard wrote:
| You should tell him his config change took down half the
| internet.
| pySilver wrote:
| Our company is loosing money with every second of downtime.
| Vista2003 wrote:
| Unfortunately downtime is just the cost of doing business.
| Everyone else using Cloudflare is in the same boat.
| vldszn wrote:
| Also seeing this on my websites hosted on cloudflare
| surrTurr wrote:
| x is also not working properly
| Folyd wrote:
| Insane, my website https://geddle.com totally down
| dutchbrit wrote:
| seems to work here
| Folyd wrote:
| down again, all my website rely on Cloudflare DNS are down
| herbst wrote:
| Some of my websites are down. Says it's the cloudflare network,
| when I click it it says generic things about my server likely
| being the issue.
| MrKh4n wrote:
| Yes, most of things are down, recapthca, dns routing, proxy,
| etc.
| j0nimost wrote:
| all of my websites are down
| skrebbel wrote:
| yep
| break_the_bank wrote:
| Update: It seems to be back, the downtime lasted maybe 5-6
| minutes
| Batman8675309 wrote:
| Still down for me.
| its_notjack wrote:
| Looks like it's going up and down intermittently, maybe
| something is only half-rolled-out.
| ndsipa_pomu wrote:
| I'm still seeing some sites not working - ironically they're
| down detectors:
|
| https://statusfield.com/status/cloudflare
| https://statusgator.com/services/cloudflare
| Sammi wrote:
| It was down, then up, then back down for me.
|
| EDIT: And it's back up.
|
| EDIT EDIT: And it's back down lol
| techguy1954 wrote:
| It came back up for me, now it's down again. EDIT: I am in the
| UK, using the London Cloudflare server.
| break_the_bank wrote:
| Spoke too soon. The "internet" is down again.
| astonfred wrote:
| Down again in London
| robotfelix wrote:
| Back up for me now
| fspoettel wrote:
| Down again
| robotfelix wrote:
| Yep - down again for me too!
| robotfelix wrote:
| Back up for me now
|
| Edit: and then back down again
| Genego wrote:
| Yes (Asia)
| ilaksh wrote:
| When this kind of thing happens it makes me feel better about my
| own programming problems.
|
| I wonder if it has anything to do with the replicate.com
| purchase? Probably not.
| bratao wrote:
| The danger of Internet centralization in Cloudflare
| snitty wrote:
| That's why I run my server on 7100 chips made for me by Sam
| Zeloof in his garage on a software stack hand coded by me, on
| copper I ran personally to everyone's house.
| cuu508 wrote:
| You are joking but working on making decentralization more
| viable would indeed be more healthy than throwing hands up
| and accepting Cloudflare as the only option.
| A_Venom_Roll wrote:
| Looks like it. Even sites like isup.me seem to be down, lots of
| cloudflare error messages across the net
| cbility wrote:
| Getting a 500 error from cloudflare in Manchester
| mittermayr wrote:
| Funny how I trusted Cloudflare first and started looking at
| restarting my servers, only to realize it's not me this time :)
| herbst wrote:
| The error even kinda says that. Still assumed it's me
| r721 wrote:
| Another thread: https://news.ycombinator.com/item?id=45963781
|
| UPD And https://news.ycombinator.com/item?id=45963949
| sammy2255 wrote:
| Turnstile is throwing 500 internal server error
| re-thc wrote:
| Downflare
| js4ever wrote:
| Cloudflare fully down for me and my team, half of internet just
| vanished
| teitoklien wrote:
| Same we use cloudflare as image cdn and its R2 service a lot,
| cant access anything over its cdn route
| mrintegrity wrote:
| It's insane to me that big internet uptime monitoring tools like
| Pingdom and Downdetector both seem to rely on Cloudflare, as both
| of those are currently unavailable as well.
| stupidityonmaxx wrote:
| down detector works in serbia
| denuoweb wrote:
| Browser Working, Portland Cloudflare Error, apps.ideal-logic.com
| Host Working
| re-thc wrote:
| Downflare
| DC-3 wrote:
| Cloudflare'nt
| Terr_ wrote:
| I'm thinking about all those quips from a few decades back, along
| the lines of: "The Internet is resilient, it's distributed and it
| routes around damage" etc.
|
| In many ways it's still true, but it doesn't feel like a given
| anymore.
| zx76 wrote:
| Unfortunately it seems like it, our service has lost a portion of
| our Cloudflare connectivity. We use their tunnels functionality.
|
| Additionally, it looks like Pingdom/Solarwinds authentication is
| affected too - not a great look for a service in that category.
| StanAngeloff wrote:
| No logging in to Cloudflare Dash, no passing Turnstile (their
| CAPTCHA Replacement Solution) on third-party websites not proxied
| by Cloudflare, the rest that are proxied throwing 500 Internal
| server error saying it's Cloudflare's fault...
|
| Feels like half the internet is down.
| woodylondon wrote:
| my cloudflare pages website is down - 500 server error :(
|
| cannot login to get to workers to check - auth errors
|
| I thought this was the point of a cached CDN!
| Mojah wrote:
| Interestingly, also noticing that websites that use Cloudflare
| Challenge (aka "I'm not a Robot") are also throwing exceptions
| with a message as "Please unblock challenges.cloudflare.com to
| proceed" - even though it's just responding with an HTTP/500.
| wslh wrote:
| Not only discriminating robots but actual people /s.
| davidee wrote:
| And others (ex. pinkbike) displaying "you have been blocked".
| comprev wrote:
| Always nice to see Pinkbike mentioned in the tech world :)
| csomar wrote:
| I think the site (front-end) thinks you have blocked the domain
| through DNS or an extension; and thus suggests you unblock it.
| It is unthinkable that Cloudflare captchas could go down /s.
| cube00 wrote:
| The state of error handling in general is woeful, they do
| anything to avoid admitting they're at fault so the negative
| screenshots don't end up on social media.
|
| Blame the user or just leave them at an infinite spinning
| circle of death.
|
| I check the network tab and find the backend is actually
| returning a reasonable error but the frontend just hides it.
|
| Most recent one was a form saying my email was already in use,
| when the actual backend error returned was that the password
| was too long.
| littlecranky67 wrote:
| This takes down AI/search on chat.bing.com (GPT5,
| unauthenticated).
|
| Funny, since I would have to prove to a an AI that I am human
| in the first place.
| gwd wrote:
| I do appreciate the visual "mea culpa":
|
| Your browser: Working
|
| Host: Working
|
| Cloudflare: Error
| mallets wrote:
| Might be the first time I have ever seen that. Though in my
| case the "Host" is Cloudflare's own Pages service.
| udev4096 wrote:
| Yeah, I was shocked. Disbelief that the host was up, which is
| what usually happens when the cloudflare's page show up
| janmo wrote:
| And at the bottom:
|
| What can I do?
|
| Please try again in a few minutes.
| mig4ng wrote:
| That is really good to be honest!
|
| I have Cloudflare running in production and it is affecting us
| right now. But at least I know what is going on and how I can
| mitigate (e.g. disable Cloudflare as a proxy if it keeps
| affecting our services at skeeled).
| Lukas_Skywalker wrote:
| They still blame the customers when you click on "Cloudflare":
|
| > If the problem isn't resolved in the next few minutes, it's
| most likely an issue with the web server you were trying to
| reach.
| mig4ng wrote:
| In terms of probability looking at the history, it is
| correct. It's mostly me messing up with the web server.
| jap wrote:
| That page has special if/endif HTML comments to handle if your
| browser is IE 6, IE 7, IE 8...
| Neil44 wrote:
| I noticed that refreshing honesty too, not that the users did
| (our wifi is down fix it pls urgent)
| ponytech wrote:
| I searched my logs for errors for about an hour before figuring
| out the problem was not on my server :D
| WhereIsTheTruth wrote:
| West's Great Firewall
|
| /s
| pelagicAustral wrote:
| Yeah, dammit! NOT TODAY!!!
| alexcroox wrote:
| It's been 15 minutes of it going up and down, still nothing on
| their status page...
| imdsm wrote:
| Well, we've seen it now, they'll have to update it eventually!
| timpera wrote:
| They've just added it less than a minute ago. I expected a
| little more responsiveness from Cloudflare...
| arkx wrote:
| Me too. What good is a status page that's not automated?
| timpera wrote:
| Especially when it's the de facto status page for 20% of
| the internet!
| ArcHound wrote:
| I can answer that - once the lawyers take interest in your
| SLAs, you need to check with them if this is really an
| incident. Otherwise, you might lose some contract money and
| nobody wants that.
| rco8786 wrote:
| There are no truly automated status pages. It's an
| impossible problem. I mean that seriously. At scale you're
| collecting 100s of thousands (or mms) of metrics/spans/logs
| across 10s or 100s of loosely coupled systems. Building a
| system that can accurately analyze these and assess what
| the status page should say, in real time, without human
| intervention, is just not possible with current technology.
|
| Even just the basic question of "are we down or is our
| monitoring system just having issues" requires a human. And
| it's never "are we down", because these are distributed
| systems we're talking about.
|
| If service X goes down entirely, does that warrant a status
| page update? Yes? Turns out system X is just running ML
| jobs in the background and has no customer impact.
|
| If service Z's p95 response latency jumps from 10ms to
| 1500ms for 5 minutes, 500s spike at the same time, but
| overall 200s rate is around 98%, are we down? is that a
| status page update? Is that 1 bad actor trying to cause
| issues? Is that indicative of 2,000 customers experiencing
| an outage and the other 98,000 operating normally? Is that
| a bad rack switch that's causing a few random 500s across
| the whole customer base and the service will reject that
| node and auto-recover in a moment?
| rco8786 wrote:
| Everything else aside, 20 minutes to get their status page
| updated seems pretty damn fast.
| red_Seashell_32 wrote:
| Just were talking about how Replicate might have better
| availability due to joining Cloudflare, and they too went down...
| Oops.
| karimf wrote:
| I've been migrating all my personal stuff to Cloudflare. They
| have good products and good pricing.
|
| At the same time I'm worried about how the internet is becoming
| even more centralized, which goes against how it was originally
| designed.
| kilroy123 wrote:
| Same here. A lot of my sites are now down.
| Mike1122 wrote:
| All my websites are down due to Cloudflare
| polisaez wrote:
| From the Cloudflare status website: "Scheduled maintenance is
| currently in progress." Maybe something went wrong while doing
| maintenance?
| Fysi wrote:
| They consistently have scheduled maintenance.
| helloguillecl wrote:
| Yes. At least in Germany and Spain. Interment.
| davisonio wrote:
| Even downdetector is down, I can't get through the Cloudflate
| captcha.
| fny wrote:
| Ironically, cloudflare.com is not down.
| ed_mercer wrote:
| Yeah I wonder how that works
| Dilettante_ wrote:
| "Don't get high on your own supply"?
| davisonio wrote:
| Even downdetector is down, I can't get through the Cloudflate
| captcha.
| rvz wrote:
| Working just fine in the UAE.
| tokioyoyo wrote:
| I had two completely unrelated tabs open (https://twitter.com and
| https://onsensensei.com) both showing the same error. Opened
| another website, same error. Kinda funny to see how much of
| entire web is ran on CloudFlare nowadays.
| burtekd wrote:
| Love how everyone plays with redundancy - multiple hosts,
| balance loader, etc, and yet half of the web relies on single
| point of failure being CF
| 6031769 wrote:
| Indeed. And it feels really good knowing that our stuff isn't
| in that half.
| Dennip wrote:
| Still down for me
| lofaszvanitt wrote:
| So it begins. Now is the time to banish the evil presence from
| the internet. :D
| Ekaros wrote:
| I was joking that after AWS, Azure that the Cloudflare would be
| next one...
|
| So which large service we have left that could take chunk of
| internet out?
| sammy2255 wrote:
| Gcloud
| drcongo wrote:
| Came back up for a few minutes and has gone down again.
| https://www.cloudflarestatus.com has nothing.
| techguy1954 wrote:
| Keeps going up and down for me, I cant access DownDetector to
| check. The first website I noticed it on was Blender Artists.
| imik768 wrote:
| i am trying to this page https://cryptoquip.net/ but it is
| showing, Cloudflare Error
| lurkersince2013 wrote:
| https://www.cloudflarestatus.com - has been updated.
| entropoem wrote:
| Everyone laughs when AWS collapses, everyone is silent when
| Cloudflare collapses. Why? Because the place to laugh has
| collapsed.
| Hedepig wrote:
| Except on HN
|
| HAHA!
|
| Our servers are still down, though
| sutton12345 wrote:
| HAHAHAHA
| imdsm wrote:
| I'm not sure whether to laugh or cry
|
| Maybe I'll do both
| sutton12345 wrote:
| YEP that's the case nowwwww
| belter wrote:
| One is every seven years... the other one is a ...monthly
| event?:
| https://hn.algolia.com/?https://hn.algolia.com/?dateRange=al...
| ceejayoz wrote:
| Most of those aren't outages, and both providers have big
| blips.
|
| Globally meaningful outages of either are quite rare.
| mvdtnz wrote:
| This is at least the third time in the past year that
| Cloudflare has caused a significant outage of my product.
| 6031769 wrote:
| And you are still using them because ... ?
| mvdtnz wrote:
| We handle ~2M requests per second and CF eliminates about
| 2/3 of those. We need CF or something like it. Multi
| edge is harder than it sounds at very large scale.
| viraptor wrote:
| There are still alternatives like Bunny
| https://status.bunny.net/history (may not be for
| everyone, but I like to post the CF alternatives so it
| becomes ever so slightly less of a default)
| belter wrote:
| Its actually way more than monthly... It seems nobody does
| outages like Cloudflare...
|
| https://www.cloudflarestatus.com/history?page=8
|
| https://www.cloudflarestatus.com/history?page=7
|
| https://www.cloudflarestatus.com/history?page=6
|
| https://www.cloudflarestatus.com/history?page=5
|
| https://www.cloudflarestatus.com/history?page=4
|
| https://www.cloudflarestatus.com/history?page=3
|
| https://www.cloudflarestatus.com/history?page=2
|
| https://www.cloudflarestatus.com/history?page=1
| curious_curios wrote:
| I at least respect them for reporting them. It feels like
| lots of cloud providers don't, or begrudgingly.
| ceejayoz wrote:
| https://health.aws.amazon.com/health/status has regular
| disruptions like this, too (click the button labeled
| "list of events"; several a month even just in North
| America).
|
| All the major cloud providers have regular incidents.
| Most go unnoticed, because they're small or short.
|
| The really big AWS ones go on
| https://aws.amazon.com/premiumsupport/technology/pes/
| AnkerSkallebank wrote:
| Everyone is silent when Cloudflare collapses. Same goes for
| Azure, but that is because noone uses it.
| imdsm wrote:
| When Azure goes down: Oh well
|
| When Cloudflare goes down: Oh no
| exe34 wrote:
| Down detector broke.... :-D
| techguy1954 wrote:
| Yeah, how ironic. The site that is designed to tell you if
| something else is down, is currently down.
| dspillett wrote:
| Life imitates art, Red Dwarf ~1989:
| Lister: What's the damage Hol? Holly: I don't
| know. The damage report machine has been damaged.
| tfiskgul wrote:
| It also reminds me of this part of Hitchhikers Guide to
| the Galaxy
|
| https://www.penguinrandomhouse.ca/books/661/mostly-
| harmless-...
| pelagicAustral wrote:
| OMG! today of all days!
| nrhrjrjrjtntbt wrote:
| Black Tuesday
| tzpbingo wrote:
| Because those who were mocking him couldn't speak, X also
| crashed.
| timeon wrote:
| This would be true in the past but now most people are not on
| Twitter.
| JdeBP wrote:
| Sadly, I can report that this has brought down 2 of the major
| Mastodon nodes in the United Kingdom.
|
| Happily, the small ones that I also use are still going
| without anyone apparently even noticing. At least, the
| subject has yet to reach their local timelines at the time
| that I write this.
|
| 2 of the other major U.K. nodes are still up, too.
| weberer wrote:
| "Most people" were never on Twitter to begin with. However
| its number of monthly active users have only grown since
| 2020.
| timeon wrote:
| > However its number of monthly active users have only
| grown since 2020.
|
| Like everywhere it is mostly bots.
|
| Look at HN frontpage, there used to be 1-2 Twitter post per
| day. Now it is barely per week. End even those are usually
| just from two accounts (Karpathy and Carmack).
| r0fl wrote:
| Peter Levels wisdom about why to host not on aws not looking so
| wise right now
| basilgohar wrote:
| There are places other than AWS to host.
| matt-p wrote:
| Cloudflare down because of a DDOS is extremely funny.
| sammy2255 wrote:
| There's no evidence to suggest it was a result of a DDoS
| attack
| simondotau wrote:
| That's true, but there _is_ evidence that it would be
| extremely funny.
| matt-p wrote:
| There's no public evidence. Lets wait for the blog post.
| brazukadev wrote:
| > Everyone laughs when AWS collapses, everyone is silent when
| Cloudflare collapses
|
| Everyone laughs when Azure collapses too
| mlrtime wrote:
| Who is silent?
| yen223 wrote:
| It's down here in Sydney as well. The status page hasn't been
| updated to reflect that
| helloguillecl wrote:
| Funny that I could not load Twitter to see if Cloudflare was
| down.
|
| I rushed to Hacker News, but it was too early. Clicking on "new"
| did the job to find this post before making it to the Homepage:)
|
| The web is still alive!
| timeon wrote:
| It was on Mastodon. That one is hardly going down.
| ksec wrote:
| HN has become the place to check if any HyperScaler + Cloudflare
| is down.
| montekristooGDB wrote:
| while my colleagues are wondering why cloudlfare isn't working
| and are afraid it might be something from us locally, I'll
| first check here to make sure it's not a Cloudflare / AWS
| problem in the first place.
| timpera wrote:
| When was HN down for the last time? :)
| zwog wrote:
| I actually came here to check because
| downforeveryoneorjustme.com and downdetector are offline as
| well.
| mchinen wrote:
| Down, but the linked status page shows mostly operational, except
| for "Support Portal Availability Issues" and planned maintenance.
| Since it was linked, I'm curious if others see differently.
|
| edit: It now says "Cloudflare Global Network experiencing issues"
| but it took a while.
| erdaltoprak wrote:
| At some point we really need to think if this is the web we want,
| one/two major actors are down and everything goes with it
|
| Not downplaying the immense work of infra / engineering at this
| scale but my neighborhood local grocery market shouldn't be down
| eptcyka wrote:
| Decentralisation is at some point directly opposed to
| operational efficiency, when the sun is shining.
| starbugs wrote:
| And centralization is ineffective long term
| raxxorraxor wrote:
| A shark is an extremely energy efficient creature, but it is
| relatively stupid.
| Dilettante_ wrote:
| At some point every website will be the equivalent of
| Surstromming
| lofaszvanitt wrote:
| Why everyone needs to be behind Cloudflare. I don't think
| DDOSing sites out of whim is so rampant that everyone needs the
| virtual umbrella.
| frameset wrote:
| It actually is.
|
| I run a small video game forum with posts going back to 2008.
| We got absolutely smashed by bots scraping for training data
| for LLMs.
|
| So I put it behind Cloudflare and now it's down. Ho hum.
| shaky-carrousel wrote:
| It'd funny if these bots were run by Cloudflare.
| frameset wrote:
| Ha, yeah. They seemed to mostly be in SE Asia.
| stevepotter wrote:
| Can you please elaborate on "smashed"? I'm very interested
| frameset wrote:
| I took a screenshot of the graph in cloudflare when I
| switched on the bot challenges.
|
| https://i.ibb.co/qHCJyY7/image.png
|
| I wrote the below to explain to our users what was
| happening, so apologies if the language is too simple for
| a HN reader.
|
| - 0630, we switched our DNS to proxy through CF, starting
| the collection of data, and implemented basic bot
| protections
|
| - Unfortunately whatever anti-bot magic they have isn't
| quite having the effect, even after two hours.
|
| - 0830, I sign in and take a look at the analytics. It
| seems like <SITE NAME> is very popular in Vietnam,
| Brazil, and Indonesia.
|
| - 0845, I make it so users from those countries have to
| pass a CF "challenge". This is similar to a CAPTCHA, but
| CF try to make it so there's no "choosing all the cars in
| an image" if they can help it.
|
| - So far 0% of our Asian audience have passed a
| challenge.
| watermelon0 wrote:
| Have you tried Anubis or similar tools? I've had similar
| issues with bot scraping of a forum taking all server
| resources, and using PoW challenge solved the problem.
|
| https://github.com/TecharoHQ/anubis
| frameset wrote:
| I did! It's very cool tech. However for our config it was
| easier to slap CF in front of it.
|
| I will say one very appealing use of Anubis I'd love to
| try is using it as a Traefik middleware to protect
| services running in docker containers.
| trollbridge wrote:
| Same problem here. If I didn't use Cloudflare, nearly all
| of my traffic would be (apparently misconfigured) scraper
| bots.
| reassess_blind wrote:
| I've been DDoS'd countless times running a small scale,
| uncontroversial SaaS. Without them I would've had countless
| downtime periods with really no other way to mitigate.
| dukeyukey wrote:
| Good chance the reason DDOSing isn't so big anymore is
| _because_ everyone is on Cloudflare.
| xslvrxslwt wrote:
| No but because all of us were arrested in 2018 for running
| DDoS-4-hire services. Bypassing cloudflare is very easy and
| I still can fry any of your websites (if i wanted to, just
| like any other skid)
| grey413 wrote:
| It's the web-scrapers. I run a tiny little mom and pop
| website, and the bots were consistently using up all of my
| servers' resources. Cloudfare more or less instantly resolved
| it.
| jwr wrote:
| You mean you outsourced to Cloudflare the decision on who
| is allowed to view your website. That could be well-
| intentioned, but it's a risky thing to do, and I would not
| to outsource that decision. Especially as I wouldn't know
| who failed to get to my website as there is no way to
| appeal the decision.
|
| As a side note, what does your site do that it's possible
| to use up all server resources? Computers are stupid fast
| these days. I find it's really difficult to build something
| that doesn't scale to at least multiple hundreds of
| requests per second.
| grey413 wrote:
| You'd be amazed how easy it is to take down a janky
| decades old LAMP stack.
| dana321 wrote:
| Caching would have been the correct answer
| luckylion wrote:
| It's not super common, but common enough that I don't want to
| deal with it.
|
| The other part is just how convenient it is with CF. Easy to
| configure, plenty of power and cheap compared to the other
| big ones. If they made their dashboard and permission-system
| better (no easy way to tell what a token can do last I
| checked), I'd be even more of a fan.
|
| If Germany's Telekom was forced to peer on DE-CIX, I'd always
| use CF. Since they aren't and CF doesn't pay for peering,
| it's a hard choice for Germany but an easy one everywhere
| else.
| BoldColdHold wrote:
| DDOSing is absolutely so rampant that you need to be behind
| something.
| lofaszvanitt wrote:
| Hm, interesting times we live in.
| shaky-carrousel wrote:
| Nope, I'm at hetzner and haven't seen a DDoS in years.
| xslvrxslwt wrote:
| Because of 2018 operation "Power OFF" but it's still
| pretty easy to take anything down.
|
| Hetzner has the WEAKEST DDoS protection out of ANYTHING
| out there - Arbor sucks.
|
| Send me your website url and I'll keep it down for DAYS
| and whenever you cry to hetzner I'll just fry it again,
| it's that easy and that's why they're the cheapest -
| because everyone ran away from them back then.
| darkwater wrote:
| So, are you an Internet bully? how would you define
| yourself?
| xslvrxslwt wrote:
| Nah, I'm just talking about the possibilities
| input_sh wrote:
| So am I and neither did I... up until a week ago. Now my
| server's being hammered with bot traffic 24/7.
| Mordisquitos wrote:
| Analogously, arson attacks against businesses in Palermo
| are absolutely so rampant that they need to be protected by
| someone.
| timpera wrote:
| I run a few websites with moderate traffic (~900K daily
| page views total) on the same VPS and never had an issue
| with DDOS. Is this specific to some industries?
| xslvrxslwt wrote:
| Literally specific to "did I make this skid angry or
| not", it takes $5 to DDoS a website (bypassing cloudflare
| included)
| BoldColdHold wrote:
| Depends on what those websites are and how lucky you are.
| RKFADU_UOFCCLEL wrote:
| > Gooo gooo gaa gaaa look at this basic cassus beli I
| swallowed!!!
|
| And yet my website is still up today, and has not been down
| for years.
| chromehearts wrote:
| Cloudflare DDOS protection is super essential (especially for
| smaller businesses)
| Tanath wrote:
| DDoS prevention may be essential, but not CloudFlare.
| Mordisquitos wrote:
| Who is motivated to launch DDoS against smaller businesses?
| What do they have to gain?
| xslvrxslwt wrote:
| Anyone that has $5.
| Mordisquitos wrote:
| I've also got EUR5, but I see greater return on
| investment in spending them on a lottery ticket than in
| DDoS'ing arbitrary small businesses.
| xslvrxslwt wrote:
| I know, but people love the feel of "power", especially
| when it's cheap or even free
| the_bear wrote:
| My small SaaS app has been DDoSed a handful of times,
| always accompanied by an email asking for a ransom in the
| form of bitcoin.
|
| The first time we switched to Cloudflare which saved us.
| Even with Cloudflare, the DDoS attempts are still
| damaging (the site goes down, we use Cloudflare to block
| the endpoints they're targeting, they change endpoints,
| etc.) but manageable. Without Cloudflare or something
| like it, I think it's possible that we'd be out of
| business.
| hennell wrote:
| Honestly it kinda is. Ai bots scrape everything now, social
| media means you can go viral suddenly, or you make a post
| that angers someone and they launch an attack just because. I
| default to cloudflare, because like an umbrella I might just
| be carrying it around most of the time, but in the case of a
| sudden downpoor it's better than getting wet.
| isodev wrote:
| There are plenty of alternatives to protect against DDoSing,
| people like convenience though. "Nobody gets fired for
| choosing Microsoft/Cloudflare". We have a culture problem
| xslvrxslwt wrote:
| I was arrested by Interpol in 2018 because of warrants issued
| by the NCA, DOJ, FBI, J-CAT, and several other agencies, all
| due to my involvement in running a DDoS-for-hire website.
| Honestly, anyone can bypass Cloudflare, and anyone that want
| to take your website down - will take it down. It's just that
| luckily for all of us most of the DDoS-4-hire websites are
| down nowadays but there are still many botnets out there that
| will get past basically any protection and you can get access
| to them for basically $5.
| FridayoLeary wrote:
| One minute, what? Can you elaborate on that. I have loads
| of questions. What exactly were you doing? What
| consequences did you face? How come you are talking about
| it?
| xslvrxslwt wrote:
| because I'm from Serbia so I was released immediately
| instead of actually being jailed like my friend from
| Croatia ~
| olalonde wrote:
| > anyone can bypass Cloudflare
|
| How?
| q3k wrote:
| Plenty of ways to leak the original server IP address if
| it isn't really well hardened against that (and most
| aren't).
| sznio wrote:
| There's plenty of DDoS if you're dealing with people petty
| enough.
|
| The VPS I use will nuke your instance if you run a game
| server. Not due to resource usage, but because it attracts
| DDoS like nothing else. Ban a teen for being an asshole and
| expect your service to be down for a week. And there isn't
| really Cloudflare for independent game servers. There's Steam
| Networking but it requires the developer to support it and of
| course Steam.
|
| Valve's GDC talk about DDoS mitigation for games:
| https://youtu.be/2CQ1sxPppV4
| RKFADU_UOFCCLEL wrote:
| > And there isn't really Cloudflare for independent game
| servers
|
| And yet game servers still work fine. Which answers this
| subthread's question ("how likely is it to get DDoSed if
| you don't have Cloudflare"), answer: not very likely, it
| happens once in a while at most.
| finghin wrote:
| There's certainly a business case for "which nines" after the
| talk of n nines. You ideally want to be available when your
| competitor, for instance, is not.
| carlosjobim wrote:
| Think about this rationally. If Cloudflare doesn't fix it
| within reasonable time, you can just point to different name
| servers and have your problem fixed in minutes.
|
| So why be on Cloudflare to start with? Well, if you have a more
| reliable way then there's no reason. If you have a less
| reliable way, then you're on average better off with
| Cloudflare.
| erdaltoprak wrote:
| Well I can't change my NS since it's on Cloudflare too but
| besides that my personal opinion was not about this outage in
| particular but more the default approach of some websites
| that don't need all this tech (yes I really was out of
| groceries)
| carlosjobim wrote:
| Is Cloudflare your domain registrar? In that case, yes I
| think you should think about being less dependent on them.
|
| As for websites which don't need Cloudflare, in my
| experience almost every website will be DdoS attacked from
| time to time.
| erdaltoprak wrote:
| I am personally really happy with Cloudflare for domains,
| pages and dns, I don't run critical stuff but some
| websites are and they should not be lazy about it
| darkwater wrote:
| > Is Cloudflare your domain registrar? In that case, yes
| I think you should think about being less dependent on
| them.
|
| And why I should overthink my architecture now? If I had
| to manage redundant systems and keep track of circular
| dependencies I just could keep managing my infra the old
| way, no?
|
| I'm being sarcastic here, obviously, but really one of
| the selling point for cloud back in the day it was "you
| don't have to care about those details". You just need to
| care about _other_ details, now.
| Semaphor wrote:
| > in my experience almost every website will be DdoS
| attacked from time to time.
|
| The place I work at has been online since 1996, not even
| a DoS yet, let alone a DDoS. Though we now use CF to
| filter all that bot traffic.
| karimf wrote:
| It's hard not to use Cloudflare at least for me: good products,
| "free" for small projects, and if Cloudflare is down no one
| will blame you since the internet is down.
| graemep wrote:
| > if Cloudflare is down no one will blame you since the
| internet is down.
|
| That is true. it is also the problem. It means the biggest
| providers do not even need to bother to be reliable because
| everyone will use them anyway.
| reassess_blind wrote:
| Well, no. If they are unreliable to the point of being an
| outlier when compared to the alternatives then people will
| switch. At this stage they're not an outlier.
| antonyh wrote:
| Maybe not, but they are approaching it. I wouldn't use it
| for anything funded with my own cash, I no longer
| recommend it as a first choice, but I'm not suggesting it
| gets replaced yet. It's somewhat in the 'legacy tech'
| category now in terms of how I perceive it and deal with
| it.
| graemep wrote:
| They are often promoted as bing more reliable.
| timeon wrote:
| > if Cloudflare is down no one will blame you since the
| internet is down.
|
| But this is not really the case. When Azure/AWS were down,
| same as this one with Cloudflare: significant amount of web
| was down but most of it was not. It just makes more obvious
| which provider you use.
| Dilettante_ wrote:
| "Accountability Sinks"
|
| https://aworkinglibrary.com/writing/accountability-sinks
| saxenaabhi wrote:
| Setting up a replica and then pointing your api requests at it
| when cloudflare request fails is trivial. This way if you have
| a SPA and as long as your site/app is open the users won't
| notice.
|
| The issue is DNS since DNS propagation takes time. Does anyone
| have any ideas here?
| isodev wrote:
| Two domains for your api perhaps, a full blown SPA could try
| one and then the other.
| viraptor wrote:
| > Setting up a replica and then pointing your api requests at
| it when cloudflare request fails is trivial.
|
| Only if you're doing very basic proxy stuff. If you stack
| multiple features and maybe even start using workers, there
| may be no 1:1 alternatives to switch to. And definitely not
| trivially.
| tambre wrote:
| Owning your IP space and using Anycast.
| tempaccount420 wrote:
| Anycast is hard, which is why only clouds bother with it in
| the first place :(
| numpad0 wrote:
| BLOCKCHAINS! I mean, some sort of P2P hosting and/or node
| discovery would be nice.
| nuker wrote:
| > At some point we really need to think if this is the web we
| want,
|
| You think we have a say in this?
| louismerlin wrote:
| You have the power to not host your own infrastructure on aws
| and behind cloudflare, or in the case of an employer you have
| the power to fight against the voices arguing for the
| unsustainable status quo.
| reassess_blind wrote:
| If you need DDoS mitigation then you essentially need to
| rely on a third party. Every third party will have
| inevitable downtime. For many it's just whether you'd
| prefer to be down while everyone else is down or not.
| rwky wrote:
| A lot of business would accept the rare downtime from
| Cloudflare in exchange for the DDoS protection.
|
| If the internet was always a nice place we wouldn't need
| Cloudflare and similar :(
| quaintdev wrote:
| You also have the power to not pay money to big tech
| squigz wrote:
| The HN crowd in particular absolutely has a say in this,
| given the amount of engineering leads, managers, and even
| just regular programmers/admins/etc that frequent here - all
| of whom contribute to making these decisions.
| deadbabe wrote:
| Believe me it's what people want. The alternative is far worse.
| severino wrote:
| It's not the web _we_ want, but it 's the web corporations
| want. And everybody else doesn't give a damn.
| neop1x wrote:
| We? I am not using it. I never used it and I will not use it.
| People should learn how to work with firewall, setup a simple
| ModSecurity WAF and stop using this bullshit. Almost everything
| goes through cloudflare and cloudflare also does TLS fronting
| for websites so basically cloudflare is MITM spying proxy but
| no one seem to care. :/
| diimdeep wrote:
| 80% of web sites I visited in last 15 minutes are not available
| anymore, LOL
| Andugal wrote:
| OK, it seems to be working again.
| kaizenb wrote:
| Nope.
| Andugal wrote:
| Indeed, it worked for 2 minutes, but not anymore.
| kaizenb wrote:
| Yes.
| johnstonnorth wrote:
| https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
| palmeida wrote:
| Just now it was acknowledge in the status page.
| dazc wrote:
| Checks Cloudflare Status - yeah, everything's hunky dory bro.
| mittermayr wrote:
| I started restarting my own servers thinking something went awry
| again, that's how much I usually trust them not to be down.
| Interesting.
| mrintegrity wrote:
| Now is as good a time as ever to look at moving our eggs into
| some other baskets
| ilaksh wrote:
| Telnyx seems to be down for me. Actually I lied, I think it is
| working. at least call connected.
| spapas82 wrote:
| Cloudflare seems to have degrated performance. Half the requests
| for my site throw cloudflare 500x errors, the other half work
| fine.
|
| However the https://www.cloudflarestatus.com/ does not really
| mention anything relevant. What's the point of having a status
| page if it lies ?
|
| _Update_ Ah I just checked the status and now I get a big red
| warning (however the problem existed for like 15 minutes before
| 11:48 UTC):
|
| > Investigating - Cloudflare is aware of, and investigating an
| issue which potentially impacts multiple customers. Further
| detail will be provided as more information becomes available.
| Nov 18, 2025 - 11:48 UTC
| maushu wrote:
| > What's the point of having a status page if it lies ?
|
| Status pages are basically marketing crap right now. The same
| thing happened with Azure where it took at least 45 minutes to
| show any change. They can't be trusted.
| bilekas wrote:
| >However the https://www.cloudflarestatus.com/ does not really
| mention anything relevant. What's the point of having a status
| page if it lies ?
|
| What is the lie ?
|
| > Cloudflare Global Network experiencing issues
|
| cloudflare has a specific service names "Network" and it's
| having issues..
| spapas82 wrote:
| Please read my comment again including the update:
|
| For 15 minute cloudflare wasn't working and the status page
| did not mentioned anything. Yes, right now the status page
| mentions the serious network problem but for some time our
| pages were not working and we didn't know what was happening.
|
| So for ~ 15 minutes the status page lied. The whole point of
| a status page is to _not_ lie, i.e to be updated
| automatically when there are problem and not by a person that
| needs to get clearance on what and how to write.
| mrsuprawsm wrote:
| Seems like ChatGPT and Claude are also affected. (CLI Codex still
| seems to work).
|
| RIP to the engineers fixing this without any AI help.
| fullstackchris wrote:
| They better not be using AI to fix this... especially if AI is
| what caused it! (looking at you, AWS)
| re5i5tor wrote:
| For me right now, Claude.ai is down, but Claude Code (terminal,
| extension) seems to be up and happy. Suggests that API is
| probably up.
| lucabrekkav wrote:
| it comes back up now
| tedmundy wrote:
| I think everyone is in the same boat with thinking they took
| something offline :^)
|
| Concerning though how much the web relies on one (great) service.
| pqdbr wrote:
| I had just deployed. Started reverting commits like crazy.
| r721 wrote:
| >Cloudflare Global Network experiencing issues
|
| >Cloudflare is aware of, and investigating an issue which
| potentially impacts multiple customers. Further detail will be
| provided as more information becomes available.
|
| >Posted 4 minutes ago
|
| https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
| robotfelix wrote:
| Things are back up (a second time) for me. It doesn't sound
| like they are confident the problem is fully fixed yet though
|
| Edit: and down again a third time!
| robotfelix wrote:
| > Cloudflare Global Network experiencing issues
|
| > Investigating - Cloudflare is aware of, and investigating an
| issue which potentially impacts multiple customers. Further
| detail will be provided as more information becomes available.
|
| Things are back up (a second time) for me.
|
| Cloudflare have updated their status page now to reflect the
| problems now. It doesn't sound like they are confident the
| problem is fully fixed yet.
|
| Edit: and down again a third time!
| grrowl wrote:
| it's back again!
| TavsiE9s wrote:
| Investigating - Cloudflare is aware of, and investigating an
| issue which potentially impacts multiple customers. Further
| detail will be provided as more information becomes available.
| Nov 18, 2025 - 11:48 UTC
|
| Yeah, those multiple customers is like 70% of the internet.
| keshavmunjal wrote:
| Cloudflare is still down
| rehanahmed wrote:
| Funny how I couldn't even check on Downdetector.com - because it
| takes me to a Cloudfare-run captcha, which is now stuck on
| loading.
|
| The internet is officially down.
| keshavmunjal wrote:
| cloudfare down in india
| mrintegrity wrote:
| It would have been ironic had this
| https://www.thewebsiteisdown.com/salesguy.html been hosted behind
| cloudflare.
| chinathrow wrote:
| It's back up.
| techguy1954 wrote:
| It would appear if you use a VPN in Europe you can still access
| Cloudflare sites, I have just tried, for me the Netherlands,
| Germany, and France work, but the UK and USA don't.
|
| EDIT: It would appear it is still unreliable in these countries,
| it just stopped working in France for me.
| brushfoot wrote:
| I've been considering Cloudflare for caching, DDoS protection and
| WAF, but I don't like furthering the centralization of the Web.
| And my host (Vultr) has had fantastic uptime over the 10 years
| I've been on them.
|
| How are others doing this? How is Hacker News hosted/protected?
| NicoJuicy wrote:
| Was back up for a moment ( within 5 minutes of being down),
| currently down again
| jjgreen wrote:
| In Yorkshire they would say "up and down like a bride's
| nightie"
| alt227 wrote:
| ...or a whores drawers
| jjgreen wrote:
| In Manchester perhaps ...
| chistev wrote:
| Yea, had trouble accessing Upwork
| ea016 wrote:
| We're on the enterprise plan, so far we're seeing Dashboard
| degradation and Turnstile (their captcha service) down. But all
| proxying/CDN and other services seem to work well.
| Semaphor wrote:
| We finally switched to CF a few weeks ago (for bot protection,
| abusive traffic started getting insane this year), finally we can
| join in on one of the global outage parties (no cloud usage
| otherwise, so still more uptime than most).
| sutton12345 wrote:
| cloudfare is down officially
| aap_ wrote:
| Crazy to think that it's apparently acceptable to centralize the
| web like that.
| rulehard wrote:
| why does no one use google cloud? literally never has any issues
| lol
| avadhesh18 wrote:
| because no one uses it
| benedictboppler wrote:
| Expensive, hard to use, a million dashboards
| eruci wrote:
| Searching for Cloudflare alternatives...
| CREAMY_BANANA wrote:
| I can't access my fav porn sites, cloudflare Singapore host is
| down. Oh man!
| oxy1tumn wrote:
| Took down both Twitter and Rateyourmusic. This is a targetted
| attack against me specifically and nobody else
| anuraagvaidya wrote:
| I got an email saying that my OpenAI auto-renewal failed, my
| credits have run out. I go to OpenAI to reauthorize the card, and
| I can't login because OpenAI uses Cloudflare for "verifying you
| are a human" that goes in infinite loop. Great.
| Piko wrote:
| oh no, the vibes won't code themselves /s
| anuraagvaidya wrote:
| Or even worse, people with apps in production with credits
| running out
| rkwasny wrote:
| downdetector.com is down because it uses cloudflare challenge....
| sutton12345 wrote:
| do any of users facing cloudfare outage
| ???!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
| vinishbhaskar wrote:
| Tried checking Cloudflare's status on Downdetector, but
| Downdetector was also behind Cloudflare. Internet checkmate.
| josephb wrote:
| It's not just websites :-/
|
| Things like Apple private relay (which way too many people seem
| to have it enabled) are tunnelled via Cloudflare, maybe using
| warp?
| StefanThorpe wrote:
| Whats peoples bets on the root cause of this...?
| redrove wrote:
| Their Oregon controlplane somehow? Either a misconfiguration of
| some sort (BGP??) or a power outage like they had before.
| outroll wrote:
| Yep, got around 100 SMSs from our uptime monitoring service that
| our Cloudflare sites are down. Nothing much we can do but wait.
| vldszn wrote:
| x.com is down
| lyfeesgood wrote:
| singapore down
| aws_eu1 wrote:
| ERROR [11:57:30 UTC]: EC2 Launch Failure. Reason: [Security
| Breach Remediation] Control Plane Metadata Service (IMDS)
| temporarily offline. System state reports: Dependency integrity
| check failed (Exit Code 0x80070002). Cannot retrieve authorized
| kernel image or block device mapping. Termination signal
| initiated for compromised worker nodes.
| flamixin wrote:
| This sounds huge.
| rulehard wrote:
| pretty much just my twitter usage as of now. my sites dont use
| cloudflare
| ilkkao wrote:
| I can't even load the dashboard to change to "DNS only". Nothing
| to do?
| mak8 wrote:
| supabase is down too
| yard2010 wrote:
| Yes. I get 500 on my website.
| bilekas wrote:
| Well it was bound to happen eventually, the "Down Roulette" has
| decided it should be Cloudflare this week!
| aws_eu1 wrote:
| CONTROL PLANE FAULT: CRITICAL SECURITY OVERRIDE enforced across
| us-east-1 and eu-west-2. ERROR CODE: STS.SecurityAuditLockout
| (403 Forbidden).
| chinathrow wrote:
| Context?
| techguy1954 wrote:
| ChatGPT and Perplexity AI are down, I was just about to try and
| use ChatGPT.
| vednig wrote:
| all the captcha's are also not working
| jamesnorden wrote:
| Almost like centralizing everything on a single service has
| consequences.
| lpcvoid wrote:
| Can we at some point acknowledge that constant cloud disruptions
| are too costly, and can we then finally move all of our hosting
| back on-prem?
| donglong wrote:
| can you define "constant"
| lpcvoid wrote:
| Well, between AWS US EAST 1 killing half the internet, and
| this incident, not even a month passed. Meanwhile, my
| physical servers don't care and happily serve many people at
| a cheaper cost than any cloud offer.
| alt227 wrote:
| We had an Azure outage in between those 2 as well.
| chistev wrote:
| How do you back up?
| lpcvoid wrote:
| We have a few colocated servers offsite, each in a
| different region, each with a zpool of mirrored spinning
| rust. We use rsync across those at different times.
| donglong wrote:
| never build on us-east-1, everyone knows that ;)
| vlovich123 wrote:
| You realize these are two different companies right? If
| you're saying "I'm an AWS customer with cloudflare in
| front" I think you've failed to realize that two 99.9%
| available services in series have a combined availability
| of ~99.8% - that's just math.
|
| Your physical servers should have similar issues if you put
| a CDN in front unless the physical server is able to
| achieve a 100% uptime (100% * 3 9s = 3 9s). Or you don't
| have a CDN but can be trivially knocked offline by the
| tiniest botnet (or even hitting hacker news front page)
| lpcvoid wrote:
| I do. But I put both into the "cloud offering off-prem
| for very much money" shoebox. I setup a CDN once using
| VPS from different hosting providers for under 100 USD a
| month, which I would vastly prefer over trusting anything
| cloud.
|
| And yes, I know that there's sites that need the scale of
| an operation like Cloudflare or AWS. But 99.9(...)% of
| pages don't, and people should start realizing that.
| mallets wrote:
| People who don't need that, also don't care much for an
| hour or two of service disruption. Most users will have
| far worse disruptions with the alternatives.
| darkwater wrote:
| Funnily and ironically enough, I was trying to check out a few
| things on Ansible Galaxy and... I ended up here trying to
| submit the link for the CF ongoing incident
| DC-3 wrote:
| It's the old IBM thing. If your website goes down along with
| everyone else's because of Cloudflare, you shrug and say
| "nothing we could do, we were following the industry standard".
| If your website goes down because of on-prem then it's very
| much your problem and maybe you get to look forward to an
| exciting debrief with your manager's manager.
| lpcvoid wrote:
| That's lazy engineering and I don't think we as technical,
| rational people should make that our way of working. I know
| the saying, but I disagree with it. My fuckups, my problem,
| but at least I can avoid fuckups actively if I am in charge.
| reassess_blind wrote:
| How do you mitigate large scale DDoS?
| lpcvoid wrote:
| I don't, since my stuff is reachable only within the
| company network/VPN. If I needed to though, I would
| consult the BSI list of official DDOS mitigation services
| [0] and evaluate each one before deciding. I would not
| auto-pick Cloudflare.
|
| [0] (German) https://www.bsi.bund.de/SharedDocs/Downloads
| /DE/BSI/Cyber-Si...
| reassess_blind wrote:
| When the solution you pick inevitably has downtime too
| you're in the same boat.
|
| DDoS mitigation is one of those areas that an on-prem
| solution just isn't well suited to solve.
| lpcvoid wrote:
| Yeah, but people aren't using Cloudflare just for DDOS
| Mitigation. Some are running pretty much everything over
| it, from DNS to edge caching to load balancing and even
| hosting. That's what I oppose mainly.
| lousken wrote:
| Unless you are really big, onprem stuff would be 90%
| internal anyway. For everything public you'd host your
| hardware in a datacenter with better high speed
| connectivity. And pretty much every single datacenter I
| interacted with in the last 5 years does have a DDOS
| protection solution that you can order for your network.
| saubeidl wrote:
| The problem is the people that sign our checks usually
| aren't technical, rational people.
|
| The system isn't designed for technical, rational decision
| making.
| lpcvoid wrote:
| That's fair, yeah, and I agree it's not always feasible -
| but if you have any influence over technical direction at
| your org, I encourage what I wrote above. Otherwise yeah,
| let the pea counters in the C-Levels dig their own grave.
| rkangel wrote:
| I would only consider doing stuff on-prem _because_ of services
| like Cloudflare. You can have some of the global features like
| edge-caching while also getting the (cost) benefits of on-prem.
| celltalk wrote:
| I am glad my personal site is not affected, what would I do
| without all those incoming traffic.
| ashsk0110 wrote:
| World infrastructure is taking a hit. First us-east and now this.
| thoughtogram wrote:
| Cant websites have an auto backup/redirect in case cloudflare or
| AWS go down?
| mberger wrote:
| Is it DNS? I went to check the isitdns.com but got a cloudflare
| error
| ablation wrote:
| It's knocked out Turnstile too, which means I can't even log in
| to my Cloudflare dash to bypass my site's proxying via
| Cloudflare.
| porker wrote:
| Even if you could, the DNS entries aren't loading. And then the
| page 404's.
| stavros wrote:
| Oh you aren't missing much, the dashboard doesn't load anyway.
| bamboozled wrote:
| This one is huge.
| drchiu wrote:
| Just when the eastern cities are waking up too.
| aws_eu1 wrote:
| ERROR [12:00:21 UTC]: CF_EDGE_ROUTING_FAILURE. Reason: Origin-
| Shield connectivity loss detected within multi-region fabric. BGP
| path withdrawal initiated for critical LCP clusters (LCP-LON,
| LCP-FRA). Status code 521/522 flood reported globally. Geo-
| location failover services degraded. DNS resolution timeout on
| 1.1.1.1/1.0.0.1. Traffic flow re-routing pending verification of
| internal control plane integrity.
| dontdoxxme wrote:
| Did you ask an LLM to try to guess the error message?
| jamesnorden wrote:
| And no lesson about single point of failure and centralization
| was learned that day.
| carlosjobim wrote:
| Where is the single point of failure? You can point to
| different name servers and swiftly remove Cloudflare from your
| setup.
| JustSkyfall wrote:
| Not if you're using Workers/Pages!
| blackjackfoe wrote:
| If your site is only hosted on one server and it catches
| fire, you can swiftly reinstall on a new server and change
| the IP your domain is pointing to, too... Still a single
| point of failure.
| carlosjobim wrote:
| Yes, everything in the world is a single point of failure
| and has always been, if we look at things that way. But if
| it can be remedied quickly, then it's not a huge concern.
| cryptonym wrote:
| Only true if your audience doesn't require Edge distribution,
| also if your Origin can handle the increased load and
| security issues, also if you don't use any advanced features
| (routing, edge compute...).
| carlosjobim wrote:
| Aren't there other companies offering edge distribution?
| pessimizer wrote:
| But you didn't, so Cloudflare ended up being a single point
| of failure for half the internet.
| carlosjobim wrote:
| If I had pointed my name servers somewhere else, then that
| of course would be the new single point of failure. You
| can't escape it, no matter how much hacker snark and down
| votes you have.
| filleduchaos wrote:
| Bold of you to assume the service you use to manage your DNS
| was not also relying on Cloudflare just like you
| ashed96 wrote:
| Suddenly feeling better about our 99.9% uptime SLA.
|
| When even Cloudflare goes down, nobody can blame the little guys.
| udev4096 wrote:
| Would be funny if it was a record breaking ddos on cloudflare
| KabuTheSilicon wrote:
| The reason why you laugh
| pstation wrote:
| Cloudflare's dashboard is currently down as well.
|
| My domain is registered with cloudflare so I'm 100% helpless to
| get things back online.
|
| I can't edit DNS records to bypass cloudflare and I can't change
| nameservers either.
| astonfred wrote:
| We are all impacted...
| lofaszvanitt wrote:
| Seemingly nobody cares about being in two different availability
| zones. Or is this a deeper problem?
| whd5015 wrote:
| just scheduled maintenance in Tahiti guys, nothing to see here
| rikafurude21 wrote:
| could be related? Tahiti is a small island in the south
| pacific, seems rather suspicious
| demontime wrote:
| Twitter too is down, almost half the internet
| MaximilianEmel wrote:
| When's the last time Cloudflare had such an outage?
| Metacelsus wrote:
| Yeah I just got a 500 error on medRxiv
| akashvekariya wrote:
| We are doomed! Is it another vibe-coding disaster?
| robertwt7 wrote:
| this should affect a lot sites? I'm trying to access tailwindcss
| and I can't as well!
| chistev wrote:
| Not my site though
|
| https://www.rxjourney.net/
| csomar wrote:
| I am using cloudflare as back-end for my site (workers) but
| have disabled _all_ their other offerings. I was affected for a
| short while but seems to be less affected than other people.
| biinjo wrote:
| Lol. I mean I love Tailwind but it seems like the least trivial
| site/service to be down right now haha.
| ImScope052300 wrote:
| Dude what's up with X???
| kmarc wrote:
| Yesterday I decided to finally write my makefiles to "mirror"
| (make available offline) the docs of the libraries I'm using.
| doc2dash for sphinx-enabled projects, and then using dash / zeal.
|
| Then I was like... "when did I last time fly for 10+ hours and
| wanted to do programming, etc, so that I need offline docs?" So I
| gave up.
|
| Today I can't browse the libs' docs quickly, so I'm resuming the
| work on my local mirroring :-)
| ImScope052300 wrote:
| Bro what's up with X???
| SallysField wrote:
| Bro no way bro what's going on bro? BRO?! Bro this is crazy bro
| like one of your cartoons that you jerk it to every day bro
| akashvekariya wrote:
| We're doomed. Is this another vibe-coding disaster?
| wolfgx wrote:
| I was using Cloudflare WARP; had to turn it off to access most of
| the websites i visit daily.
| lambdaba wrote:
| it's funny I first noticed this visiting a random blog, then went
| on X and got the same error... is Cloudflare the Internet now?
| fullstackchris wrote:
| Supabase is down bad too... need to work on my project!
| fullstackchris wrote:
| Haha they updated their status page: "Identified - A global
| upstream provider is currently experiencing an outage which is
| impacting platform-level and project-level services"
|
| A _global upstream provider_ :)
| farhadhf wrote:
| Pretty much everything is down (checking from the Netherlands).
| The Cloudflare dashboard itself is experiencing an outage as
| well.
|
| Not-so-funny thing is that the Betterstack dashboard is down but
| our status page hosted by Betterstack is up, and we can't access
| the dashboard to create an incident and let our customers know
| what's going on.
|
| Edit: wording.
| ablation wrote:
| This is a big one.
| csomar wrote:
| Seems like workers are less affected and maybe betterstack has
| decided to bypass cloudflare "stuff" for the status pages?
| (maybe to cut down costs). My site is still up though some
| GitHub runners did show it failed at certain points.
| tyingq wrote:
| I have a workers + kv app that seems fine right now.
| csomar wrote:
| Pretty sure they went down for a while because I have 4xx
| errors they returned but apparently it was short-lived. I
| wonder if their workers infra. failed for a moment and that
| let to a total collapse of all of their products?
| biinjo wrote:
| Same here. We're using OhDear. The status page is available but
| I can't post an incident because their service is also behind
| Cloudflare.
| Mojah wrote:
| Co-founder here, we'll be working on better ways to handle
| this over the coming days.
|
| Update: our app is available again without Cloudflare, you'll
| be able to post updates to status pages smoothly again.
| pell wrote:
| It's that time of the year again where we all realize that
| relying on AWS and Cloudflare to this degree is pretty
| dangerous but then again it's difficult to switch at this
| point.
|
| If there is a slight positive note to all this, then it is that
| these outages are so large that customers usually seem to be
| quite understanding.
| tommica wrote:
| > It's that time of the year again
|
| It's monthly by now
| fusl wrote:
| Happy to hear anyone's suggestions about where else to go or
| what else to do in regards to protecting from large-scale
| volumetric DDoS attacks. Pretty much every CDN provider
| nowadays has stacked up enough capacity to tank these kind of
| attacks, good luck trying to combat these yourself these
| days?
| isodev wrote:
| Anubis and/or Bunny are good alternatives/combination
| depending on your exact needs
|
| - https://anubis.techaro.lol/
|
| - https://bunny.net/
| fusl wrote:
| Unfortunately Anubis doesn't help where my pipe to the
| internet isn't fat enough to just eat up all the
| bandwidth that the attacker has available. Renting tens
| of terabits of capacity isn't cheap and DDoS attacks
| nowadays are in the scale of that. BunnyCDN's DDoS
| protection is unfortunately too basic to filter out
| anything that's ever so slightly more sophisticated.
| Cloudflare's flexibility in terms of custom rulesets and
| their global pre-trained rulesets (based on attacks
| they've seen in the past) is imo just unbeatable at this
| time.
| isodev wrote:
| The Bunny Shield is quite similar to the Cloudflare
| setup. Maybe not 100% overlap of features but unless
| you're Twitter or Facebook, it's probably enough.
|
| I think at the very least, one should plan the ability to
| switch to an alternative when your main choice fails...
| which together with AWS and GitHub is a weekly event now.
| immibis wrote:
| We live in the world of mass internet surveillance. DDoS
| like this are not very common, partly because people who
| do it keep going to jail.
| Doman wrote:
| bunny.net is not reachable for me too... really funny
|
| https://imgur.com/a/8gh3hOb
| haar wrote:
| I clicked the image thinking I was seeing the message you
| were getting (geoblocked in the UK), then realised I'd
| clicked an imgur link :facepalm:
|
| (Note: Zero negative sentiment towards imgur here)
| isodev wrote:
| All the edges are gone! :)
| RKFADU_UOFCCLEL wrote:
| Why do people on a technical website suggest this? It's
| literally the same snake oil as Cloudflare. Both have an
| endgame of total web DRM; they want to make sure users
| "aren't bots". Each time the DRM is cracked, they will
| increase its complexity of the "verifier". You will be
| running arbitrary code in your big 4 browser to ensure
| you're running a certified big 4 browser, with 10
| trillion man hours of development, on an certified OS.
| callalex wrote:
| Because there is a real problem that needs to be solved
| one way or another.
| RKFADU_UOFCCLEL wrote:
| Anubis doesn't solve anything, bud.
| q3k wrote:
| Just accept that a DDoS might happen and that there's
| nothing you can do about it. It's fine, it's just how the
| Internet works.
| herbst wrote:
| That was possible when a DDos was usually still an
| occasional attack by a bad actor.
|
| Most time I get ddosed now it's either Facebook directly,
| Something something Azure or any random AI.
| q3k wrote:
| That sounds like an app-level (D)DoS, which is generally
| something you can mitigate yourself.
| geerlingguy wrote:
| It's harder when it's a new group of IPs and happens 2-3x
| every month.
| herbst wrote:
| And if you do rule based blocking they just change their
| approach. I am constantly blocking big corps these days,
| barely any work with normal bad actors.
|
| And lots of real users time wasted for captchas.
| nhecker wrote:
| How (or to what end) would Facebook want to directly DoS
| someone?
| herbst wrote:
| What do they even have an spider for? I never saw any
| actual traffic with source Facebook. I don't understand
| either, but it's their official IPs, their official bot
| headers and it behaves exactly like someone who wants my
| sites down.
|
| Does it make sense? Nah, but is it part of the weird
| reality we live in. Looks like it
|
| I have no way of contacting Facebook. All I can do is
| keep complaining on hackernews whenever the topic
| arrises.
|
| Edit:// Oh and I see the same with Azure, however there I
| have no list of IPs to verify it's official just because
| it looks like it.
| inferiorhuman wrote:
| I got DoS'd by them once, email not HTTP traffic though.
| Quick slip of their finger and bam low cost load testing.
| peanut-walrus wrote:
| So accept that your customers won't be able to use your
| services whenever some russian teenager is bored? Yeah,
| good luck with justifying that choice.
| q3k wrote:
| And how often does that happen?
| peanut-walrus wrote:
| For the service I'm responsible for, 4 times in the last
| 24 hours.
| q3k wrote:
| Congratulations, you're the exception rather than the
| norm.
| bandrami wrote:
| Is a DDOS more frequent and/or worse than stochastic CDN
| outages?
| trollbridge wrote:
| Somehow KiwiFarms figured it out with their own "KiwiFlare"
| DDOS mitigation. Unfortunately, all of the other
| Cloudflare-like services seem exceptionally shady, will be
| less reliable than Cloudflare, and probably share data with
| foreign intelligence services I have even less trust for
| than the ones Cloudflare possibly shares them with.
| lbreakjai wrote:
| If I choose AWS/cloudflare and we're down with half of the
| internet, then I don't even need to explain it to my boss'
| bosses, because there will be an article in the mainstream
| media.
|
| If I choose something else, we're down, and our competitors
| aren't, then my overlords will start asking a lot of
| questions.
| jfengel wrote:
| And if everyone else is down, and you are not, you will get
| no credit.
| lbreakjai wrote:
| Or _you_ aren't down, but a third-party you depend on is
| (auth0, payment gateway, what have you), and you invested
| a lot of time and effort into being reliable, but it was
| all for less than nothing, because your website loads but
| customers can't purchase, and they associate the problem
| with you, not with the AWS outage.
| trollbridge wrote:
| Right. Whereas if we get whacked with a random DDoS,
| that's my fault.
| stevepotter wrote:
| Yup. AWS went down at a previous job and everyone basically
| took the day off and the company collectively chuckled.
| Cloudflare is interesting because most execs don't know
| about it so I'd imagine they'd be less forgiving. "So what
| does cloudflare do for us exactly? Don't we already have
| aws?"
| timeon wrote:
| In reality it is not half of the internet. That is just
| marketing. I've personally noticed one news site while
| others were working. And I guess sites like that will get
| the blame.
| weird-eye-issue wrote:
| Oh no, we had 30 minutes of downtime this year :(
| pell wrote:
| I do think this is tenable as long as these services are
| reliable. Even though there have been some outages I would
| argue that they're incredibly reliable at this point. If
| though this ever changes the costs to move to a competitor
| won't be as simple as pushing a repository elsewhere,
| especially for AWS. I think that's where some of the
| potential danger lies.
| weird-eye-issue wrote:
| > especially for AWS
|
| CF can be just as difficult if not more to migrate off of
| especially when using things like durable objects
| swyx wrote:
| > 30 minutes of downtime
|
| > this is tenable as long as these services are reliable
|
| do you hear yourself, this is supposed to be a
| distributed CDN. imagine if HTTP had 30 minutes of
| downtime a year.
|
| and judging by the HN post age, we're now past minute 60
| of this incident.
| weird-eye-issue wrote:
| > and judging by the HN post age, we're now past minute
| 60 of this incident.
|
| Huh? It's been back up during most of this time. It was
| up and then briefly went back down again but it's been up
| for a while now. Total downtime was closer to 30 minutes
| swyx wrote:
| twitter still down for me
| mrkramer wrote:
| Twitter is down while Mastodon is proudly and strongly
| still standing up. I knew this day would come.
| swyx wrote:
| i also can host apps for 100 users
| CableNinja wrote:
| 5 9's is like 7 minutes a year. They are breaking SLAs and
| impacting services people depend on
|
| Tbh though this is sort of all the other companies fault,
| "everyone" uses aws and cf and so others follow. now not
| only are all your chicks in one basket, so is everyone
| elses. When the basket inevitably falls into a lake....
|
| Providers need to be more aware of their global impact in
| outages, and customers need to be more diverse in their
| spread.
| weird-eye-issue wrote:
| > Providers need to be more aware of their global impact
| in outages
|
| So you think the problem is they aren't "aware"?
| CableNinja wrote:
| These kinds of outages continue to happen and continue to
| impact 50+% of the internet, yes, they know they have
| that power, but they dont treat changes as such, so no,
| they arent aware. Awareness would imply more care in
| operations like code changes and deployments.
|
| Outages happen, code changes occur; but you can do a lot
| to prevent these things on a large scale, and they simply
| dont.
|
| Where is the A/B deployment, preventing a full outage?
| What about internally, where was the validation before
| the change, was the testing run against a prodlike
| environment or something that once resembled prod but
| hasnt forever?
|
| They could absolutely mitigate impacting the entire
| global infra in multiple ways, and havent, despite their
| many outages.
| richardwhiuk wrote:
| They are aware. They don't want to pay the cost benefit
| tradeoff. Education won't help - this is a very heavily
| argued tradeoff in every large software company.
| world2vec wrote:
| 99.999% availability is around 5 minutes or so of
| downtime per year.
| isodev wrote:
| Unless you're say at airport trying to file a luggage claim
| ... or at the pharmacy trying to get your prescription. I
| think as a community we have a responsibility to do better
| than this.
| sigilis wrote:
| You aren't cloudflare's customer in these examples. It
| depends on the companies that are actually paying for and
| using the service to complain. Odds are that they won't
| care on your behalf due to how our society is structured.
|
| Not really sure how our community is supposed to deal with
| this.
| isodev wrote:
| "We" are the ones making the architecture and the
| technical specs of these services. Taking care for it to
| still work when your favourite FAANGMC is down seems like
| something we can help with.
| ChrisMarshallNY wrote:
| _> I think as a community we have a responsibility to do
| better than this._
|
| I have always felt so, but my opinion is definitely in the
| minority.
|
| In fact, I find that folks have _extremely negative_
| responses to any discussion of improving software Quality.
| mosura wrote:
| Merely reducing external dependencies causes people to
| come out in rashes.
|
| A large proportion of "developers" enjoy build vs buy
| arguments far too much.
| abustamam wrote:
| I always see such negative responses when HN brings up
| software bloat ("why is your static site measured in
| megabytes").
|
| Now that we have an abundance of compute and most people
| run devices more powerful than the devices that put man
| on the moon, it's easier than ever to make app bloat,
| especially when using a framework like Electron or React
| Native.
|
| People take it personally when you say they write poor
| quality software, but it's not a personal attack, it's an
| observation of modern software practices.
|
| And I'm guilty of this, mainly because I work for
| companies that prioritize speed of development over
| quality of software, and I suspect most developers are in
| this trap.
| ChrisMarshallNY wrote:
| What I find annoying, is people making fun of folks that
| choose to "roll their own."
|
| The typical argument that I see, is homemade encryption,
| which is quite valid.
|
| However, encryption is just a tiny corner of the surface.
|
| Most folks don't want to haul in 1MB of junk, just so
| they can animate a transition.
|
| Well, I guess I should qualify that: Most _normal_ folks
| wouldn 't want to do that, but, apparently, it's _de
| rigueur_ for today 's coders.
| abustamam wrote:
| I think we have a new normal now though. Most web devs
| starting now don't know a world without
| React/Vue/Solid/whatever. Like, sure you can roll your
| own HTML site with JS for interactivity, but employers
| now don't seem to care about that; if you don't know
| React then don't bother.
| dlisboa wrote:
| > If there is a slight positive note to all this, then it is
| that these outages are so large that customers usually seem
| to be quite understanding.
|
| Which only shows that chasing five 9s is worthless for almost
| all web products. The idea is that by relying on AWS or
| Cloudflare you can push your uptime numbers up to that
| standard, but these companies themselves are having such
| frequent outages that customers themselves don't expect that
| kind reliability from web products.
| esskay wrote:
| When its back up, do yourself a favour and rent a $5/mo vps in
| another country from a provider like OVH or Hetzner and stick
| your status page on that.
|
| "Yes but what if they go down" - it doesnt matter, having it
| hosted by someone who can be down for the same reason as your
| main product/service is a recipe for disaster.
| hcaz wrote:
| https://cachethq.io/ is great for this
| fusl wrote:
| Been using Cachet for quite a while before inevitably
| migrating to Atlassian's Statuspage.io. I'm a huge fan of
| self-hosting and self-managing every single thing in
| existence but Cachet was just such a PITA to maintain and
| there was just no other good alternative to Cachet that was
| also open source.
| jwr wrote:
| Amusingly enough, it is down right now because of
| Cloudflare :-)
| jwr wrote:
| Or use a service like https://updown.io/ (I host my status
| page there).
| fodi wrote:
| Definitely. Tangentially, I encountered 504 Gateway Timeout
| errors on cloudflarestatus.com about an hour ago. The error
| page also disclosed the fact that it's powered by CloudFront
| (Amazon's CDN).
| postalcoder wrote:
| Cloudflare dashboard is down-ish, not _totally_ down. If you
| 're persistent you can turn off the turnstile and proxy.
|
| It took a few minutes but I got https://hcker.news off of it.
| farhadhf wrote:
| I'm already logged in on the cloudflare dashboard and trying
| to disable the CF proxy, but getting "404 | Either this page
| does not exist, or you do not have permission to access it"
| when trying to access the DNS configuration page.
| skywhopper wrote:
| Not saying not to do this to get through, but just as an
| observation, it's also the sort of thing that can make these
| issues a nightmare to remediate, since the outage can
| actually draw more traffic just as things are warming up,
| from customers desperate to get through.
|
| But then, that's what Cloudflare signed up to be.
| trollbridge wrote:
| I can't sign in since Turnstile is down so I can't complete
| the captcha to log in.
|
| I also can't log in via Google SSO since Cloudflare's SSO
| service is down.
| celltalk wrote:
| I think there is a big business opportunity here. Make a site
| that let companies put their status update on local vps for
| $100.
| ramon156 wrote:
| on-demand status balancing!
| colinbartlett wrote:
| Even the Cloudflare status page, hosted by Atlassian
| Statuspage, is suffering. Probably due to the traffic crush.
| nrhrjrjrjtntbt wrote:
| Status pigeons.
| alt227 wrote:
| Atlassian has this business model sewn up
|
| https://www.atlassian.com/software/statuspage
| lc64 wrote:
| It's worth noting that cloudflare's status page is hosted
| there. Pretty good proof that it works
| dmit wrote:
| And I got a 504 error (served by CloudFront) on that
| status page earlier. The error message suggested there
| may have been a great increase in traffic that caused it.
| codethief wrote:
| Maybe that's precisely what Cloudflare did and now their
| status page is down because it's receiving an unusual amount
| of traffic that the VPS can't handle.
| celltalk wrote:
| They should have had Cloudflare on it.
| chrisandchris wrote:
| I don't get why you need such a service for a status page with
| 99.whatever% uptime. I mean, your status page only has to be up
| if everything else is down, so maybe 1% uptime is fine.
|
| /s
| huijzer wrote:
| Yep that's also my experience. Except HN because it does not
| use *** Cloudflare because it knows it is not necessary. I just
| wrote a blog titled "Do Not Put Your Site Behind Cloudflare if
| You Don't Need To" [1].
|
| [1]: https://huijzer.xyz/posts/123/
| MinimalAction wrote:
| Yes, I never understand this obsession for centralized
| services like Cloudflare. To be fair though, if our tiny
| blogs anyway had a hundred or so visitors monthly, does it
| matter if it had an outage for a day?
| ThunderSizzle wrote:
| I think partially is not having to worry about certs is a
| nice reason to hide behind the proxy. Also, to help hide
| your IP address, I guess.
|
| Of course, on the other hand, I know that relying on
| Cloudflare cert's is basically inviting a MITM attack.
| huijzer wrote:
| > I think partially is not having to worry about certs is
| a nice reason to hide behind the proxy.
|
| Use Caddy. I never worry about certs.
| ThunderSizzle wrote:
| Interesting. I've done a lot of manual work to set up a
| whole nginx layer to properly route stuff through one
| domain to various self-hosted services, with way to many
| hard lessons when I started this journey (from trying to
| do manual setup without docker, to moving onto repeatable
| setups via docker, etc.).
|
| The setup appears very simple in Caddy - amazingly
| simple, honestly. I'm going to give it a good try.
| immibis wrote:
| Or certbot-plugin-nginx if you prefer a bit less magic.
| ptx wrote:
| Don't you need a cert anyway to secure the connection
| from Cloudflare to your server?
| ThunderSizzle wrote:
| You could use a self-signed cert, since cloudflare
| doesn't care about that.
| omcnoe wrote:
| Cloudflare explicitly supports customers placing insecure
| HTTP only sites behind a cloudflare HTTPS.
|
| It's one of the more controversial parts of the business,
| it makes the fact that the traffic is unencrypted on
| public networks invisible to the end user.
| Illniyar wrote:
| Does HN not experience DDOS? I would imagine being as popular
| as it is it'll experience DDOS.
| q3k wrote:
| It does: https://m5hosting.status.io/pages/incident/5407b8e
| 2b00244251...
|
| But turns out that's fine :).
| everybodyknows wrote:
| > Issues are stable at this time. The targeted customer
| has implemented CloudFlare, and we have taken steps to
| mitigate this event.
|
| I'm still confused. Does this mean that HN switches CF on
| or off in response to recent volume of bot traffic?
| firecall wrote:
| Sadly, AI bots and crawlers have made CF the only affordable
| way to actually keep my sites up without incurring excessive
| image serving costs.
|
| Those TikTok AI crawlers were destroying some of my sites.
|
| Millions of images served to ByteSpider bots, over and over
| again. They wouldn't stop. It was relentless abuse. :-(
|
| Now I've just blocked them all with CF.
| flakeoil wrote:
| > Now I've just blocked them all with CF.
|
| Yeah, they for sure let nothing through right now. ;)
| marcosdumay wrote:
| There isn't too much of a difference from their normal
| behavior.
| zenmac wrote:
| Wouldn't it be trivial to just to write a uwf to block the
| crawler ips?
|
| At time like this really glad we self-hosted.
| firecall wrote:
| Maybe :-)
|
| But for a small operation, AKA just me, it's one more
| thing for me to get my head around and manage.
|
| I don't run just one one website or one service.
|
| It's 100s of sites across multiple platforms!
|
| Not sure I could ever keep up playing AI Crawler and IP
| Whack-A-Mole!
| cornedor wrote:
| No, since they're simply too many. For an e-commerce site
| I work for, we once had an issue where some bad-actor
| tried to crawl the site to set up scam shops. The list of
| IPs were way too broad, and the user-agents way too
| generic or random.
| 72deluxe wrote:
| Could you not also use an ASN list like
| https://github.com/brianhama/bad-asn-list and add blocks
| of IPs to a blocklist (eg. ipset on Linux)? Most of the
| scripty traffic comes from VPSs.
| jeroenhd wrote:
| Thanks to widespread botnets, most scrapers fall back to
| using "residential proxies" the moment you block their
| cloud addresses. Same load, but now you risk accidentally
| blocking customers coming from similar net blocks.
|
| Blocking ASNs is one step of the fight, but unfortunately
| it's not the solution.
| immibis wrote:
| Hypothetically, as a cyber-criminal, I'd like to thank
| the blacklist industry for bringing so much money into
| criminal enterprises by making residential proxies
| mandatory for all scraping.
| tpetry wrote:
| Its not one IP to block. Its thousands! And they're also
| scatter through different ip networks so no simple cidr
| block is possible. Oh, and just for the fun, when you
| block their datacenter ips they switch to hundreds of
| residential network ips.
|
| Yes, they are really hard to block. In the end I switched
| to Cloudflare to just so they can handle this mess.
| Bender wrote:
| _Wouldn 't it be trivial to just to write a uwf to block
| the crawler ips?_
|
| Probably more effective would be to get the bots to
| exclude your IP/domain. I do this for SSH, leaving it
| open on my public SFTP servers on purpose. [1] If I can
| get 5 bot owners to exclude me that could be upwards of
| 250k+ nodes _mostly mobile IP 's_ that stop talking to
| me. Just create something that confuses and craps up the
| bots. With SSH bots this is trivial as most SSH bot
| libraries and code are unmaintained and poorly written to
| begin with. In my ssh example look for the
| VersionAddendum. Old versions of ssh, old ssh libraries
| and code that tries to implement ssh itself will choke on
| a long banner string. _Not to be confused with the text
| banner file._
|
| I'm sure the clever people here could make something
| similar for HTTPS and especially for GPT/LLM bots _at the
| risk of being flagged "malicious"_.
|
| [1] - https://mirror.newsdump.org/confuse-some-ssh-
| bots.html
|
| _About 90%+ of bots can not visit this URL, including
| real people that have disabled HTTP /2.0 in their
| browser._
| Aeolun wrote:
| > Now I've just blocked them all with CF.
|
| You realize it was possible to block bad actors before
| Cloudflare right? They just made it easier, not possible in
| the first place.
| firecall wrote:
| Of course :-)
|
| And my image CDN blocked ByteSpider for me.
|
| For a while I also blocked the entirety of Singapore due
| to all the bots coming out of AWS over there!
|
| But it's honestly something I just dont need to be
| thinking about for every single site I run across a
| multitude of platforms.
|
| Having said that, I will now look at the options for the
| business critical services I operate for clients!
| delfinom wrote:
| Bad actors now have access to tens of thousands of IPs
| and servers on the fly.
|
| The cost of hardware and software resources these days is
| absolute peanuts compared to 10 years ago. Cloud services
| and APIs has made managing them also trivial as hell.
|
| Cloudflare is simply a evolution in response to the other
| side also having evolved greatly, both legitimate and
| illegitimate users.
| unethical_ban wrote:
| I don't understand. What exactly are they doing, what are
| their goals? I'm not trying to argue, I genuinely don't get
| it.
|
| edit: I guess I understand "AI bots scraping sites for data
| to feed LLM training" but what about the image serving?
| immibis wrote:
| How many requests is your site getting, and how long does
| your site require to process a request, and why is it that
| long?
| UltraSane wrote:
| Can you use per-IP rate limiting?
| zzzeek wrote:
| ~~two~~ three comments on that:
|
| 1. DDOS protection is not the only thing anymore, I use
| cloudflare because of vast amounts of AI bots from thousands
| of ASNs around the world crawling my CI servers (bloated Java
| VMs on very undersized hosts) and bringing them down
| (granted, I threw cloudflare onto my static sites as well
| which was not really necessary, I just liked their analytics
| UX)
|
| 2. the XKCD comic is mis-interpreted there, that little block
| is small because it's a "small open source project run by one
| person", cloudflare is the opposite of that
|
| 3. edit: also cloudflare is awesome if you are migrating
| hosts, did a migration this past month, you point cloudflare
| to the new servers and it's instant DNS propagation (since
| you didnt propagate anything :) )
| dboreham wrote:
| Why are your CI servers open to the public network?
| zzzeek wrote:
| because we're an open source project that accepts pull
| requests on github and we'd like our PR submitters to see
| why their PRs are failing tests
| ramon156 wrote:
| Last time I tried this I got DDoS'd so I don't see a reason
| to step away from CF. That said, this is the price I pay
| davedx wrote:
| All my stuff is working. Things on GCP. Things on Fly.io.
| Tooling I use.
|
| "Only" 10% of the internet is behind Cloudflare so far ;)
| grabcadder wrote:
| Happy for you :)
|
| I am curious about these two things:
|
| 1- Does GCP also have any outages recently similar to AWS,
| Azure or CF? If a similar size (14 TB?) DDoS were to hit GCP,
| would it stand or would it fail?
|
| 2- If this DDoS was targeting Fly.io, would it stand? :)
| progbits wrote:
| For GCP network that would be a rounding error. Of course
| GCP sometimes has outages too, all providers do.
| davedx wrote:
| I actually spoke too soon, and accept I have egg on my
| face!
|
| Apparently prisma's `npm exec prisma generate` command
| tries to download "engine binaries" from
| https://binaries.prisma.sh, which is behind... guess
| what...
|
| So now my CI/CD is broken, while my production env is down,
| and I can't fix it.
|
| Amazing lol
| compumike wrote:
| Could always just use a status page that updates itself. For my
| side project Total Real Returns [1], if you scroll down and
| look at the page footer, I have a live status/uptime widget [2]
| (just an <img> tag, no JS) which links to an externally-hosted
| status page [3]. Obviously not critical for a side project, but
| kind of neat, and was fun to build. :)
|
| [1] https://totalrealreturns.com/
|
| [2]
| https://status.heyoncall.com/svg/uptime/zCFGfCmjJN6XBX0pACYY...
|
| [3] https://status.heyoncall.com/o/zCFGfCmjJN6XBX0pACYY
| jcfrei wrote:
| This is unrelated to the cloudflare incident but thanks a lot
| for making that page. I keep checking it from time to time
| and it's basically the main data source for my long term
| investing.
| compumike wrote:
| I appreciate that, thank you! :)
| touristtam wrote:
| Thankfully the usual social media are still up ... oh wait
| https://www.bbc.co.uk/news/articles/c629pny4gl7o
| talonx wrote:
| BetterStack did report issues with some of their services, but
| they were not very informative.
| nnyms1 wrote:
| I got several emails from some uptime monitors I setup due to
| failing checks on my website and funnily enough I cannot log into
| any of them.
|
| BetterStack, InStatus and HetrixTools seemingly all use
| Cloudflare on their dashboards, which means I can't login but I
| keep getting "your website/API is down" emails.
|
| Update: I also can't login to UptimeRobot and Pulsetic. Now, I am
| getting seriously concerned about the sheer degree of
| centralization we have for CDNs/login turnstiles on Cloudflare.
| MrKh4n wrote:
| What is happening to cloudflare, anybody knows? Everything is
| down!
| chromehearts wrote:
| And here I was wondering why my website shut down & why I
| couldn't tweet about it
| prityush14 wrote:
| when ig went down, I came to X to post Now that X is down, is hn
| the place to be
| m11a wrote:
| So ... any bets the cause isn't DNS?
| pc2g4d wrote:
| We still doing BGP update typos?
| marcosdumay wrote:
| Nope. Looks like they have a DNS-like configuration manager...
| reassess_blind wrote:
| Who wants to join me at the Winchester for a pint, and wait for
| this all to blow over?
| Dilettante_ wrote:
| Got some red on you...
| enceeee wrote:
| poland down
| BilalBudhani wrote:
| everything is down except HN :D
| jsheard wrote:
| The just-one-big-server-in-someones-basement stack remains
| undefeated.
| gjsman-1000 wrote:
| I don't know; HN historically has had way worse uptime than
| Cloudflare.
| sph wrote:
| [citation needed]
| mambru wrote:
| Except it isn't that big?
| allanmacgregor wrote:
| Digg.com is working perfectly hahahaha
| mrweasel wrote:
| It's a little surprising how little it affect me. I believe
| it's around 20% of websites that use CloudFlare in some form or
| another.
| 52-6F-62 wrote:
| Bunnycdn lives
| mlashcorp wrote:
| Pagerduty is up
| ask2sk wrote:
| Yes, all sites are down. Getting a 500 error from India.
|
| Update: Looks like the issue has been resolved now. All sites are
| operational now.
| enceeee wrote:
| Poland down
| bobbywalnuts wrote:
| Down in Australia and New Zealand
| mak8 wrote:
| it took X , supabase down as well.
| kgwxd wrote:
| My Window System seems to be working fine.
| taneq wrote:
| Aw man, how dare this affect me personally? :P (Tried to get to
| openstreetmap.org which is behind cloudflare.)
| baccatore wrote:
| Tokyo too
| elborbentley wrote:
| Down... "Please unblock challenges.cloudflare.com to proceed." On
| every Cloudflare hosted website that I try. This timing
| SUCKS.......... please resolve fast! <3
| binanabread wrote:
| aaa the only night im free to watch nana and cloudfare sabotages
| me T_T
| chistev wrote:
| How come HN is never down with all these outages?
| meeb wrote:
| Because HN doesn't use Cloudflare.
| chistev wrote:
| Also doesn't use AWS or Azure because it didn't go down with
| them either.
| us0r wrote:
| I thought the same until I started looking:
|
| Name Server: NS-225.AWSDNS-28.COM Name Server:
| NS-1411.AWSDNS-48.ORG Name Server: NS-1914.AWSDNS-47.CO.UK
| Name Server: NS-556.AWSDNS-05.NET
|
| At least for DNS. Data center appears to be Lightedge.
| Bender wrote:
| HN sometimes _rarely_ is pointed to AWS when under attack.
| ardfard wrote:
| Does HN self-host too?
| ArcHound wrote:
| I'd love to read an article describing the HN setup. Seems
| that they got a lot of things right - self registration,
| influx of people during outages and plenty others. Admins,
| if you see this, please write about your craft!
| Akronymus wrote:
| I am pretty sure @dang wrote about it on here before, but
| I can't seem to find it.
| b2ccb2 wrote:
| Here https://news.ycombinator.com/item?id=28478379 and
| https://news.ycombinator.com/item?id=27452276
| ArcHound wrote:
| Thanks! Seems that's what we'll get as there are
| apparently serious money which can be made by knowing the
| guts of HN as per
| https://news.ycombinator.com/item?id=27454354.
| ArcHound wrote:
| Seems like so, these submissions from them look related,
| but not quite it:
|
| - [Sorry I broke the
| server](https://news.ycombinator.com/item?id=9052128)
|
| - [New attempt at mobile
| markup](https://news.ycombinator.com/item?id=10489499)
|
| - [Clickable domains and
| QoL](https://news.ycombinator.com/item?id=10223645)
|
| - [New features and a
| moderator](https://news.ycombinator.com/item?id=12073675)
|
| - [Thanks to thehodge and littlewarden, this site is up
| today](https://news.ycombinator.com/item?id=28472350)
|
| Maybe one day. Seeing all of these big providers
| stumbling, an article about HN staying on top of
| everything would surely resonate.
| dang wrote:
| Yes, except when we've shot ourselves in both feet:
|
| https://news.ycombinator.com/item?id=45750608
| sph wrote:
| How? It's literally impossible to run a major website these
| days without Cloudflare.
|
| /s
|
| This is what you get for being lazy and choosing to making
| the internet more centralized.
| Iulioh wrote:
| Well, to be fair, while recognizing how influential it is,
| HN is not a major website lol
| chistev wrote:
| What counts as major? I estimate they get millions of
| hits here
| Bender wrote:
| HN is just one active and one standby server at M5 Computer
| Security running BSD.
| pmlnr wrote:
| There are things out there which are running from a bare metal
| host, without relying on someone else's computer (aka the
| cloud). HN is one of them.
| duckerduck wrote:
| They stopped using Cloudflare some time ago
|
| https://news.ycombinator.com/item?id=18188832
| rwaksmunski wrote:
| FreeBSD on bare metal hooked up to a nice network.
| philipwhiuk wrote:
| HN is running on the server the rest of the cloud rents time
| from.
|
| The outages are the Roomba.
| materikzen wrote:
| facing cloudflare downtime
| gbil wrote:
| The irony is that if you follow the relevant link [1]in the error
| page , you get this
|
| > If the problem isn't resolved in the next few minutes, it's
| most likely an issue with the web server you were trying to
| reach.
|
| [1] https://www.cloudflare.com/5xx-error-
| landing/?utm_source=err...
| jpartridge1 wrote:
| using a cloudflare tunnel for local dev work, completely down
| gbajson wrote:
| Poland. Most of the popular sites are down. Including community
| forum on Cloudflare.
| djsjajah wrote:
| I went to check how many services are being impacted on down
| detector, but it was down.
| stachoo wrote:
| Warsaw, Los Angeles and Newark down
| greatgib wrote:
| What is funny us that on their global status list for services,
| everything looks green except "network" that is "offline".
| virajk_31 wrote:
| I was shouting at network guy/colleague, how come
| challenges.cloudflare.com got blocked!! damn, I must apologise to
| him.
| yipbub wrote:
| Probably better not to shout in the first place.
| virajk_31 wrote:
| It was friendly fire, nothing serious. haha
| weird-eye-issue wrote:
| > friendly fire
|
| Yeah I don't think you are using this phrase correctly
| virajk_31 wrote:
| Just trying to fit the jargons...
| bofadeez wrote:
| That reply doesn't make sense either. Sounds like you're
| just a "bad person" haha
| Fiveplus wrote:
| is that how someone gets cancelled?
| souvlakee wrote:
| ..that's how they canceled a good guy based on telepathic
| diagnostics.
| weird-eye-issue wrote:
| Yeah this guy got completely bullied lol it's all a
| little bit ironic
| dagss wrote:
| "Friendly banter" is the word you are looking for.
| prodent wrote:
| Friendly fire is when you shoot someone from your own team.
| enidjv wrote:
| Don't worry beer gonna fix everything
| uniq7 wrote:
| Even if he blocked it by accident, that is not a reason to
| shout.
|
| Shouting will not prevent errors, and you are only creating a
| hostile work environment where not acting is better than the
| risk of making a mistake and triggering an aggressive response
| from your part.
| virajk_31 wrote:
| It wasn't aggressive exchange, but will definitely consider
| your comment.
| mlrtime wrote:
| There is nothing else to do since CF is down... so.
|
| There is nothing wrong with shouting during a perceived
| outage. Shouting is just raising your voice to give a notion
| of urgency. Yelling is different.
|
| How often have you heard "shout at me", or something like
| that?
|
| OP, continue you to shout when its needed, just don't yell at
| people you work with ;)
| davidmurdoch wrote:
| If he needs to apologize then his shouting was of the
| yelling kind
| KabuTheSilicon wrote:
| akxeder.eth.ac is working
| jpartridge1 wrote:
| ironically downdetector.com is down because they use cloudflare
| for challenges
| rvz wrote:
| Probably a good time to contact the CEO of Cloudflare.
|
| Looking forward to the post-mortem.
| punk_hazard wrote:
| It also took chatgpt and claude , trying to access from pk
| dodyg wrote:
| chatgpt.com is not working because they are relying on cloudflare
| for challenges
| conradfr wrote:
| I thought I would be clever by switching domain endpoints from
| proxied to dns but Cloudflare admin page is also not working
| correctly ;)
|
| edit: it's up!
|
| edit: it's down!
| buybackoff wrote:
| The main bike rental Velib in Paris has the app not working, but
| the bikes can be taken with NFC. However, my station, which is
| always full at this time, is now empty, with only 2 bad bikes. It
| maybe related. Yet, push notifications are working.
|
| I'm going to take the metro now and thinking how long do we have
| until the entire transit network goes down because of a similar
| incident.
| terezatizkova wrote:
| Europe down
| varshneydevansh wrote:
| I was about to cream watching my fav video on X and it is down
| donglong wrote:
| genuinely makes me sad for the people there. this must be a
| living nightmare right now.
| vntok wrote:
| Why? If any company has enough technical people, resources &
| processes in place it must be them, no?
| bongocot wrote:
| Down in Taiwan
| ehsamm wrote:
| YESS
| password999 wrote:
| 500 from Thailand
| finlayy wrote:
| Ironic, cloudflare taking downdetector with it
| ehsamm wrote:
| yess ddos
| password999 wrote:
| 500 from thailand
| miiniiiie wrote:
| Feels like 25% of the Internet is down just because of fuckin'
| cloudflare.
|
| I'm leaving the redaction because I couldn't work atm...
|
| Time for a beer , greetings from germany!
| Blivpedia wrote:
| I thought it was problem with my network lol XD
| dmezzetti wrote:
| This one is bigger than the AWS East outage...
| finlayy wrote:
| Ironic, Cloudflare taking DownDetector with it
| Odablock wrote:
| Forced to play Runescape now
| ehsamm wrote:
| I learned from reliable sources about a denial-of-service attack;
| everything went down.
| ahmadrifai123 wrote:
| Indonesia down also
| GooniestGoon wrote:
| Bruh I was watching porn on Twitter, I thought the FBI got my
| ahh.
| benedictboppler wrote:
| username fits...
| SallysField wrote:
| Who the fuck talks like this
| mak8 wrote:
| It took supabase and X down for me
| KabuTheSilicon wrote:
| akxeder.eth.ac is not down
| acomjean wrote:
| The non profit I volunteer at is unreachable. It gives a
| cloudflare error page which is sort of helpful. It tells me the
| the site is ok but cloudflare has an 500.
|
| It's been great, but I always wonder when a company starts doing
| more than it's initially calling. There have been a ton of large
| attacks, tons of bot scrappers so it's the Wild West.
| jpartridge1 wrote:
| yes they're spreading themselves very thin with lots of new
| releases/products - but they will lose a lot of customers if
| their reliability comes into question
| MTGR wrote:
| Such a shame though. I wonder how long it's going to take before
| they bring it back up
| neya wrote:
| The biggest learning for me from this incident - NEVER make your
| DNS provider and CDN provider the same vendor. Now, I can't login
| into the dashboard, even to switch the DNS. Sigh.
| rollulus wrote:
| Classic. I see issues. Vendor's status page is all green. Go to
| HN to find the confirmation. Applies to AWS, GH, everyone.
|
| Edit: beautiful, this decentralised design of the internet.
| markild wrote:
| I get the feeling that all "serious" businesses have manual
| processes for publicly facing status pages, for political
| reasons.
|
| I don't like it.
| bnjm wrote:
| SLA breaches have consequences, no big conspiracy there
| markild wrote:
| Not at all saying it's a conspiracy, I just think it's a
| lack of transparency.
|
| I get why, but it would give me more confidence if they
| would tell me about everything.
| mewpmewp2 wrote:
| I guess a dirty little secret might be that something is
| always acting up or being noisy and it would spam the
| status page completely.
| zulban wrote:
| They don't make more money by giving you more confidence
| in their systems.
| skywhopper wrote:
| At some level, the status updates have to be manual. Any
| automation you try to build on top is inevitably going to
| break in a crisis situation.
| mlrtime wrote:
| They are manual AND political (depending on how big the
| company is). Because having a dashboard go to red usually
| has a bunch of project work behind it.
| pimterry wrote:
| I found GitHub's old "how many visits to this status page
| have there been recently" graph on their status page to be
| an absurdly neat solution to this.
|
| Requires zero insight into other infrastructure, absolutely
| minimal automation, but immediately gives you an idea
| whether it's down for just you or everybody. Sadly now
| deceased.
| Kodiack wrote:
| I like that https://discordstatus.com/ shows the API
| response times as well. There's times where Discord will
| seem to have issues, and those correlate very well with
| increased API response times usually.
|
| Reddit Status used to show API response times way back in
| the day as well when I used to use the site, but they've
| really watered it down since then. Everything that goes
| there needs to be manually put in now AFAIK. Not to
| mention that one of the few sections is for
| "ads.reddit.com", classic.
| tom1337 wrote:
| https://steamstat.us still has this - while not official
| it's pretty nice.
| sjsdaiuasgdia wrote:
| Yeah, this is something people think is super easy to
| automate, and it is for the most basic implementation of
| something like a single test runner. The most basic
| implementation is prone to false positives, and as you say,
| breaking when the rest of your stuff breaks.
|
| You can put your test runner on different infrastructure,
| and now you have a whole new class of false positives to
| deal with. And it costs you a bit more because you're
| probably paying someone for the different infra.
|
| You can put several test runners on different
| infrastructure in different parts of the world. This
| increases your costs further. The only truly clear signals
| you get from this are when all are passing or all are
| failing. Any mixture of passes and fails has an opportunity
| for misinterpretation. Why is Sydney timing out while all
| the others are passing? Is that an issue with the test
| runner or its local infra, or is there an internet event
| happening (cable cut, BGP hijack, etc) beyond the local
| infra?
|
| And thus nearly everyone has a human in the loop to
| interpret the test results and make a decision about
| whether to post, regardless of how far they've gone with
| automation.
| Aurornis wrote:
| I've written before on HN about when my employer hired
| several ex-FAANG people to manage all things cloud in our
| company.
|
| Whenever there was an outage they would put up a fight
| against anyone wanting to update the status page to show the
| outage. They had so many excuses and reasons not to.
|
| Eventually we figured out that they were planning to use the
| uptime figures for requesting raises and promos as they did
| at their FAANG employer, so anything that reduced that uptime
| number was to be avoided at all costs.
| cj wrote:
| Are there companies that actually use their statuspage as a
| source of truth for uptime numbers?
|
| I think it's way more common for companies to have a public
| status page, and then internal tooling that tracks the
| "real" uptime number. (E.g. Datadog monitors, New Relic
| monitoring, etc)
|
| (Your point still stands though.)
| Aurornis wrote:
| I don't know, but I will say that this team that was
| hired into our company was so hyperfocused on any numbers
| they planned to use for performance reviews that it
| probably didn't matter which service you chose to measure
| the website performance. They'd find a way to game it. If
| we had used the internal devops observability tools I bet
| they would have started pulling back logging and reducing
| severity levels as reported in the codebase.
|
| It's obviously not a problem at every company because
| there are many companies who will recognize these
| shenanigans and come down hard on them. However you could
| tell these guys could recognize any opportunity to game
| the numbers if they thought those numbers would come up
| at performance review time.
|
| Ironically our CEO didn't even look at those numbers. He
| used the site and remembered the recent outages.
| darccio wrote:
| [Datadog employee here] https://updog.ai tracks the
| uptime of multiple services by real impact across Datadog
| customers.
| mvkel wrote:
| It's because if you automate it, something could/would happen
| to the little script that defines "uptime," and if that goes
| down, suddenly you're in violation of your SLA and all of
| your customers start demanding refunds/credits/etc. when
| everything is running fine.
|
| Or let's say your load balancer croaks, triggering a "down"
| status, but it's 3am, so a single server is handling traffic
| just fine? In short, defining "down" in an automated way is
| just exposing internal tooling unnecessarily and generates
| more false positives than negatives.
|
| Lastly, if you are allowed 45 minutes of downtime per year
| and it takes you an hour to manually update the status page,
| you just bought yourself an extra hour to figure out how to
| fix the problem before you have to start issuing
| refunds/credits.
| dogleash wrote:
| >you just bought yourself an extra hour to figure out how
| to fix the problem before you have to start issuing
| refunds/credits
|
| No. Not if you're not defrauding your customers, you
| didn't.
| mrgoldenbrown wrote:
| FWIW, cloudflare's status page is showing red currently.
| agos wrote:
| I usually get notifications from the sales/CS team way before
| the status page/incident list has any blip. This time was not
| an exception
| webdoodle wrote:
| It's as if they wanted an internet kill switch. /S
| Bro1309 wrote:
| All Cloudflare websites are down!!!! When will it get fixed? I
| dont have time.!!
| miiniiiie wrote:
| Couldn't work. Fuckin' cloudflare . Feels like 25% of the
| Internet is down.
|
| Im going home. Time for a beer .
|
| Greetings from germany
| saubeidl wrote:
| Cloud in general was a mistake. We took a system explicitly
| designed for decentralization and resilience and centralized it
| and created a few neat points of failure to take the whole damn
| thing down.
| is_true wrote:
| It took me a while to understand it, but the beauty of it is
| that when it fails, lot of things fail.
|
| Almost no one gets mad if your site and half the internet were
| down.
| saubeidl wrote:
| Sure, but that is also a giant weakness. Say in a future
| conflict with Russia or China, or hell, even North Korea.
|
| They'd only have to take down a few services to completely
| cripple the West - the exact case ARPANET was designed to
| prevent.
| is_true wrote:
| Yep. You are right. Aren't those services regulated to
| avoid that from happening?
| mosselman wrote:
| Cloudflare provides some nice services that have nothing to do
| with cloud or not. You can self-host private tunnels,
| application firewalls, traffic filtering, etc, or you can focus
| on building your application and managing your servers.
|
| I am a self-host enthousiast. So I use Hetzner, Kamal and other
| tools for self-managing our servers, but we still have
| Cloudflare in front of them because we didn't want to handle
| the parts I mentioned (yet, we might sometime).
|
| Calling it a mistake is a very narrow look at it. Just because
| it goes down every now and then, it isn't a mistake. Going for
| cloud or not has its trade-offs and I agree that paying 200
| dollars a month for a 1GB Heroku Redis instance is complete
| madness when you can get a 4GB VPS on Hetzner for 3,8 a month.
| Then again, some people are willing to make that trade-off for
| not having to manage the servers.
|
| Cloud servers have taught me so much about working with servers
| because they are so easy and cheap to spin up, experiment with
| and then get rid of again. If I had had to buy racks and host
| them each time I wanted to try something, I would've never done
| it.
| locallost wrote:
| Thanks, I was too lazy to write this, and noticed this
| comment multiple times now. It's good to be sceptical at
| times, but in this case it simply misses the mark.
| saubeidl wrote:
| Sure, it's a great fair-weather technology, makes some things
| cheap and easy.
|
| But in the face of adversity, it's a huge liability. Imagine
| Chinese Hackers taking down AWS, Cloudflare, Azure and GCP
| simultaneously in some future conflict. Imagine what that
| would do to the West.
|
| I don't believe in Fukuyamas End of History. History is still
| happening, and the choices we make will determine how it
| plays out.
| nijave wrote:
| Threat actors (DDoS) and AI scraping already threw a wrench in
| decentralization. It's become quite difficult to host anything
| even marginally popular without robust infrastructure that can
| eat a lot of traffic
| firecall wrote:
| 10.30pm here in Australia...
|
| and my alarms are going off my and support line is ringing...
|
| I cant even login to my CF dashboard to disable the CDN!
|
| Edit: It's back. Hopefully it will stay up!
|
| Edit 2: 1 Hour Later.
|
| Narrator: It didn't stay up :/
| jpdus wrote:
| germany as well. Claude down too
| Culonavirus wrote:
| X, Chatgpt, all kinds of sites and services around the eu, it's
| a massive outage
| ReedorReed wrote:
| There is an election in Denmark today, I wonder if this will
| affect that. The governments website is not accessible at the
| moment because it uses Cloudflare.
| baaron wrote:
| My tinfoil hat has me wondering if it's just coincidence.
| orangeyjuicey wrote:
| So do we have a guarantee that posts are not made by AI for a few
| minutes?
| tzpbingo wrote:
| 20% of websites worldwide are down.
| ramon156 wrote:
| source? would love to see
| ed_mercer wrote:
| I got an invoice from them right before the outage. Hopefully
| when they restore everything, they'll have forgotten about it!
| MTGR wrote:
| It's back up, sites are working. Still wonder how long it's going
| to last. IF there's another blackout.
| Bro1309 wrote:
| sites aint working in india
| coolgang wrote:
| yes bro
| thearnabsarkar wrote:
| Did something happen to the Cloudflare lava lamps ?
| coolgang wrote:
| even in india cloud fare has a outage
| Brajeshwar wrote:
| Ah! Well, all of my websites are down! I'm going to take
| screenshots and have it as part of my Time Capsule Album, "Once
| upon a Time, my websites used to go down."
| KabuTheSilicon wrote:
| Once upon a time, the end of the world happened
| jameslk wrote:
| Cloudflare's own status page is not responding. I guess it's down
| too?
| udev4096 wrote:
| A good reminder for advancing decentralization and p2p networks!
| cipixul wrote:
| Bucharest Cloudflare down too
| coolgang wrote:
| even in India there is cloudfare's outage
| StefanThorpe wrote:
| Strange thing is this is in multiple CD regions all using bot &
| WAF are down, just got a colueuge to check our site and both
| London & Singapour cloudflare servers are out... And I cant even
| login to the cloudflare dash to re-route critical traffic .
| Likely this is accidental, but one day there will be something
| malicous that will have big impacts with how centralised the
| internet now is.
| clownelon wrote:
| Who is laughing now Elon ?
| haik90 wrote:
| It is a relief that they hosted the status page on someone else's
| infrastructure.
| AL1033 wrote:
| yes absolutly yes, i have tried severak region and all of then
| receive 500 error
| diginova wrote:
| https://www.cloudflarestatus.com/
|
| even status page is giving 504 Gateway Timeout ERROR The request
| could not be satisfied. now in India
| 3xstphvs wrote:
| Australia here. plenty down rn
| coolgang wrote:
| LETS GO ITS BACK ONN
| coolgang wrote:
| guys hype up its backkk
| cf_py wrote:
| Didn't have my site on cloudflare bc it would be faster for
| chinese users (its main demographic) so i THOUGHT i was fine for
| a second until i remembered the data storage api is behind
| cloudflare
| cipixul wrote:
| claude.ai down too... lots of programmers are gonna have to
| pretend they code in another way...
| wesrobin wrote:
| It's working fine for me
| cipixul wrote:
| Yep, seems to work cause my nearby colleague started
| copy/pasting massive chunks of code again.
|
| I guess claude is more important than your average site :)
| re5i5tor wrote:
| For me right now, Claude.ai is down, but Claude Code
| (terminal, extension) seems to be up and happy. Suggests that
| API is probably up.
| re5i5tor wrote:
| Ironically, Claude Desktop--which could be up if it simply
| used the API instead of doing the Electron webapp wrapper
| thing--is down.
| nrhrjrjrjtntbt wrote:
| Easy: "the site is done, it is fantastic, but cloudflare is
| down so you cant see it"
| sharathnryn wrote:
| the sheer number of websites this is taken down!
| testingofindia wrote:
| even popular apps like x.com, chatgpt.com are down.
| tebbers wrote:
| My uptime monitor OnlineOrNot is also down...
| rozenmd wrote:
| OnlineOrNot's fallen back to AWS for monitoring, so you should
| still be getting alerts.
|
| The dashboard's API server runs on Cloudflare and is currently
| blocking all logins, will fix.
| testingofindia wrote:
| even the famous applications like Chatgpt, x.com are down
| forthaven wrote:
| I'm betting on DNS fail
| Inheritor wrote:
| Well that was quick. I saw a status saying server maintenance.
| And then it changed to "we're looking into this" must've made an
| oopsie I suppose
| ChewbyP wrote:
| Just happy I wasnt IP banned lol
| forthaven wrote:
| I'm betting on another DNS failure
| Bro1309 wrote:
| Its back!
| bazinga1020 wrote:
| asd
| raxxorraxor wrote:
| Seems to work again. 40min downtime for many services it seems.
| pc2g4d wrote:
| I was reading up on home lab server racks, and every single site
| is down with a Cloudflare error. So much for DIY!
| codeaholicguy wrote:
| Yes, it impacted our services https://www.cloudflarestatus.com/
| Dukeares wrote:
| Singapore Cloudeflare Server is down.
| tristanperry wrote:
| ChatGPT and Claude are down as a result, too.
|
| Y'know, along with most other SAAS services.
| miiniiiie wrote:
| ELON! GO AND KICK THOSE CLOUDFLARE ASSES!
|
| or search a new job for yourself. Maybe digging to the earth
| core. Why? Idk. Because then you can say: I did it, or so.
| crazyooly wrote:
| We really do have two surprise holidays every year: AWS Day and
| Cloudflare Day. Happy outages, everyone.
| 3xstphvs wrote:
| it works, then stops, then works
| MrPasancalla wrote:
| Just when I was assigned a task yesterday but decided to do it
| today early morning.
| sansiro9 wrote:
| Oh, look! Cloudflare is down. Let's check down detector to make
| sure it's not just me > Downdetector is using Cloudflare captcha.
| Yep, it's down.
| Tixx7 wrote:
| some sites are already up again, including the cf dash and
| downdetector, both ironically down a few minutes ago
| Woods369 wrote:
| syntax error: unexpected semicolon
| fungungun wrote:
| its back !
| adrianhacar wrote:
| The sites I host on Cloudflare are all down. Also, even ChatGPT
| was down for a while, showing the error: "Please unblock
| challenges.cloudflare.com to proceed."
| dsco wrote:
| This is still the case for me
| pelagicAustral wrote:
| Things seem to be coming back up... been almost 45 minutes, since
| my first alert came at 0836
| djiexpert wrote:
| My site https://mediamistrz.pl/ not working
| vednig wrote:
| hacked.stream was down too
| ChubbyChief wrote:
| Down is Missouri
| philipwhiuk wrote:
| Looks like the status page is suffering too because it can't load
| jQuery:
|
| (index):64 Uncaught ReferenceError: $ is not defined at
| (index):64:3
| iamawacko wrote:
| Wow, so much is down. Nothing Cloudflare protected is loading for
| me in Indiana, and the Cloudflare dashboard is broken as well.
|
| I hope it gets resolved in the next hour or two, or it could be a
| serious problem for me.
| mpaepper wrote:
| Seems like the merging with Replit didn't work so well :p
| vxworkss wrote:
| Glad to see things are actually working here! Also, my website
| (halomate.ai) is using CF too, and surprisingly, it's working
| fine as well
| duncans wrote:
| You spoke too soon!
| nhatcher wrote:
| Off topic, but the 500 page from prusa3d is quite good:
|
| https://www.prusa3d.com/
|
| https://imgur.com/a/OW5KL8r
| mig4ng wrote:
| Cloudflare Dashboard/Clicky clicky UI is down. I really
| appreciate that their API is still working. Small change in our
| Terraform configuration and now I can go lunch in peace knowing
| our clients at skeeled can keep working if wanted:
|
| resource "cloudflare_dns_record"
|
| - proxied = true
|
| + proxied = false
| NicoJuicy wrote:
| Seems to be back up
| elcapitan wrote:
| Black HN ribbon for the Internet
| tristanperry wrote:
| I assume you're joking, but as an FYI, Rebecca Heineman died:
|
| https://www.pcgamer.com/gaming-industry/legendary-game-desig...
| theideaofcoffee wrote:
| Love that for them. Congrats on building such a re-centralized
| web!
| rvz wrote:
| Cloudflare is now a systemic risk for a state-sponsored
| attacker to bring down the entire web.
| ntoskrnl_exe wrote:
| It's been 45 minutes and I'm already looking forward to the day
| Kevin Fang makes a video about this
| xprice wrote:
| at least https://xprice.ro is up, dont know how and why cuz we
| use cloudflare and we're hosting in germany/hetzner
| namannarula wrote:
| down for india :/
| xprice wrote:
| interesting, working in Romania. So it's somehow related to
| they geo-balancing infrastructure.
| tgv wrote:
| Seems to be over.
| ravikinhajaat wrote:
| anyone have reference for flutter opening +917494920753
| mittermayr wrote:
| Even Cloudflare Status is now down, oh boy :)
| https://postimg.cc/LJVKYmks
| carter-0 wrote:
| Even your postimg.cc link is down for me.. (at least their CSS
| is)
|
| https://ibb.co/QF6X0pX9
| mittermayr wrote:
| amazing
| hirako2000 wrote:
| Postimg's CDN is down
| HocusLocus wrote:
| catbox.moe is up
|
| https://files.catbox.moe/9r3zgr.png
| wazoox wrote:
| I discovered the problem by trying to access
| https://downforeveryoneorjustme.com/ ironically :)
| martiuk wrote:
| API still seems to work if you already have a script to hand to
| unproxy everything.
| kumarski wrote:
| I'm going to buy more.
| chid wrote:
| Took quite a while for it to show up oddly.
| joemonsters wrote:
| Now I can switch everything off and go home. We are not using CF
| at our site, but CF error it is a good reason to have a day off
| dheatov wrote:
| Seems like coudflare activated the maximum llm-scraper-bot-
| protection for everyone.
| digid8a wrote:
| just yesterday cloudflare announced it was acquiring replicate
| (ai to "help" it's workers) i believe
| nrhrjrjrjtntbt wrote:
| You
|
| Browser Working
|
| San Jose
|
| Cloudflare Error
|
| mysite.com
|
| Host Working
|
| Lol! Like a solar eclipse!
| bpavuk wrote:
| Ukraine. Sporadic outages as well. Error pages blame Cloudflare
| Warsaw servers.
| hamish-b wrote:
| What a wild ride, the traffic to my site is more akin to a
| rollercoaster. Got better for a few mins and then fell back
| apart.
| kosolam wrote:
| Still ongoing. Some requests going through. Some get the cf error
| page.
| Fiveplus wrote:
| Cloudflare is the real backbone of the internet in 2025. It
| should be a globalized property like ICANN or something
| aeve890 wrote:
| How would that prevent outages? Honest question
| ahmedhossamdev wrote:
| lol
| dheatov wrote:
| Reminder that this is not the web we want.
| yaratixal wrote:
| It's down worldwide practically, in the US,UK,NZ,AU
| ArvindRSingh wrote:
| Even IN.
| vednig wrote:
| https://hacked.stream/
| seydor wrote:
| More proof that central planning doesn't work
| sidcool wrote:
| Even Twitter is down. Most of my customers are shouting at the
| top of their head!
| sidcool wrote:
| The top black bar is appropriate /s
| Fiveplus wrote:
| Even twitter is gone. Where will I post memes mocking cloudflare?
| gethly wrote:
| Ha ha ha hahahahahaaaa hahahahahaaaahahaha, fuck 'em.
| Dilettante_ wrote:
| Our support portal provider is currently experiencing issues
|
| Are they using Cloudflare perchance? (scnr)
| raghava wrote:
| Most down-detectors are down due to their dependency on CF.
|
| Cloudflare Mumbai, Bengaluru, Chennai, Hyderabad edge-nodes also
| unable to serve content.
|
| x.com down.
|
| Few quick-commerce apps are acting up at times.
| import wrote:
| Funny that their status page shows almost all locations
| "Operational" but they're not. Are they updating the page
| manually and keep it green?
| kuylar wrote:
| I assume the locations are operating fine, since you can see
| the error pages. The culprit here is probably the Network,
| which at the time of writing, shows up as offline
| lilppdavidson wrote:
| I'm assuming hard rock (bet) is run by cloudflare also
| azholder wrote:
| what's a good alternative for their WAF, that isn't enterprise
| expensive?
| nrhrjrjrjtntbt wrote:
| Is it DNS or BGP?
| meepmorp wrote:
| Why settle for just one?
| hypeatei wrote:
| Why do people use the reverse proxy functionality of Cloudflare?
| I've worked at small to medium sized businesses that never had
| any of this while running public facing websites and they
| were/are just fine.
|
| Same goes for my personal projects: I've never been worried about
| being targeted by a botnet so much that I introduce a single
| point of failure like this.
| xyzzy9563 wrote:
| It gives really good caching functionality so you can have
| large amounts of traffic and your site can easily handle it.
| Plus they don't charge for egress traffic.
| arccy wrote:
| people think that running nodejs servers are a good idea, and
| those fall over if there's ever so much as a stiff breeze, so
| they put cloudflare in front and call it a day.
| lofaszvanitt wrote:
| It's chic. Young bois or adult pepl with boi like mentality.
|
| What, they have Cloudflare and we don't? We also must have
| cloudflare. Don't ask why.
|
| Now that you have it, you are at least level 15 and not a
| peasant.
|
| Same applies to every braindead framework on the web. The
| gadget mind of the bois is the cause for all this.
| thoroughburro wrote:
| I'm surprised your projects aren't plagued by massive waves of
| scraping traffic like the rest of us. Count yourself lucky, not
| superior.
| q3k wrote:
| What exactly are you serving that bot traffic affects your
| quality of service?
|
| I've seen an RPi serve a few dozen QPS of dynamic content
| without issue... The only service I've had actually get
| successfully taken down by benign bots is a Gitea-style git
| forges (which was 'fixed' by deploying Anubis in front of
| it).
| pqdbr wrote:
| Any project that starts gaining any bit of traction get's
| hammered with bots (the ones that try every single /wp url even
| tough you don't even use Wordpress), frequent DDoS attacks, and
| so on.
|
| I consider my server's real IP (or load balancer IP) as a
| secret for that reason, and Cloudflare helps exactly with that.
|
| Everything goes through Cloudflare, where we have rate
| limiters, Web firewall, challenges for China / Russian inbound
| requests (we are very local and have zero customers outside our
| country), and so on.
| digid8a wrote:
| just yesterday cloudflare announced it was acquiring replicate
| (ai platform) "the Workers Platform mission: Our goal all along
| has been to enable developers to build full-stack applications
| without having to burden themselves with infrastructure"
| according to cloudflare's blog, are we cooked?
| lilppdavidson wrote:
| I'm assuming Hard Rock (Bet) is run by Cloudflare too
| azholder wrote:
| Lots of valid concern about us all using CF, but is their an
| alternative to their WAF that isn't enterprise expensive?
| PhilippGille wrote:
| Depends on your needs, but for example there's Bunny Shield:
| https://bunny.net/shield/
| azholder wrote:
| Thanks for the pointer. They'd still wind up being a couple
| thousand dollars more annually than what we pay CF now.
|
| Frustrating, because I know I'll get asked today if we have
| an alternative to using CF, and I don't have a good answer.
| ovx wrote:
| Yep, bunny.net is great, we also use it. And look at
| https://altcha.org as a Turnstile replacement
| Fiveplus wrote:
| For fun, I asked google what's an alternative to Cloudflare. It
| says, "A complete list of Cloudflare alternatives depends on
| which specific service (CDN, security, Zero Trust, edge
| computing, etc.) you are replacing, as no single competitor
| offers the exact same all-in-one suite"
| chr15m wrote:
| Imagine using an all-in-one suite.
| ArvindRSingh wrote:
| Using Cloudflare is a tradeoff between facing DDoS and other
| attacks, and the downtime of Cloudflare.
| Fiveplus wrote:
| I would love to be a bee on the wall in the room where Cloudflare
| response engineers are working right now.
| littlecranky67 wrote:
| AWS, then Azure, now Cloudflare. Welcome to the AI era. Meanwhile
| my hetzner vServer has been running for three years without
| issues.
| yeehaw125 wrote:
| The irony of being in the middle of reading how Basecamp got off
| the cloud and the external link being down with a CF error :D
| EvanAnderson wrote:
| We've traded DDoS for centralized DoS.
| rikafurude21 wrote:
| centralized incompetency causes distributed denial of service
| mintflow wrote:
| Recently my multiple VPN server nodes just randomly cannot
| connect to cloudflare CDN IPs, from different provider VPS, while
| the Host Linux network does not have the issue; vpp share the
| same address with Linux and use tc stateless NAT to do the trick.
|
| I finally work around this by change the tcp options sent by vpp
| tcp stack.
|
| But the whole thing made me worry there must be something
| deployed which cause this issue.
|
| But I do not think that related with this network issue, it just
| reminds me the above, I feel there are frequently new articles
| about cloudflare networking, maybe new method or new deployment
| sort of related high probability of issues
| rco8786 wrote:
| Is it me or has there been a very noticeable uptick in large
| scale infra-level outages lately? AWS, Cloudflare, etc have all
| been way under whatever SLA they publish.
| alt227 wrote:
| For me the only silver lining to all these cloud outages is now
| we know that their published SLA times mean absolutely nothing.
| The number of 9's used to at least give an indication of intent
| of reliability, now they are twisted to whatever metric the
| company wants to represent and dont actually represent
| guaranteed uptime anywhere.
| bojangleslover wrote:
| So true. AWS for example gives only platform credits in the
| event of an outage. Basically no recourse or insurance.
| op00to wrote:
| Doesn't everyone do that? I've never worked for a place
| that the base policy wasn't credits. You might have special
| contract language stating otherwise, but for almost
| everyone, it's credits.
| dimastopel wrote:
| GCP was down recently as well
| driverdan wrote:
| It's you. Everything does down once in a while.
| codethief wrote:
| Don't forget Azure Front Door / half of Azure.
| weberer wrote:
| Yeah, but that's just standard for Azure.
| AsmaraHolding wrote:
| Some of the other commenters here have posited a "vibe code
| theory". As the amount of vibe code in production increases, so
| does the number of bugs and, therefore, the number of outages.
| ACCount37 wrote:
| None of the recent major outages were traced down to "vibe
| coding" or anything of the sort. They appear to be the kind
| of misconfigurations and networking fuckups that existed
| since Internet became more complex than 3 routers.
| Seb-C wrote:
| The "vibe thinking" trend where people stop using their
| brain and rely on whatever random output the LLM tells them
| is harder to diagnose, but it's certainly there and at
| least as bad as vibe coding.
| Aurornis wrote:
| What about the "vibe thinking" trend where people project
| their own narratives on to every situation, even if the
| information available shows that it's a rise in large
| scale DDoS attacks?
| ACCount37 wrote:
| Unfortunately, not a trend. Just human nature. I hope
| they'll find a fix for that one day.
| davey48016 wrote:
| Wasn't the recent AWS a race condition that's existed since
| before vibe coding was a thing?
| ceejayoz wrote:
| How likely are we to know when a "misconfiguration or
| networking fuckup" is due to someone asking ChatGPT how to
| do the task?
| shufflerofrocks wrote:
| >misconfigurations and networking fuckups that existed
| since Internet became more complex than 3 routers.
|
| Yet there has been an uptick in frequency of outages only
| in the recent few months. Correlation correlation.
|
| Why assume that these misconfigs are not the result of
| someone asking AI how to do them?
| voidUpdate wrote:
| Is it a statistically significant uptick though? Random
| events doesn't mean equally spaced, sometimes there will
| be more, sometimes there will be less
| fransje26 wrote:
| Speaking of "vibe-coding", I wonder how much their own outage
| is affecting their ability to vibe-code their way out of it..
| :-)
|
| The openai login page says: Please unblock
| challenges.cloudflare.com to proceed.
| swed420 wrote:
| > Some of the other commenters here have posited a "vibe code
| theory". As the amount of vibe code in production increases,
| so does the number of bugs and, therefore, the number of
| outages.
|
| Likely this coupled with the mass brain damage caused by
| never-ending COVID re-infections.
|
| Since vaccines don't prevent transmission, and each re-
| infection increases the chances of long COVID complications,
| the only real protection right now is wearing a proper
| respirator everywhere you go, and basically nobody is doing
| that anymore.
| DaSHacka wrote:
| Are you being hyperbolic? It's clearly not this, and very
| likely not GP's proposal either.
| swed420 wrote:
| No, and it's easy to find ample research backing it up.
| Swannie wrote:
| Agreed.
|
| Most people are not self reflective reflective enough to
| notice. Need to trust the studies.
|
| Far more plausible than the AI ideas.
|
| I find it far more likely these are smart people running
| without oversight for years pre-COVID, relying on being
| smart at 2am change windows. Now half or a full std. dev.
| lower on the IQ scale, hubris means fewer guard rails
| before change, and far lower ability to recover during
| change window.
| swed420 wrote:
| Exactly. The effects can include both "brain fog" as well
| as impaired judgement, since the brain areas affected
| have to do with executive function.
|
| We can even see (measure) it in driving behavior
| patterns.
|
| Another data point is how Hollywood has gone to great
| lengths to keep the whole thing hush hush, because such a
| downer is bad for business:
|
| https://old.reddit.com/r/ZeroCovidCommunity/comments/1ncm
| clw...
| chid wrote:
| it definitely feels like it.
| tristanperry wrote:
| My theory is a state-sponsored actor targeting some of these
| services, but maybe that's just too 'tinfoil hat' of me, who
| knows.
| wepple wrote:
| There are usually very comprehensive post mortems for these
| events, and none have suggested that at all
| bflesch wrote:
| This only amplifies the often-repeated propaganda about the
| "very powerful" enemies of democracy, who in fact are very
| fragile dictatorships. There's enough incompetence at tech
| companies to f up their own stuff.
| rozap wrote:
| My theory is DNS.
| roxolotl wrote:
| The theory I've heard is holiday deploy freezes coupled with Q4
| goals creates pressure to get things in quickly and early. It's
| all been in the last month or so which does line up.
| grobins2 wrote:
| The obvious answer is to cancel holidays.
| rco8786 wrote:
| What's different about this Q4 vs the last 20 years of Q4s?
| trollbridge wrote:
| Coincidentally, large tech companies have been conducting mass
| layoffs and claim they're going to rely on AI much more to
| replace junior developers.
| MaxHoppersGhost wrote:
| And they are offshoring roles to lower quality devs.
| Aurornis wrote:
| That does seem to be a coincidence, as the recent outages
| making headlines (including this one according to early
| reports) have been associated with huge traffic spikes. It
| seems DDoS are reaching a new level.
| rco8786 wrote:
| AWS's most recent blow-up was not a DDoS
| darknavi wrote:
| Maybe a laid-off engineer is bored and started
| orchestrating DDoS campaigns in their newly-found free
| time.
| xnx wrote:
| By similar thinking, you could blame large tech companies if
| they hired too many juniors.
| 6c696e7578 wrote:
| Interestingly, chatgpt was unavailable due to the same
| cloudflare outage.
| creatonez wrote:
| Imagine vibe coding something in production, it breaks half
| the internet, then you can't vibe code it back because it
| broke the LLM providers. A real catch-22 for the modern
| age!
| elric wrote:
| I suspect the number of outages is the same, but the number of
| sites putting all of their eggs into these two baskets has
| grown considerably.
| mikece wrote:
| Any chance our friend Vladamir is behind this?
| kqr wrote:
| If it's any guidance, US cyber risk insurance (which covers
| among other things disruptions due to supplier outages) has
| continuously dropped in price since Q1 2023, with a handful of
| percent per year.
|
| If you excuse the sloppy plot manually transcribed from market
| index data: https://i.xkqr.org/cyberinsurancecost.png
| zulban wrote:
| Unless you're making that determination statistically, it's
| probably pereidolia. See here:
| https://behavioralscientist.org/yates-expect-unexpected-why-...
| whalesalad wrote:
| Somewhere, at a floating desk behind a wall of lava lamps, in a
| nyancatified ghostty terminal with 32 different shader plugins
| installed:
|
| You're absolutely right! I shouldn't have force pushed that
| change to master. Let me try and roll it back. *
| Confrobulating* Oh no! Cloudflare appears to be down and I
| cannot revert the change. Why don't you go make a cup of coffee
| until that comes back. This code is production ready, it's
| probably just a blip.
| 1zael wrote:
| Well AWS runs on Cloudflare...so thanks Cloudflare team!
| agile-gift0262 wrote:
| In the beginning I thought my IP fell on the wrong side of
| Cloudflare and thought I was being blocked from ~80% of the
| internet. I was starting to panic
| BatteryMountain wrote:
| What have you been looking at citizen?
| BatteryMountain wrote:
| Whole bunch of local South African sites are dead, with
| cloudflare http 500 errors. Can see Lisbon & Amsterdam crashing
| out.
| BrouteMinou wrote:
| Do you remember when the Internet was redundant and resilient?
|
| It seems 20% of the Internet is down every two weeks now.
| KabuTheSilicon wrote:
| 70% of the internet is down
| chakintosh wrote:
| The whole internet hinges on this one company
| DC-3 wrote:
| Certainly most of the independent internet.
| Ilikeruby wrote:
| if you include amazon its actually 2
| turnsout wrote:
| This is reason 1, 2 and 3 on my "Top 3 Reasons to not Put All
| Eggs in One Basket" list.
| TheForeseer wrote:
| Israel must be testing it's doomsday plans again
|
| Wonder if the internet will soon be deleted.
| nonsensewatcher wrote:
| Sure, blame the Jews. Idiot.
| sxzygz wrote:
| To be clear, they blamed Israel. Not that it helps make what
| they said any less idiotic.
| aurise wrote:
| Waking up in East coast USA to all sites being down, yay...
| IsaGuyHere wrote:
| What is that bright yellow thing in the sky?
| nottorp wrote:
| They offer a great service for now, i hear.
|
| Unfortunately, that means they can also break 75% of the
| internet.
| skeptrune wrote:
| Feels like it's been a rough year for huge infra outages man :(.
| leejongyon wrote:
| Under Attack? SO DO I!
| NicoJuicy wrote:
| Third time's the charm? Seems more stable now.
| karel-3d wrote:
| More vibe code gets into production. AWS, Azure and Cloudflare
| all have major issues.
|
| Coincidence? I think not.
| brenoRibeiro706 wrote:
| Ironically, I was trying to access
| https://downforeveryoneorjustme.com/ at the time, which also went
| down due to Cloudflare.
| madjam002 wrote:
| Can't even change my nameservers away from Cloudflare as
| Namecheap use Cloudflare!!
| brightball wrote:
| Oh seriously! Thats one I didn't realize.
| 52-6F-62 wrote:
| The nameservers themselves seem to be working fine if anyone is
| wondering.
|
| I run my applications on OVH behind BunnyCDN and all is well.
| sdoering wrote:
| Just checked INWX from here in Germany. I was able to log in
| and get to my DNS records. Just if you should be looking for an
| alternative after all this.
| madjam002 wrote:
| Yes I will be looking, thanks for the rec!
| b0ner_t0ner wrote:
| Namesilo as well.
| theli0nheart wrote:
| You can't even turn off caching from Cloudflare because...the
| Cloudflare dashboard is down.
|
| So everyone who's wrapped their host with Cloudflare is stuck
| with it.
| isodude wrote:
| When will Cloudflare actually split into several totally
| independent companies to remedy that they bring down the Internet
| every time they have a major issue?
| ericzawo wrote:
| I tried to go to Downdetector before coming to Hacker News...
| laurentiurad wrote:
| Probably they adopted vibe coding as the main way to write code
| Chrovus wrote:
| are we cooked :l
| Chrovus wrote:
| are we cooked?
| bamboozled wrote:
| Not out of the woods yet it seems...
| Chrovus wrote:
| im just concerned now :l
| cornonthecobra wrote:
| I'm weary of the broader internet having spofs like AWS and
| Cloudflare. Can't change routing or DNS horizons to get around
| it. Things are just _broken_ in ways that are not only opaque,
| but destructive due to so much relying on fragile sync state.
|
| Will my Spelling Bee QBABM count today, or will it fail and
| tomorrow I find out that last MA(4) didn't register, ruining my
| streak? Society cannot function like this! /s
| p1dda wrote:
| Cloudflare fucking sucks
| misswaterfairy wrote:
| > During our attempts to remediate, we have disabled WARP [their
| VPN service] access in London. Users in London trying to access
| the Internet via WARP will see a failure to connect. Posted 4
| minutes ago. Nov 18, 2025 - 13:04 UTC
|
| Is Cloudflare being attacked...?
| fergie wrote:
| This line also gave me that vibe
| misswaterfairy wrote:
| > We have made changes that have allowed Cloudflare Access
| [their 'zero-trust network access solution'] and WARP to
| recover. Error levels for Access and WARP users have returned
| to pre-incident rates. > We have re-enabled WARP access in
| London.
|
| > We are continuing to work towards restoring other services.
| > Posted 12 minutes ago. Nov 18, 2025 - 13:13 UTC
|
| Now I'm really suspicious that they were attacked...
| philipwhiuk wrote:
| I will bet it's routing misconfig.
| 0xbkt wrote:
| It always is.
| aberoham wrote:
| Someone running cloudflared accidentally advertising a critical
| route into their Warp namespace and somehow disrupting routes
| for internal Cloudflare services doesn't seem too far fetched.
|
| We vibe coded a tool to mass disconnect Cloudflare Warp for
| incident responders: https://github.com/aberoham/unwarp
|
| To go along with the shenanigans around dealing with MITM
| traffic inspection https://github.com/aberoham/fuwarp
| wiradikusuma wrote:
| I used to say, "Don't worry, we host it on Cloudflare. If it's
| down, then 30% internet is down. It's highly unlikely."
|
| Well...
| brycewray wrote:
| You weren't wrong, but There Will Be Days Like This.
| ProllyInfamous wrote:
| Three _this year_ , so far.
| ziml77 wrote:
| Yes but you also get to say "We're down? Yes of course we're
| down, 30% of the internet is down. Nothing we can do"
| afroboy wrote:
| Like the old saying: Nobody Ever Got Fired for Buying IBM.
| benedictboppler wrote:
| "The issue has been identified and a fix is being implemented."
| According to CF a minute ago:
| https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
| nixass wrote:
| Let me guess, DNS issues?
| gianpaj wrote:
| i use checklyhq.com for my website status page and those are down
| as well...
|
| https://sexyvoice.checkly-dashboards.com
| skc wrote:
| Bluesky still chugging along.
|
| Just saying.
| pogue wrote:
| They are decentralized with servers all on the East coast that
| they self host. They do have points of failure that can take
| down the whole network, however.
| lordofgibbons wrote:
| How did we get to a place where either Cloudflare or AWS having
| an outage means a large part of the web going down? This
| centralization is very worrying.
| ekianjo wrote:
| because cloudfare protection blah blah, until cloudfare is down
| itself and then you are back to "who watches the watchmen"
| k12sosse wrote:
| That's easy, the watchmen watchmen watch the watchmen.
| ec109685 wrote:
| Totally cooked if you have Cloudflare fronting us-east-1, with
| no redundancies.
| tacker2000 wrote:
| The mother of all bad infra decisions.
| a012 wrote:
| They have multi cloud infra, between us-east-1 and Azure
| lbreakjai wrote:
| It could be worse. You could have a backup on Azure.
| cj wrote:
| Except businesses love it.
|
| A lot (and I mean a lot) of people in IT like centralization
| specifically because it's hard to blame people for doing
| something that everyone else is doing.
| iso1631 wrote:
| And HN users love it too. I've had people on this site say
| how great it is that their system routes 30% of traffic on
| the internet.
|
| I'd be horrified. That's not the internet or computing
| industries I grew up with, or started working in.
|
| But as long as the SPY keeps hitting > 10% returns each year,
| everyone's happy.
| chb wrote:
| "No one gets fired for buying IBM!"
| deathhand wrote:
| "No one gets fired for buying Microsoft" "No one gets fired
| for buying AWS" "No one gets fired for buying Cloudflare"
|
| Perhaps the most graceful death of a tech company is that
| sentiment? Before some perception shift?
| lforster wrote:
| They're using cloudfare for multicloud, but still have
| cloudfare as a single point of failure. Should make a cloudfare
| for cloudfare to solve this.
| MichaelZuo wrote:
| If there's clearly a single point of failure shouldn't it be
| called a single cloud pretending to be "multicloud"?
| nexttk wrote:
| Like the infamous "smiling through the pain" meme:
|
| "I added a load-balancer to improve system reliability"
| (happy)
|
| "Load balancer crashed" (smiling-through-the-pain)
| PunchyHamster wrote:
| Reliability have very weird curve frankly.
|
| Technically, multi-node cluster with failover (or full on
| active-active) will have far higher uptime than just a
| single node.
|
| Practically, to get the multi-node cluster (for any non
| trivial workload) to work right, reliably, fail-over in
| every case etc. is far more work, far more code (that can
| have more bugs), and even if you do everything right and
| test what you can, unexpected stuff can still kill it. Like
| recently we had uncorrectable memory error which just
| happened to hit the ceph daemon just right that one of the
| OSDs misbehaved and bogged down entire cluster...
| kevin_thibedeau wrote:
| Failover to Akamai.
| cortesoft wrote:
| As someone who has worked for a CDN for over a decade, this
| is what most big customers do. Under normal circumstances,
| they send portions of traffic to different CDNs, usually
| based on cost (and or performance in various regions). When
| an issue happens, they will pull traffic from the problem
| CDN.
|
| Of course, if a big incident happens for a big CDN, there
| might not be enough latent capacity in the other CDNs to
| take all the traffic. CDNs are a cutthroat business, with
| small margins, so there usually isn't a TON of unused
| capacity laying around.
| amalcon wrote:
| You jest, but this actually does exist. Multiple CDNs sell
| multi-CDN load balancing (divide traffic between 2+ CDNs per
| variously-complicated specifications, with failover) as a
| value add feature, and IIRC there is at least one company for
| which this is the marquee feature. It's also relatively
| doable in-house as these things go.
| joeiq wrote:
| Is avoiding single point of failure in anyone's playbook?
| -\\_(tsu)_/-
| whstl wrote:
| We only care about it when it's time to complain about the
| work of individual people.
|
| Companies can always do as they please and people will
| rationalize anything.
| seydor wrote:
| Slowly and with full conscience of where we were heading to.
| majani wrote:
| Now that network effects and data lock-in have taken root,
| downtime is not as big of a concern as it was in the 2000s
| swyx wrote:
| except, yknow, where peoples lives and livelihoods depend on
| access to information/being able to do things on exact time.
| aws and cloudflare are disqualifying themselves from
| hospitals and military and whatnot.
| kordlessagain wrote:
| For example, Cloudflare employees make money on promises to
| mitigate such attacks, but then can't guarantee they will,
| and take all their customers down at once. It's a shared
| pain model.
| amw-zero wrote:
| What does this even mean? Because people have locked in their
| data, they're ok with downtime? I can't imagine a world where
| this is true.
| wahnfrieden wrote:
| It's just that customers are more understanding when they
| see their Netflix not working either otherwise they just
| think you're less professional. Try talking to customers
| after an outage and you will see.
| Ambolia wrote:
| It costs a lot of money to move, you don't know if the
| alternative will be any better, and if it affects a lot of
| companies then it's nobody's fault. "Nobody ever got fired
| for buying Cloudflare/AWS" as they say.
| serf wrote:
| it's not just that, it's the creation of a sorta status
| symbol, or at least of symbol of normality.
|
| there was a point (maybe still) where not having a netflix
| subscription was seen as 'strange'.
|
| if that's the case in your social circles -- and these kind
| of social things bother you -- you're not going to cancel
| the subscription due to bad service until it becomes a
| socially accepted norm.
| bikamonki wrote:
| Compliance. If you wanna sell your SAAS to big corpo, their
| compliance teams will feel you know what you're doing if they
| read AWS or Cloudflare on your architecture, even if you do not
| quite know what you're doing.
| ulrikrasmussen wrote:
| Yeah, I went to HN after the third web page didn't work. I am
| not just worried about the single point of failure, I am much
| more worried about this centralization eventually shaping the
| future standards of the web and making it de facto impossible
| to self-host anything.
|
| Well that and the fact that when 99% goes through a central
| party, then that central party will be very interesting for
| authoritarian governments to apply sweeping censorship rules
| to.
| popcorncowboy wrote:
| > eventually shaping the future standards of the web and
| making it de facto impossible to self-host anything
|
| Eventually?
| sankalpmukim wrote:
| It is already nearly impossible/very expensive in my country
| to be able to get a public IP address (Even IPv6) which you
| could host on. World is heavily moving towards centrally
| dependant on these big Cloud providers.
| ghshephard wrote:
| What part of the world has any ipv6 limitations? In the USA
| An ISP will give you a /48 from their /32 if you have any
| colo arrangement without even a blink. That gives you 2^16
| _networks_ with essentially infinite number of hosts on
| each network. Zero additional charge.
| bilekas wrote:
| This topic is raised every time there is an outage with
| cloudflare and the truth of the matter is, they offer an
| incredible service, there is not a bit enough competition to
| deal with it. By definition their services are so good BECAUSE
| their adoption rate is so high.
|
| It's very frustrating of course, and it's the nature of the
| beast.
| blazinglyfast wrote:
| False dichotomy. Both can be true.
| bilekas wrote:
| > False dichotomy.
|
| Not sure I follow, I didn't say it wasn't worrying or an
| issue. Just the reasons for it getting to this point are
| valid.
| fithisux wrote:
| Hacking software or hardware is so old school.
|
| The target these days is the user.
|
| The make-believe worm.
| afavour wrote:
| Because no one cares enough, including users.
|
| Oddly this centralization allows a complete deferral of blame
| without you even doing anything: if you're down, that's bad.
| But if you're down, Spotify is down, social media is down...
| then "the internet is broken" and you don't look so bad.
|
| It also reduces your incentive to change, if "the internet is
| down" people will put down their device and do something else.
| Even if your web site is up they'll assume it isn't.
|
| I'm not saying this is a good thing but I'm simply being
| realistic about why we ended up where we are.
| alentred wrote:
| There is an upside too. Us humans, we also need our _down
| time_ occasionally.
| 867-5309 wrote:
| globally coordinated
| eastburnn wrote:
| Businesses and peoples' livelihoods are online nowadays,
| it's not just scrolling Twitter for fun.
|
| The internet can't afford to just "give people mental
| health breaks."
| drunkpotato wrote:
| Actually, yes, it can. Chill a bit.
| zamadatix wrote:
| The vast majority of the internet can afford that though,
| and not the entire thing needs to operate the same way.
| KronisLV wrote:
| > Businesses and peoples' livelihoods are online nowadays
|
| What happened to having a business continuity plan? E.g.
| when your IT system is down, writing down incoming orders
| manually and filling them into the system when it's
| restored?
|
| I have a creeping suspicion that people don't care about
| that, in which case they can't really expect more than to
| occasionally be forced into some downtime by factors
| outside of their control.
|
| Either it's important enough to have contingencies in
| place, or it's not. Downtime will happen either way, no
| matter how brilliant the engineers working at these large
| orgs are. It's just that with so much centralization
| (probably too much) the blast range of any one outage
| will be really large.
| gspencley wrote:
| My wife and I own a small theatre. We can process orders
| in-store just fine. Our customers can even avoid online
| processing fees if they purchase in-store. And if our POS
| system went down, we could absolutely fall back to pencil
| and paper.
|
| Doesn't change the fact that 99% of our ticket sales
| happen online. People will even come in to the theatre to
| check us out (we're magicians and it's a small magic shop
| + magic-themed theatre - so people are curious and we get
| a lot of foot traffic) but, despite being in the store,
| despite being able to buy tickets right then and there
| and despite the fact that it would cost less to do so ...
| they invariably take a flyer and scan the QR code and buy
| online.
|
| We might be kind of niche, since events usually sell to
| groups of people and it's rare that someone decides to
| attend an event by themselves right there on the spot. So
| that undoubtedly explains why people behave like this -
| they're texting friends and trying to see who is
| interested in going. But I'm still bringing us up as an
| example to illustrate just how "online" people are these
| days. Being online allows you to take a step back, read
| the reviews, price shop, order later and have things
| delivered to your house once you've decided to commit to
| purchasing. That's just normal these days for so many
| businesses and their customers.
| HWR_14 wrote:
| Does the flyer make it clear that in person sales will
| have a discount?
| jergason wrote:
| Most businesses are totally fine if they have a few hours
| of downtime. More uptime is better, but treating an
| outage like a disaster or an e-commerce site like a power
| plant is more about software engineer egos than business
| or customer needs.
|
| If AWS is down, most businesses on AWS are also down, and
| it's mostly fine for those businesses.
| serial_dev wrote:
| If an hour outage costs you on average a million dollars,
| you have another 8.759 billion dollars to cover for the
| loss...
| blitzar wrote:
| Shitposting on twitter should never have been a business
| or livelihood in the first place.
| jbreckmckye wrote:
| Why not?
|
| It's better to have diverse, imperfect infrastructure,
| than one form of infra that goes down with devastating
| results.
|
| I'm being semi-flippant but people do need to _cope_ with
| an internet that is less than 100% reliable. As the youth
| like to say, _you need to touch grass_
|
| Being less flippant: an economy that is completely
| reliant on the internet is one vulnerable to
| cyberattacks, malware, catastrophic hardware loss
|
| It also protects us from the malfeasance or incompetence
| of actors like Google (who are great stewards of internet
| infrastructure... until it's no longer in their
| interests)
| afavour wrote:
| I'm not so sure about that. The pre-internet age had a
| lot of forced "mental health breaks". Phone lines went
| down. Mail was delayed. Trains stalled. Businesses and
| livelihoods continued to thrive.
|
| The idea that we _absolutely_ need 24 /7 productivity is
| a new one and I'm not that convinced by it. Obviously
| there are some scenarios that need constant connectivity
| but those are more about safety (we don't want the
| traffic lights to stop working everywhere) than profit.
| SJC_Hacker wrote:
| > we absolutely need 24/7 productivity is a new one
|
| We don't need it, the owners want it
| chipsrafferty wrote:
| We don't need it, but we might need it to maintain our
| quality of life now.
| oceanplexian wrote:
| Just want to correct the record here, as someone who
| worked at a local CLEC where we took availability quite
| seriously before the age of the self-defeatist software
| engineer.
|
| Phone lines absolutely did not go down. Physical POTS
| lines (Yes, even the cheap residential ones) were
| required to have around 5 9s of availability, or
| approximately 5 minutes per year. And that's for a
| physical medium affected by weather, natural disasters,
| accidents, and physical maintenance. If we or the LEC did
| not meet those targets contracts would be breached and
| worst case the government would get involved.
| kazen44 wrote:
| also, the availability of the routing and switching
| infrastructure of the internet must be atleast a factor
| higher then that of the world wide web.
|
| Physical network equipment is redundant and reliant
| enough that getting 5 minutes of downtime or less per
| year is totally doable.
|
| the web however... is a far different beast (and in my
| opinion, with an incentive which does not factor in
| reliability)
| JustExAWS wrote:
| I've worked in cloud consulting for a little over five
| years. I can say 95% of the time when I discuss the cost
| and complexity tradeoffs of their websites being down vs
| going multi region or god forbid "multi cloud", they
| shrug and say, it will be fine if they are down for a
| couple of hours.
|
| This was the same when I was doing consulting inside (ie
| large companies willing to pay the premium cost of AWS
| ProServe consultants) and outside working at 3rd party
| companies.
| luc_ wrote:
| > "give people mental health breaks."
|
| try going outside
| lxgr wrote:
| Oh, if only computers could continue working while I take a
| break, or teams continue working while I'm on PTO...
| timeon wrote:
| But Spotify was not down. One social media was down.
|
| This:
|
| > if you're down, that's bad. But if you're down, Spotify is
| down, social media is down... then "the internet is broken"
| and you don't look so bad.
|
| is just marketing. If you are down with some other websites
| it is still bad.
| monooso wrote:
| > If you are down with some other websites it is still bad.
|
| In some cases, absolutely. For the vast majority, it
| really, really doesn't matter.
|
| (Source: my personal website is down and nobody cares,
| including me)
| afavour wrote:
| Admittedly when I wrote that I was thinking about the
| recent AWS outage. Anecdotally, I asked friends and family
| about their experience and they assumed the internet was
| down. Almost everything at my work runs on Google cloud so
| we were still running but we observed a notable dip in
| traffic during the outage all the same.
|
| > it is still bad
|
| No doubt. But there's a calculation to make, is it bad
| _enough_ to spend the extra money on mitigations, to hire
| extra devops folks to manage it all... and in the majority
| of end user facing cases the answer is no, it isn't.
| lillecarl wrote:
| Where I've worked and we've been in the cloud I've always
| promoted just running in one AZ, I run my own things in
| one Hetzner DC (hel1). I've done hybrid cloud as well and
| in that case we only have one AZ for the on-premise stuff
| anyways (plus offsite backup)
|
| That one time when an AZ goes down and your infra
| successfully fails over to the other two isn't worth it
| for a lot of my scale companies, ops consultants seem to
| be chasing high cloud spend to justify their own high
| cost. I also factor in that I live in Sweden where most
| infrastructure outages are exceptionally rare.
|
| Ofc it depends on what kind of company you are and what
| you're providing.
| BeFlatXIII wrote:
| > if "the internet is down" people will put down their device
| and do something else
|
| In this case, the internet should be down more often.
| jesterp wrote:
| If the internet is down is what it takes to get you to put
| it down once in a while, I think thats probably the
| problem.
| tjoff wrote:
| Users have no options because... everything has been
| centralized. So it doesn't matter if users care or not.
|
| Users are never a consideration today anyway.
| kordlessagain wrote:
| It's time to revolt.
| jballanc wrote:
| More like it's time for the pendulum to swing back...
|
| We had very decentralized "internet" with BBSes, AOL,
| Prodigy, etc.
|
| Then we centralized on AOL (ask anyone over 40 if they
| remember "AOL Keyword: ACME" plastered all over roadside
| billboards).
|
| Then we revolted and decentralized across MySpace, Digg,
| Facebook, Reddit, etc.
|
| Then we centralized on Facebook.
|
| We are in the midst of a second decentralization...
|
| ...from an information consumer's perspective. From an
| internet infrastructure perspective, the trend has been
| consistently toward more decentralization. Initially,
| even after everyone moved away from AOL as their sole
| information source online, they were still accessing all
| the other sites over their AOL dial-up connection.
| Eventually, competitors arrived and, since AOL no longer
| had a monopoly on content, they lost their grip on the
| infrastructure monopoly.
|
| Later, moving up the stack, the re-centralization around
| Facebook (and Google) allowed those sources to centralize
| power in identity management. Today, though, people
| increasingly only authenticate to Facebook or Google in
| order to authenticate to some 3rd party site. Eventually,
| competitors for auth will arrive (or already have _ahem_
| passkeys _coughcough_ ) and, as no one goes to Facebook
| anymore anyway, they'll lose grip on identity management.
|
| It's an ebb and flow, but the fundamental _capability_
| for decentralization has existed in the technology behind
| the internet from the beginning. Adoption and
| acclimatization, however, is a much slower process.
| 0ldblu3 wrote:
| These centralized services do and did solve problems. I'm
| old enough to remember renting a quarter rack, racking my
| own server and other infrastructure, and managing all
| that. That option hasn't gone away, but there are layers
| of abstraction at work that many people probably haven't
| and don't want to be exposed to.
| ajmurmann wrote:
| Aaand even if we ignore the "benefit" of Cloudflare and
| AWS outages being blamed on them, rather than you, what
| does uptime look like for artisanaly hosted services on a
| quarter rack vs your average services on AWS and
| Cloudflare?
| netdevphoenix wrote:
| It is a trade-off between convenience and freedom. Netflix
| vs buying your movies. Spotify vs mp3s. Most tech products
| have alternatives. But you need to be flexible and adjust
| your expectations. Most people are not willing to do that
| sigbottle wrote:
| The issue is that real life is not adaptable. Resources
| and capital are _slow_.
|
| That's the whole issue with monopolies for example,
| innit? We envision "ideal free market dynamics" yet in
| practice everybody just centralizes for efficiency gains.
| ajmurmann wrote:
| > The issue is that real life is not adaptable. Resources
| and capital are slow. > That's the whole issue with
| monopolies for example, innit?
|
| The much bigger issue with monopolies is that there is no
| pressure on the monopolist to compete on price or quality
| of the offering.
| sigbottle wrote:
| Right, and my point is that "ideal free market dynamics"
| conveniently always ignore this failure state that seems
| to always emerge as a logical consequence of its tenets.
|
| I don't have a better solution, but it's a clear problem.
| Also, for some reason, more and more people (not you)
| will praise and attack anyone who doesn't defend state A
| (ideal equilibrium). Leaving no room to point out state B
| as a logical consequence of A which requires
| intervention.
| Edman274 wrote:
| The definition of a monopoly basically resolves to "those
| companies that don't get pressured to meaningfully
| compete on price or quality", it's a tautology. If a firm
| has to compete, it doesn't remain a monopoly. What's the
| point you're making here?
| casey2 wrote:
| That's just a post hoc rationalization. If the capital
| owners don't want something to happen then market
| dynamics don't matter a lick
| ajmurmann wrote:
| There absolutely are options but we aren't using them
| because nobody cares enough about these downsides. bsky is
| up, with Mastodon you even have choice between tons of
| servers and setting up your own. Yet, nobody cares enough
| about the occasional outage to switch. It's such a minor
| inconvenience that it won't move the needle one bit. If
| people actually cared, businesses would lose customers and
| correct the issue.
| ocdtrekkie wrote:
| This is essentially the entire IT excuse for going to
| anything cloud. I see IT engineers all the time justifying
| that the downtime stops being their problem and they stop
| being to blame for it. There's zero personal responsibility
| in trying to preserve service, because it isn't "their
| problem" anymore. Anyone who thinks the cloud makes service
| more reliable is absolutely kidding themselves, because
| everyone who made the decision to go that way already knows
| it isn't true, it just won't be their problem to fix it.
|
| If anyone in the industry actually cared about reliability
| and took personal stake in their system being up, everyone
| would be back on-prem.
| RajT88 wrote:
| Reliability is not even how the cloud got sold to the C
| Suite. Good God, when my last company started putting
| things on Azure back in 2015 stuff would break weekly,
| usually on Monday mornings.
|
| No, the value proposition was always about saving money,
| turning CapEx into OpEx. Direct quote from my former CEO
| maybe 9 years ago: We are getting out of the business of
| buying servers.
|
| Cloud engineering involves architecting for unexpected
| events: retry patterns, availability zones, multi-region
| fail over, that sort of thing.
|
| Now - does it all add up to cost savings? I could not tell
| you. I have seen some case studies, but I also have been
| around long enough to take those with a big grain of salt.
| dig1 wrote:
| > ...does it all add up to cost savings?
|
| IMHO it adds, but only if you are big enough. Netflix
| level. At that level, you go and dine with Bezos and
| negotiate a massive discount. For anyone else, I'd
| genuinely love to see the numbers that prove otherwise.
|
| > There's zero personal responsibility
|
| Unfortunately, this seems to be the unspoken mantra of
| modern IT management. Nobody wants to be directly
| accountable for anything, yet everyone wants to have
| their fingerprints on everything. A paradox of
| collaboration without ownership.
| RajT88 wrote:
| Cloud providers have formalized these deals actually. If
| you promise to spend X amount over Y period, you get Z
| discounts.
|
| And this is not reserved instances, this is an org level
| pricing deal. Some have been calling it anti-competitive
| and saying the regulators need to look at the practice.
| SJC_Hacker wrote:
| > IMHO it adds, but only if you are big enough. Netflix
| level. At that level, you go and dine with Bezos and
| negotiate a massive discount. For anyone else, I'd
| genuinely love to see the numbers that prove otherwise.
|
| It adds if you're smart about using resources
| efficiently, at any level. And engineer the system to
| spin up / spin down as customers dictate.
|
| For situations where resources are allocated but are only
| being utilized a low percentage (even < 50% in some
| cases), it is not cost effective. All that compute / RAM
| / disk / network etc. is just sitting there wasted.
| mosura wrote:
| No the value was bypassing IT.
|
| You no longer needed them to approve a new machine, you
| just spun it up how you want. Sped things up massively
| for a while.
| darkwater wrote:
| That might have been true for some kind of organization,
| but definitely not for every kind. On the other side,
| there were start-ups that wanted the elasticity and no
| commitments. But both sides at least partially liked the
| "it's not on me anymore" feature.
| ocdtrekkie wrote:
| It's amazing how there's so many cybersecurity incidents
| now. Bypassing IT will _always_ backfire spectacularly,
| IT is the people that stop you from dumbing.
| mosura wrote:
| The opposite was/is true. If your cloud box can only be
| used by two people and IT don't even know about it then
| IT can never be persuaded to provide the keys to the rest
| of the company as they were predisposed to doing.
|
| I saw this stuff too many times, and it is precisely why
| the cloud exploded in use in about 2010.
| ocdtrekkie wrote:
| What you're telling me is two people potentially have
| regulated or confidential data not secured by IT, which
| nobody knows if got leaked.
|
| For many organizations, that's literally illegal, and
| anyone who does this should be fired.
| mosura wrote:
| One notable example was signing keys for builds for
| distribution actually. And IT had a habit of handing them
| out to absolutely everyone. Being able to audit who did
| the signing was done in spite of IT who could, of course,
| never be persuaded of the merit of any process they don't
| own.
|
| But sure jump to more conclusions if you want.
| ocdtrekkie wrote:
| I won't discount your IT can be bad, but also if you're
| keeping something as core to your security as signing
| keys somewhere your IT can't audit, you are just as bad.
| And your IT won't be the ones fired when your keys leak.
| mosura wrote:
| You are under the erroneous impression IT would be fired
| for leaking keys and not simply impose a new process that
| blames everyone else.
|
| And this is in Fortune 500 of course.
| pixl97 wrote:
| >, that's literally illegal, and anyone who does this
| should be fired.
|
| I mean yea, but who knows how long that box would sit
| around before it was discovered.
| serial_dev wrote:
| I mean in the end it's about making a trade off that makes
| sense for your business.
|
| If the business can live with a couple of hours downtime
| per year when "cloud" is down, and they think they can ship
| faster / have less crew / (insert perceived benefit), then
| I don't know why that is a problem.
| PunchyHamster wrote:
| More like "don't have choice". It's not like service provider
| gonna go to competition, because before you switch, it will
| be back.
|
| Frankly it's a blessing, always being able to blame the cloud
| that management forced company to migrate to be "cheaper"
| (which half of the time turns out to be false anyway)
| marticode wrote:
| As a user I do care, because I waste so much time on
| Cloudflare's "prove you are human" blocking-page (why do I
| have to prove it over and over again?), and frequently run on
| websites blocking me entirely based on some bad IP-blacklist
| used along with Cloudflare.
| woooooo wrote:
| I just realized, why don't they have some "definitely
| human" third party cookie that caches your humanness for
| 24h or so? I'm sure there's a reason, I've heard third
| party cookies were less respected now, but can someone
| chime in on why this doesn't work and save a ton of
| compute?
| acureau wrote:
| Because people will solve the challenge once, and then
| use the cookie in automation tools. It already happens
| with shorter expiration cookies.
| woooooo wrote:
| Thanks, I'm now shaking my head at my naivete :)
| lotsofpulp wrote:
| I assume that will be for Apple (and eventually Alphabet)
| to implement via digital IDs linked to real world IDs.
|
| https://www.apple.com/newsroom/2025/11/apple-introduces-
| digi...
| philipwhiuk wrote:
| Don't worry, Sam Altman is selling the protection too --
| https://en.wikipedia.org/wiki/World_(blockchain)
| basilikum wrote:
| https://developers.cloudflare.com/waf/tools/privacy-pass/
| arbol wrote:
| Are you really posting this today?
| octoberfranklin wrote:
| Yes, there are several, and the good one (linked below)
| lets you use the "humanness" token across different
| websites _without_ them being able to use it as a
| tracking signal / supercookie. It's very clever.
|
| https://github.com/ietf-wg-privacypass/base-drafts
|
| https://privacypass.github.io/
| jakub_g wrote:
| I hate it as much (and the challenge time seems to be
| getting longer, 10s lately for me, what the hell?)
|
| But we can all say thank you to all the AI crawlers who
| hammer websites with impossible traffic.
| pixl97 wrote:
| I mean, it was a problem before AI crawlers with just
| bots and attacks in general.
| olyjohn wrote:
| It wasn't nearly as bad.
| edm0nd wrote:
| Congrats, you now know what it's like to be a daily Tor
| user trying to hit normie sites from exit node IPs xD
| replwoacause wrote:
| Why would anyone be a daily Tor user and trying to hit
| clear-net sites on top of that? This sounds like a
| bizarre usecase.
| pinko wrote:
| Privacy through uniformity, operational security by
| routine, herd immunity for privacy, traffic
| normalization, "anonymity set expansion", "nothing to
| hide" paradox, etc.
|
| I.e., if you use Tor for "normie sites", then the fact
| that someone can be seen using Tor is no longer a
| reliable proxy for detecting them trying to see/do
| something confidential and it becomes harder to identify
| & target journalists, etc. just because they're using
| Tor.
| replwoacause wrote:
| Huh never thought about that. I wonder how many people do
| that? Seems like a public service.
| milderworkacc wrote:
| It certainly feels like one at times!
| dooglius wrote:
| In addition to the reasons in sibling comment, this also
| acts as a filter for low-quality ad-based sites; same
| reason I close just about any website that gives me a
| popup about a ToS agreement.
| tempest_ wrote:
| Unfortunately the internet sucks in 2025.
|
| If you have a site with valuable content the LLM crawlers
| hound you to no end. CF is basically a protection racket at
| this point for many sites. It doesnt even stop the more
| determined ones but it keeps some away.
| seniorThrowaway wrote:
| Yep for anyone unaware of how awful things truly are,
| look up what a "residential proxy" is. Back in my day we
| called that a botnet.
| nananana9 wrote:
| Oh, they're still botnets. We just look the other way
| because they're useful.
|
| And they're pretty tame as far as computer fraud goes -
| if my device gets compromised I'd much rather deal with
| it being used for fake YouTube views than ransomware or a
| banking trojan.
| deadbabe wrote:
| You can make a little bit of cash on the side letting
| companies use your bandwidth a bit for proxying. You
| won't even notice. $50/month. Times are tough!
| jamwil wrote:
| Of course the risk here being whatever nefarious or
| illegal shit is flowing through your pipes, which you
| consented to and even received consideration for.
| deadbabe wrote:
| No worries it's encrypted traffic
| j2kun wrote:
| And yet half the HN front page every day is promoting LLM
| stuff.
|
| "The internet sucks", yes, but we're doing it to
| ourselves.
| kadushka wrote:
| Would you rather not have LLMs?
| inferiorhuman wrote:
| Yes
| j2kun wrote:
| IMO LLMs have been a net negative on society, including
| my life. But I'm merely pointing out the stark contrast
| on this website, and that fact that we can choose to live
| differently.
| kadushka wrote:
| Are you anti-AI in general, or are you unhappy about the
| current LLMs?
| j2kun wrote:
| I am not anti-AI, nor unhappy about how any current LLM
| works. I'm unhappy about how AI is used and abused to
| collective detriment. LLM scraper spam leading to
| increased centralization and wider impacting failures is
| just one example.
| kadushka wrote:
| Your position is similar to saying that medical drugs
| have been a net negative on society, because some drugs
| have been used and abused to collective detriment (and
| other negative effects, such as doctors prescribing pills
| instead of suggesting lifestyle changes). Does it mean
| that we would be better off without any medical drugs?
| foobarchu wrote:
| Absolutely. They have dramatically worsened the world,
| with little to no net positive impact. Nearly every (if
| not all) positive impacts have an associated negative
| that that dwarfs it.
|
| LLMs aren't going anywhere, but the world would be a
| better place if they hadn't been developed. Even if they
| had more positive impacts, those would not outweigh the
| massive environmental degradation they are causing or the
| massive disincentive they created against researching
| other, more useful forms of AI.
| kazen44 wrote:
| LLM's to me sound like a "boiling the ocean" kind of
| approach to solving a problem.
| salawat wrote:
| Can I raise that to no LLMs or SEO?
| worik wrote:
| Yes
|
| LLMs have become a crucial compendium of knowledge, that
| had become hidden behind SEO
| ToucanLoucan wrote:
| Yes.
|
| A solid secondary option is making LLM scraping for
| training opt-in, and/or compensating sites that were/are
| scraped for training data. Hell, maybe then you could not
| knock websites over incentivizing them to use Cloudflare
| in the first place.
|
| But that means LLM researchers have to respect other
| people's IP which hasn't been high on their todo lists as
| yet.
|
| bUt ThAT dOeSn'T sCaLe - not my fuckin problem chief. If
| you as an LLM developer are finding your IP banned or you
| as a web user are sick of doing "prove you're human"
| challenges, it isn't the website's fault. They're trying
| to control costs being arbitrarily put onto them by a
| disinterested 3rd party who feels entitled to their
| content, which it costs them money to deliver. Blame the
| asshole scraping sites left and right.
|
| Edit: and you wouldn't even need to go THAT far. I scrape
| a whole bunch of sites for some tools I built and a
| homemade news aggregator. My IP has never been flagged
| because I keep the number of requests down wherever
| possible, and rate-limit them so it's more in line with
| human like browsing. Like so much of this could be solved
| with basic fucking courtesy.
| captainkrtek wrote:
| hard yes, all of the technical discussion aside, the
| constant advertising deluge of every company touting AI
| is mind numbing.
| davidhaymond wrote:
| Absolutely. And while we're at it, let's do away with
| social media.
| lenerdenator wrote:
| Not to speak for the other poster, but... That's not a
| good-faith question.
|
| Most of the problems on the internet in 2025 aren't
| because of one particular technology. They're because the
| modern web was based on gentleman's agreements and
| handshakes, and since those things have now gotten in the
| way of exponential profit increases on behalf of a few
| Stanford dropouts, they're being ignored writ large.
|
| CF being down wouldn't be nearly as big of a deal if
| their service wasn't one of the main ways to protect
| against LLM crawlers that blatantly ignore robots.txt and
| other long-established means to control automated
| extraction of web content. But, well, it is one of the
| main ways.
|
| Would it be one of the main ways to protect against LLM
| web scraping if we investigated one of the LLM startups
| for what is arguably a violation of the Computer Fraud
| and Abuse Act, arrested their C-suite, and sent each
| member to a medium-security federal prison (I don't know,
| maybe Leavenworth?) for multiple years after a fair
| trial?
|
| Probably not.
| chasing0entropy wrote:
| I'm Sure there will be an investigation... By the SEC
| when the bubble pops and takes the S&P with it. No prison
| though, probably jobs at the next ponzi scheme
| j2kun wrote:
| Well said.
| seanw444 wrote:
| It's helped me learn some things quicker, but I
| definitely prefer the old days.
| LtWorf wrote:
| Yes.
| nhhvhy wrote:
| Yes.
| therein wrote:
| Yes, they are terrible and more a negative force than a
| positive one in every way imaginable. I would take no
| LLMs all day every day.
| stalfosknight wrote:
| Yes
| BrenBarn wrote:
| Good lord yes. No question.
| roflyear wrote:
| Unfortunately the problem isn't just "the internet sucks"
| it's "the internet sucks, and everyone uses it" - meaning
| people are not doing stuff offline, and a lot of our
| lives require us to be online.
| worik wrote:
| The Internet is huming along beautifully
|
| It is the Web that is being degraded
| hollerith wrote:
| CF would be a protection racket only if CF is the cause
| of the problem CF is charging money to solve.
| crazygringo wrote:
| But that's not a problem caused by Cloudflare.
|
| That's a problem caused by bots and spammers and DDoSers,
| that Cloudflare is trying to alleviate.
|
| And you generally don't have to prove it over and over
| again unless there's a high-risk signal associated with
| you, like you're using a VPN or have cookies disabled, etc.
| Which are great for protecting your privacy, but then
| obviously privacy means you do have to keep demonstrating
| you're not a bot.
| BarryMilo wrote:
| You might say the problem CloudFlare is causing is lesser
| than the ones it's solving, but you can't say they're not
| causing a new, separate problem.
|
| That they're _trying_ counts for brownie points, it 's
| not an excuse to be satisfied with something that still
| bothers a lot of people. Do better, CloudFlare.
| crazygringo wrote:
| Do better, how?
|
| If you have any ideas on how to protect against bad
| actors in a way that is just as effective but easier for
| users, please share it.
|
| Because as far as I can tell, this isn't a question of
| effort. It's a question of fundamental technological
| limitations.
| baxtr wrote:
| Who cares if a couple of websites are down a day or even two?
|
| As long as HN is up and running, everything is going to be
| O.K.!
| lobsterthief wrote:
| There was a problem posting your comment.
| 52-6F-62 wrote:
| Wealthy, investment-bloated software companies will be
| fine.
|
| Smaller companies that provide real world services or goods
| to make a much more meagre living that rely on some of the
| services sold to them by said software companies will be
| impacted much more greatly.
|
| Losing a day or two of sales to someone who relies on
| making sales every day can be a growing hardship.
|
| This doesn't just impact developers. It's exactly this kind
| of myopic thinking that leads to scenarios like mass
| outages.
| carlosjobim wrote:
| You don't lose a day of sales, customers come back when
| the site is up again.
| LamaOfRuin wrote:
| Some of them do. There are very few products or services
| where _all_ of them will.
| HWR_14 wrote:
| Sure, but some people who were going to buy your
| competitors product forget about that and will instead
| find your product. I assume it all evens out.
| xeromal wrote:
| So Say We All!
| LtWorf wrote:
| > Because no one cares enough, including users.
|
| When have users been asked about anything?
| lxgr wrote:
| Many people care, but none of them can (sufficiently) change
| the underlying incentive structure to effect the necessary
| changes.
| mistrial9 wrote:
| > Because no one cares enough, including users.
|
| this is like a bad motivational speaker talk.. heavy
| exhortations with a dramatic lack of actual reasoning.
|
| Systems are difficult, people. It is "incentives" of parties
| and lockin by tech design and vendors, not lack of individual
| effort.
| delfinom wrote:
| Eh? It's because they are offering a service too good to
| refuse.
|
| The internet this day is fucking dangerous and murderous as
| hell. We need Cloudflare just to keep services up due to the
| deluge of AI data scrapers and other garbage.
| thr0w wrote:
| > But if you're down, Spotify is down, social media is
| down... then "the internet is broken" and you don't look so
| bad.
|
| In my direct experience, this isn't true if you're running
| something even vaguely mission-critical for your customers.
| Your customer's workers just know that they can't do their
| job for the day, and your customer's management just knows
| that the solution they shepherded through their organization
| is failing.
| acedTrex wrote:
| It's really quite funny, many of the ACTUALLY vital systems
| to running the world as we know it are running off of very
| different softwares. Cloudflare appears to have a much
| higher % of non vital systems running on it than say
| something like akamai.
|
| If akamai went down i have a feeling you'd see a whole lot
| more real life chaos.
| kazen44 wrote:
| i also find the sentiment of "well we use a third party
| so blame them" completely baffeling.
|
| if you run anything even remotely mission critical, not
| having a plan B which is executable and of which you are
| in control (and a plan C) will make you look completely
| incompetent.
|
| There are very, very few events which some people who run
| mission critical systems accept as force majeur. Most of
| those are of the scale "national emergency" or worse.
| jclardy wrote:
| Which "user" are you referring to? Cloudflare users or end
| product users?
|
| End product users have no power, they can complain to support
| and maybe get a free month of service, but the 0.1% of
| customers that do that aren't going to turn the tide and have
| anything change.
|
| Engineering teams using these services also get "covered" by
| them - they can finger point and say "everyone else was down
| too."
| tjwebbnorfolk wrote:
| > if "the internet is down" people will put down their device
| and do something else.
|
| oh no
| ozgrakkurt wrote:
| On the other hand, it is cool to be up when the internet is
| down
| Hrun0 wrote:
| > It also reduces your incentive to change, if "the internet
| is down" people will put down their device and do something
| else. Even if your web site is up they'll assume it isn't.
|
| I agree. When people talk about the enshittification of the
| internet, Cloudflare plays a significant role.
| ge96 wrote:
| Also it's free (the basic domain protection offered by CF
| anyway)
| falcor84 wrote:
| 100% this. While in my professional capacity I'm all in for
| reliability and redundancy, as an individual, I quite like
| these situations when it's obvious that I won't be getting
| any work done and it's out of my control, so I can go run
| some errands to or read a book, or just finish early.
| pancsta wrote:
| > if you're down, Spotify is down, social media is down...
| then "the internet is broken" and you don't look so bad.
|
| Which changes nothing to you actually being down, youre only
| down more. CF proxies always sucked - not your domain, not
| your domain...
| nntwozz wrote:
| Two ways. Gradually, then suddenly.
| burnt-resistor wrote:
| A key risk of monopolies is that they lead to monoculture
| SPoFs.
| pixel_popping wrote:
| Mostly since the AWS craze started a decade ago, developers
| have gone away from Dedicated servers (which are actually
| cheaper, go figure), which is causing all this mess.
|
| It's genuinely insane that many companies are designing a great
| amount of fallbacks... on the software level but almost none is
| thought on the hardware/infrastructure level, common-sense
| dictate that you should never host everything on a single
| provider.
| geerlingguy wrote:
| I tried as hard as I could to stay self hosted (and my
| backend is, still), but getting constant DDoS attacks and not
| having the time to deal with fighting them 2-3x a month was
| what ultimately forced me to Cloudflare. It's still worse
| than before even with their layers of protection, and now I
| get to watch my site be down a while, with no ability to
| switch DNS to point back to my own proxy layer, since CF is
| down :/
| VladVladikoff wrote:
| This is wild. Was your website somehow controversial? Ive
| been running many different websites for over 30+ years
| now, and have never been the target of a DDOS. The closest
| I've seen was when one website had a blind time based sql
| injection vulnerability and the attacker was abusing it,
| all the SLEEP() injected into the database brought the
| server to a crawl. But that's just one attacker from a
| handful of IPs, hardly what i would call a DDOS.
| pclmulqdq wrote:
| I think people sometimes mistake legitimate traffic
| spikes for DDOS attacks. My blog has the former, but no
| site I have ever hosted has seen the latter.
| geerlingguy wrote:
| I made the mistake of telling people it was hosted on a
| Pi cluster in a YouTube video a couple years ago, and
| asked nobody to try DDoSing it. I was a bit more naive
| thinking the YouTube viewer community was more like HN
| where people may joke about it, but nobody would actually
| do it.
|
| I was wrong, and ever since I've dealt with a targeted
| attack (which was evolving as I added more CF firewall
| rules). At this point it's taken care of, but only
| because I have most things completely blocked at the CF
| firewall layer.
| Imustaskforhelp wrote:
| Jeff the reason why I think is that youtube community is
| more main-stream and I would consider you to be a really
| nice youtuber but even that might attract some bad faith
| actors just because of how main-stream youtube is as well
| compared to HN which is more niche-overall
|
| (also congrats on 1 million subscribers but I know you
| must be tired of listening it but have a nice day jeff!
| Your videos are awesome!)
| bobdvb wrote:
| Until I changed job recently, I spent the past 8 years
| working in an area of tech that many people on places
| like HN and Reddit think that the work is a horrific
| waste of effort (DRM and content security for a streaming
| company).
|
| The idea that if companies like my former employer would
| stop doing DRM their audience would embrace it is rife
| idealism. But based on bitter experience so enough people
| will do bad things just for the lulz that you need to
| cover your ass.
|
| My home lab will never have an open port, I'll always put
| things behind a CDN or zero trust system, even then...
|
| FWIW, it's worthwhile just for educational reasons to
| look at abuseipdb.com quite revealing.
| engeljohnb wrote:
| I'm a lay person, but do you mean DRM isn't just copy-
| protection? Is it also network security?
| shagmin wrote:
| When I was younger and living in military dorms, I put a
| old throw away laptop hosting a simple website via Apache
| on the internet. Every time I checked the log it'd be
| full of so many random, wild spurts of attacks (granted I
| had basically 0 legit traffic).
| imglorp wrote:
| With the state of constant attack from AI scrapers and DDOS
| bots, you pretty much need to have a CDN from someone now, if
| you have a serious business service. The poor guys with
| single prem boxes with static HTML can /maybe/ weather some
| of this storm alone but not everything.
| spurgu wrote:
| Yeah this is the gist of it. Cloudflare provides an
| important service that is quite challenging to implement by
| yourself.
| herbst wrote:
| This is the sad reality behind it. My websites would be
| constantly down because of AI scrapers. If anyone knows a
| good alternative, that doesn't cost an arm and a leg I am
| very open to hear!
| PaulHoule wrote:
| ... dedis are cheaper if you are rightsized. If you are
| wrongsize they just plain crash and you may or may not be
| able to afford the upgrade.
|
| I was at Softlayer before I was at AWS and what catalyzed the
| move was the time I needed to add another hard drive to a
| system and somehow they screwed it up. I couldn't put a
| trouble ticket it to get it fixed because my database record
| in their trouble ticket system was corrupted. The next day I
| moved my stuff to AWS and the day after that they had a top
| sales guy talk to me to try to get me to stay but it was too
| late.
| elondaits wrote:
| I self hosted on one of the company's servers back in the
| late 90s. Hard drive crashes (and a hack once, through an
| Apache bug) had our services (http, pop, smtp, nfs, smb, etc
| ) down for at least 2-3 days (full reinstall,
| reconfiguration, etc).
|
| Then, with regular VPSs I also had systems down for 1-2 days.
| Just last week the company that hosts NextCloud for us was
| down the whole weekend (from Friday evening) and we couldn't
| get their attention until Monday.
|
| So far these huge outages that last 2-5 hours are still lower
| impact for me, and require me to take less action.
| bungle wrote:
| Solving issue for a few, and making issues for millions,
| including perhaps the few. It is easier to sleep at nights
| though, for a few.
| powerpixel wrote:
| Maintainance cost is the main issue for on-prem infra,
| nowadays add things like DDOS protection and/or scraping
| protection, which can require dedicated team or for your
| company to rely on some library or open source project that
| is not guaranteed to be maintained forever (unless you give
| them support, which i believe in)... Yeah I can understand
| why companies shift off of on-prem nowadays
| nzach wrote:
| > developers have gone away from Dedicated servers (which are
| actually cheaper, go figure)
|
| It depends on how you calculate your cost. If you only
| include the physical infrastructure having a dedicated server
| is cheaper. But by having some dedicated server you loose a
| lot of flexibility. Needs more resources? Just scale up your
| ec2, and with a dedicated server there is a lot more work
| involved.
|
| Do you want a 'production-ready' database? With AWS you can
| just click a few buttons and have a RDS ready to use. To roll
| out your own PG installation you need someone with a lot of
| knowledge(how to configure replication? backups? updates?
| ...).
|
| So if you include salaries in the calculation the result
| changes a lot. And even if you already have some experts in
| your payroll by putting them to work in deploying a PG
| instance you won't be able to use them to build other things
| that may generate more value to you business than the premium
| you pay to AWS.
| slightwinder wrote:
| Cloud-Hoster are that hardware-fallback. They started with
| offering better redundancy and scaling than your homemade
| breadbox. But it seems they lost something along the way and
| now we have this.
| MattSayar wrote:
| I like the idea of having my own rack in a data center
| somewhere (or sharing the rack, whatever) but even a tiny
| cost is still more than free. And even then, that data center
| will also have outages, with none of the benefits of a
| Cloudflare Pages, GitHub Pages, etc.
| martinald wrote:
| Many reasons but DDoS protection has massive network effects.
| The more customers you have (and therefore bandwidth provision)
| the easier it is to hold up against a DDoS, as DDoS are
| targeting just one (usually) customer.
|
| So there are massive economies of scale. Small CDN with (say)
| 10,000 customers and 10mbit/sec per customer can handle
| 100gbit/s DDoS (way too simplistic, but hopefully you get the
| idea) - way too small.
|
| If you have the same traffic provisioned on average per
| customer and have 1 million customers, you can handle a DDoS
| 100x the size.
|
| Only way to compete with this is to massively overprovision
| bandwidth per customer (which is expensive, as those customers
| won't pay more just for you to have more redundancy because you
| are smaller).
|
| In a way (like many things in infrastructure) CDNs are natural
| monopolies. The bigger you get -> the more bandwidth and PoP
| you can have -> more attractive to more customers (this repeats
| over and over).
|
| It was probably very astute of Cloudflare to realise that
| offering such a generous free plan was a key step in this.
| codedokode wrote:
| In my opinion, DDoS is possible only because there is no
| network protocol for a host to control traffic filtering on
| upstream providers (deny traffic from certain subnets or
| countries). In this case everybody would prefer write their
| own systems rather than rely on a harmful monopoly.
| powerpixel wrote:
| > here is no network protocol for a host to control traffic
| filtering on upstream providers (deny traffic from certain
| subnets or countries).
|
| There is no network protocol per se, but there is
| commercial solutions like fortinet that can block countries
| iirc, but to note that it's only ip range based so it's not
| worth a lot
| wbl wrote:
| Unless you filter at the far end of the bottleneck you
| still go offline.
| jabart wrote:
| I'm pretty sure BGP magic will let you blackhole a whole
| subnet.
| mrktf wrote:
| I think parent means: there no network protocol which can
| propagate blocking in sane manner between providers
| (something like bgp for firewalls)
|
| edit: yes, you can you bgp to blockhole subnet traffic -
| the standard doesn't play well if you want blackhole
| unrelated subnets from upstream network
| peanut-walrus wrote:
| What traffic would you request the upstream providers to
| block if getting hit by Aisuru? Considering the botnet
| consists of residential routers, those are the same
| networks your users will be originating from. Sure, in best
| case, if your site is very regional, you can just block all
| traffic outside your country - but most services don't have
| this luxury.
|
| Blocking individual IP addresses? Sure, but consider that
| before your service detects enough anomalous traffic from
| one particular IP and is able to send the request to block
| upstream, your service will already be down from the
| aggregate traffic. Even a "slow" ddos with <10 packets per
| second from one source is enough to saturate your 10Gbps
| link if the attacker has a million machines to originate
| traffic from.
| amaccuish wrote:
| Could it not be argued that ISPs should be forced to
| block users with vulnerable devices?
|
| They have all the data on what CPE a user has, can send a
| letter and email with a deadline, and cut them off after
| it expires and the router has not been updated/is still
| exposed to the wide internet.
| encom wrote:
| It could be argued that ISPs should not snoop on my
| traffic, barring a court order.
| SJC_Hacker wrote:
| Xfinity did exactly this to me a few years ago. I wasn't
| compromised but tried running a blockchain node on my
| machine. The connection to the whole house was blocked
| off until I stopped it.
| hombre_fatal wrote:
| My dad's small town ISP called him to say his household
| connection recently started saturating the link 24/7 and
| to look into whether a device had been compromised.
|
| (Turns out some raspi reseller shipped a product with
| empty uname/password)
|
| While a cute story, how do you scale that? And what about
| all the users that would be incapable of troubleshooting
| it, like if their laptop, roku, or smart lightbulb were
| compromised? They just lose internet?
|
| And what about a botnet that doesn't saturate your
| connection, how does your ISP even know? They get full
| access to your traffic for heuristics? What if it's just
| one curl request per N seconds?
|
| Not many good answers available if any.
| mschuster91 wrote:
| > While a cute story, how do you scale that? And what
| about all the users that would be incapable of
| troubleshooting it, like if their laptop, roku, or smart
| lightbulb were compromised? They just lose internet?
|
| Uh, _yes_. Exactly and plainly that. We also go and
| suspend people 's driver licenses or at the very least
| seriously fine them if they misbehave on the road,
| including driving around with unsafe cars.
|
| Access to the Internet should be a privilege, not a
| right. Maybe the resulting anger from widespread
| crackdowns would be enough of a push for legislators to
| demand better security from device vendors.
|
| > And what about a botnet that doesn't saturate your
| connection, how does your ISP even know?
|
| In ye olde days providers had (to have to) abuse@
| mailboxes. Credible evidence of malicious behavior
| reported to these did lead to customers getting told to
| clean up shop _or else_.
| codedokode wrote:
| In many cases the infected devices are in developing
| countries where none of your customers is. Many sites are
| regional, for example, a medium business operating within
| one country, or even city.
|
| And even if the attack comes from your country, it is
| better to block part of the customers and figure out what
| to do next rather than have your site down.
| gnfargbl wrote:
| The recent Azure DDoS used 500k botnet IPs. These will have
| been widely distributed across subnets and countries, so
| your blocking approach would not have been an effective
| mitigation.
|
| Identifying and dynamically blocking the 500k offending IPs
| would certainly be _possible_ technically -- 500k /32s is
| not a hard filtering problem -- but I seriously question
| the operational ability of internet providers to perform
| such granular blocking in real-time against dynamic
| targets.
|
| I also have concerns that automated blocking protocols
| would be widely abused by bad actors who are able to
| engineer their way into the network at a carrier level
| (i.e. certain governments).
| tw04 wrote:
| It also completely overlooks the fact that some of the
| traffic has spoofed source IP addresses and a bad actor
| could use automated black holing to knock a legitimate
| site offline.
| codedokode wrote:
| > a bad actor could use automated black holing to knock a
| legitimate site offline.
|
| No, in my concept the host can only manage the traffic
| targeted at it and not at other hosts.
| tw04 wrote:
| That already exists... that's part of cloudflare and
| other vendors mitigation strategy. There's absolutely no
| chance ISPs are going to extend that functionality to
| random individuals on the internet.
| __alexs wrote:
| > 500k /32s is not a hard filtering problem
|
| Is this really true? What device in the network are you
| loading that filter into? Is it even capable of handling
| the packet throughput of that many clients while also
| handling such a large block list?
| nine_k wrote:
| But this is not one subnet. It is a large number of IPs
| distributed across a bunch of providers, and handled
| possibly by dozens if not hundreds of routers along the
| way. Each of these routers won't have trouble blocking a
| dozen or two IPs that would be currently involved in a
| DDoS attack.
|
| But this would require a service like DNSBL / RBL which
| email providers use. Mutually trusting big players would
| exchange lists of IPs currently involved in DDoS attacks,
| and block them way downstream in their networks, a few
| hops from the originating machines. They could even
| notify the affected customers.
|
| But this would require a lot of work to build, and a
| serious amount of care to operate correctly and
| efficiently. ISPs don't seem to have a monetary incentive
| to do that.
| kordlessagain wrote:
| Your argument is technically flawed.
|
| In a CDN, customers consume bandwidth; they do not contribute
| it. If Cloudflare adds 1 million free customers, they do not
| magically acquire 1 million extra pipes to the internet
| backbone. They acquire 1 million new liabilities that require
| more infrastructure investment.
|
| All you are doing is echoing their pitch book. Of course they
| want to skim their share of the pie.
| __alexs wrote:
| I imagine every single customer is provisioned based on
| some peak expected typical traffic and that's what they
| base their capital investment in bandwidth on.
|
| However most customers are rarely at their peak, this gives
| you tremendous spare capacity to use to eat DDoS attacks,
| assuming that the attacks are uncorrelated. This gives you
| huge amounts of capacity that's frequently doing nothing.
| Cloudflare advertise this spare capacity as "DDoS
| protection."
|
| I suppose in theory it might be possible to massively
| optimise utilisation of your links, but that would be at
| the cost of DDoS protection and might not improve your
| margin very meaningfully, especially is customers care a
| lot about being online.
| LMYahooTFY wrote:
| You're missing the economies of scale part.
|
| OP is saying it's cheaper overall for a 10 million customer
| company to add infrastructure for 1 million more than it is
| for a 10,000 customer company to add infrastructure for
| 1000 more people.
|
| If you're looking at this as a "share of the pie", it's
| probably not going to make sense. The industry is not zero
| sum.
| bawolff wrote:
| > In a CDN, customers consume bandwidth; they do not
| contribute it
|
| They contribute money which buys infrastructure.
|
| > If Cloudflare adds 1 million free customers,
|
| Is the free tier really customers? Regardless most of them
| are small that it doesn't cost cloudflare much anyways. The
| infrastructure is already there anyways. Its worth it to
| them for the good will it generates which leads to future
| paying customers. It probably also gives them visibility
| into what is good vs bad traffic.
|
| 1 million small sites could very well cost less to
| cloudflare than 1 big site.
| jiveturkey wrote:
| You aren't understanding economy of scale, and peak to
| average ratios.
|
| The same reason I use cloud compute -- elastic
| infrastructure because I can't afford the peaks -- is the
| same reason large service providers "work".
|
| It's funny how we always focus on Cloudflare, but all cloud
| providers have this same concentration downside. I think
| it's because Cloudflare loves to talk out of both sides of
| their mouth.
| kordlessagain wrote:
| The "economies of scale" defense of Cloudflare ignores a
| fundamental reality: 23.8 million websites run on
| Cloudflare's free tier versus only 210,000 paying
| customers or so. Free users are not a strategic asset.
| They are an uncompensated cost, full stop. Cloudflare
| doesn't absorb this loss out of altruism; they monetize
| it by building AI bot-detection systems, charging for bot
| mitigation, and extracting threat intelligence data.
| Today's outage was caused by a bug in Cloudflare's
| service to combat bots.
|
| That's AI bots, BTW. Bots like Playwright or Crawl4AI,
| which provide a useful service to individuals using
| agentic AI. Cloudflare is hostile to these types of
| users, even though they likely cost websites nothing to
| support well.
|
| The "scale saves money" argument commits a critical
| error: it counts only the benefits of concentration while
| externally distributing the costs.
|
| Yes, economies of scale exist. But Cloudflare's scale
| creates catastrophic systemic risk that individual
| companies using cloud compute never would. An estimated
| $5-15 billion was lost for every hour of the outage
| according to Tom's Guide. That cost didn't disappear. It
| was transferred to millions of websites, businesses, and
| users who had zero choice in the matter.
|
| Again, corporations shitting on free users. It's a bad
| habit and a dark pattern.
|
| Even worse, were you hoping to call an Uber this morning
| for your $5K vacation? Good luck.
|
| This is worse than pure economic inefficiency. Cloudflare
| operates as an authorized man-in-the-middle to 20% of the
| internet, decrypting and inspecting traffic flows. When
| their systems fail, not due to attacks, but to internal
| bugs in their monetization systems, they don't just lose
| uptime.
|
| They create a security vulnerability where encrypted
| connections briefly lose their encryption guarantee.
| They've done this before (Cloudbleed), and they'll do it
| again. Stop pretending to have rational arguments with
| irrational future outcomes.
|
| The deeper problem: compute, storage, and networking are
| cheap. The "we need Cloudflare's scale for DDoS
| protection" argument is a circular justification for the
| very concentration that makes DDoS attractive in the
| first place. In a fragmented internet with 10 CDNs, a
| successful DDoS on one affects 10% of users. In a
| Cloudflare-dependent internet, a DDoS, or a bug, affects
| 50%, if Cloudflare is unable to mitigate (or DDoSs
| themselves).
|
| Cloudflare has inserted themselves as an unremovable
| chokepoint. Their business model depends on staying that
| chokepoint. Their argument for why they must stay a
| chokepoint is self-reinforcing. And every outage proves
| the model is rotten.
| jiveturkey wrote:
| hang on, you're reading some kind of cloudflare advocacy
| in my post. apologies if i implied that. i don't like to
| come off as a crank is all. IMO cloudflare is an evil
| that needs to be defeated. i'm just explaining how their
| business model "works" and why massive economy of scale
| matters, to support the GP poster.
|
| i don't even think they are evil because of the
| concentration of power, that's just a problematic issue.
| the evil part is they convince themselves they aren't the
| bad guys. that they are saving us from ourselves. that
| the things they do are net positives, or even absolute
| positives. like the whole "let's defend the internet from
| AI crawlers" position they appointed themselves sheriff
| on, that i think you're referencing. it's an extremely
| dangerous position we've allowed them to occupy.
|
| > they monetize it
|
| yes, and they can't do this without the scale.
|
| > scale saves money
|
| any company, uber for example, can design their infra to
| not rely on a sole provider. but why? their customers
| aren't going to leave in droves when a pretty reliable
| provider has the occasional hiccup. so it's not worth the
| cost, so why shouldn't they externalize it? uber isn't in
| business to make the internet a better place. so yes,
| scale does save money. you're arguing something at a
| higher principle than how architectural decisions are
| made.
|
| i'm not defending economy of scale as a necessary evil.
| i'm just backing up that it's how cloudflare is built,
| and that it is in fact useful to customers.
| karmelapple wrote:
| And how many companies want to also be able to build out
| their own CDN?
|
| Not every company can be an expert at everything.
|
| But perhaps many of us could buy a different CDN than the
| major players if we want to reduce the likelihood of mass
| outages like this though.
| hhthrowaway1230 wrote:
| Don't think there is anything wrong with a centralised service
| being down, you just make a conscious decision if you want that
| and can afford that?
|
| People not being ready for cloudflare/[insert hyperscaler] to
| be possibly down is the only fault.
| sotix wrote:
| This might sound crazy as a software engineer, but I actually
| like the occasional "snow day" where everything goes down. It's
| healthy for us to all disconnect from the internet for a bit.
| The centralization unintentionally helps facilitate that. At
| least, that's my glass half full perspective.
| gspencley wrote:
| I can understand that sentiment. Just don't lose sight of the
| impact it can have on every day people. My wife and I own a
| small theatre and we sell tickets through Eventbrite. It's
| not my full time job but it is hers. Eventbrite sent out an
| email this morning letting us know that they are impacted by
| the outage. Our event page appears to be working but I do
| wonder if it's impacting ticket sales for this weekend's
| shows.
|
| So while us in tech might like a "snow day", there are
| millions of small businesses and people trying to go about
| their day to day lives who get cut off because of someone
| else's fuck-ups when this happens.
| telepromptereye wrote:
| Absolutely solid point; there are a couple of apps I use
| daily for productivity, chores, even for alarm scheduling,
| that with the free versions, the ads wouldn't load so I
| couldn't use them but some of them were updated already.
| Made me realize I forgot that we're kind of like cyborgs
| relying on technology that's integrated so deeply into our
| lives that all it takes is an EMP blast like a monopolistic
| service going down to bring -us- down until we take a
| breath and learn how to walk again. Wild time.
| hashim wrote:
| I'm guessing you're employed and your salary is guaranteed
| regardless. Would you have the same outlook if you were the
| self-employed founder of an online business and every minute
| of outage was costing you money?
| nondrool wrote:
| What are you paying in order to be down?
|
| Even if you were making a million a minute, typically, it
| still didn't cost you a thing, nor have you lost anything.
|
| You're not making as much, sure, but neither a cost, nor a
| loss.
| hashim wrote:
| For businesses whose profit margins are already slim,
| which is most traditional businesses trading online,
| making less money than they usually would will put them
| into the red, and even for those that are still in
| profit, making less money than you usually would means
| you have less money to pay the expenses that you usually
| do, expenses that are predicated on you making a certain
| amount of revenue.
| vntok wrote:
| If you're an event organizer whose big event is in two
| days, for example, then every minute your website's down
| translates to people not paying to attend your paid
| event. Bonus points because as event managers know,
| people often wait until 2 days before the event to
| subscribe for good. Bonus points if you knew this and
| therefore ran a costly email campaign just before the
| outage, a campaign that is now sitting at a near-0% click
| rate.
|
| Don't ask me how I know.
| gosub100 wrote:
| your house isn't going into foreclosure because your shop
| went down for a day.
| hashim wrote:
| That's a weirdly flippant response to what's a serious
| issue, but I'll give it the courtesy of a reply anyway -
| maybe not, but a business not making enough profit might
| go under, or they might _only_ have to fire someone to
| prevent that from happening.
| homebrewer wrote:
| You're living in a bubble. I know enough people who live
| paycheck to paycheck and always have exactly $0 in their
| pocket before the end of the month. It's pretty normal in
| some parts of the world, maybe even most of them.
| LarryClapp wrote:
| I mean, you don't really know that, do you?
|
| Maybe Tuesdays tend to be a big day for me, and instead
| of "down for a day", it's "lose almost a quarter of my
| income for the month".
|
| Cloudflare is pretty pervasive, there are all kinds of
| people and businesses, in all kinds of situations,
| impacted by this.
| ljm wrote:
| If the internet was just social media, SaaS productivity
| suites, and AI slop, sure...
|
| But there are systems that depend on Cloudflare, directly or
| not, and when they go down it can have a serious impact on
| somebody's livelihood.
| cultofmetatron wrote:
| > This might sound crazy as a software engineer, but I
| actually like the occasional "snow day" where everything goes
| down
|
| As as software engineer, I get it. as a CTO, I spent this
| morning triaging with my devops ai(actual Indian) to find
| some workaround (we found one) while our CEO was doing damage
| control with customers (non technical field) who were angry
| that we were down and they were losing business by the
| minute.
|
| sometimes I miss not having a direct stake in the success of
| the business.
| strict9 wrote:
| And all of these outages happening not long after most of them
| dismissed a large amount of experienced staff while moving jobs
| offshore to save in labor costs.
| Lammy wrote:
| It's because single points of traffic concentration are the
| most surveillable architecture, so FVEY et al economically
| reward with one hand those companies who would build the
| architecture they want to surveil with the other hand.
| rcarmo wrote:
| What is worrying is that distributed systems don't seem to be
| that distributed in practice.
| expedition32 wrote:
| Designed to survive a first strike from the USSR. Taken down
| by Cloudflare.
| kazen44 wrote:
| oh mind you, the actual routing architecture of the
| internet was. Cloudflare being down mainly just affects
| layer 7...
| drob518 wrote:
| IMO, centralization is inevitable because the fundamental
| forces drive things in that direction. Clouds are useful for a
| variety of reasons (technical, time to market, economic), so
| developers want to use them. But clouds are expensive to build
| and operate, so there are only a few organizations with the
| budget and competency to do it well. So, as the market matures
| you end up with 3 to 5 major cloud operators per region, with
| another handful of smaller specialists. And that's just the way
| it works. Fighting against that is to completely swim upstream
| with every market force in opposition.
| mobiuscog wrote:
| How did we get to a place where Cloudflare being down means we
| see an outage page, but on that page it tells us explicitly
| that the host we're trying to connect to is up, and it's just a
| Cloudflare problem.
|
| If it can tell us that the host is up, surely it can just
| bypass itself to route traffic.
| ralferoo wrote:
| "... surely it can just ..."
|
| Congratulations, you've successfully completed Management
| Training 101.
| kordlessagain wrote:
| The technical term for it is a man in the middle. It's better
| to call it what it is that way you aren't fooled into thinking
| it's not, because it is.
| an-allen wrote:
| Well the centralisation without rapid recovery and practices
| that provide substantial resiliency... that would be worrying.
|
| But I dare say the folks at these organisations take these
| matters incredibly seriously and the centralisation problem is
| largely one of risk efficiency.
|
| I think there is no excuse, however, to not have multi region
| on state, and pilot light architectures just in case.
| kilpikaarna wrote:
| Currently at the public library and I can't use the customer
| inventory terminals to search for books. They're just a web
| browser interface to the public facing website, and it's hosted
| behind CF. Bananas.
| gmiller123456 wrote:
| Don't forget the CloudStrike outage: One company had a bug that
| brought down almost everything. Who would have thought there
| are so many single points of failure across the entire
| Internet.
| baq wrote:
| because efficiency trumps redundancy in the short term, which
| is all that matters in a super competitive environment.
| bsoles wrote:
| > How did we get to a place where either Cloudflare or AWS
| having an outage means a large part of the web going down?
|
| As always, in the name of "security". When are we going to
| learn that anything done, either by the government or by a
| corporation, in the name of security is always bad for the
| average person?
| alkonaut wrote:
| Because it's better to have a really convenient and cheap
| service that works 99% of the time, than a resilient that is
| more expensive or more cumbersome to use.
|
| It's like github vs whatever else you can do with git that is
| truly decentralized. The centralization has such massive
| benefits that I'm very happy to pay the price of "when it's
| down I can't work".
| ljm wrote:
| I would be less worried if Cloudflare and AWS weren't involved
| in many more things than simply running DNS.
|
| AWS - someone touches DynamoDB and it kills the DNS.
|
| Cloudflare - someone touches functionality completely unrelated
| to DNS hosting and proxying and, naturally, it kills the DNS.
|
| There is this critical infrastructure that just becomes one
| small part of a wider product offering, worked on by many
| hands, and this critical infrastructure gets taken down by what
| is essentially a side-effect.
|
| It's a strong argument to move to providers that just do one
| thing and do it well.
| rglover wrote:
| Most developers don't care to know how the underlying
| infrastructure works (or why) and so they take whatever the
| public consensus is re: infra as a statement of fact (for the
| better part of the last 15 years or so that was "just use the
| cloud"). A shocking amount of technical decisions are socially,
| not technically enforced.
| mvkel wrote:
| This was always the case. There was always a "us-east" in some
| capacity, under Equinix, etc. Except it used to be the only
| "zone," which is why the internet is still so brittle despite
| having multiple zones. People need to build out support for
| different zones. Old habits die hard, I guess.
| telepromptereye wrote:
| We take the idea of the internet always being on for granted.
| Most people don't understand the stack and assume that when
| sites go down it's isolated, and although I agree with you,
| it's just as much complacency and lack of oversight and
| enforcement delays in bureaucracy as it is centralization. But
| I guess that's kind of the umbrella to those things... lol
| moralestapia wrote:
| 5 mins. of thought to figure out why these services exist?
|
| Dialogue about mitigations/solutions? Alternative services?
| High availability strategies?
|
| Nah! It's free to complain.
|
| Me personally, I'd say those companies do a phenomenal job by
| being a de facto backbone of the modern web. Also Cloudflare,
| in particular, gives me a lot of things for free.
| kaonwarb wrote:
| I recommend this Ben Thompson piece on why resiliency has
| declined: https://stratechery.com/2025/resiliency-and-scale/
| exasperaited wrote:
| Re: Cloudflare it is because developers actively pushed "just
| use Cloudflare" again and again and again.
|
| It has been dead to me since the SSL cache vulnerability thing
| and the arrogance with which senior people expected others to
| solve their problems.
|
| But consider how many people still do stupid things like use
| the default CDN offered by some third party library, or use
| google fonts directly; people are lazy and don't care.
| abtinf wrote:
| Because they are great services, are generally pretty easy to
| get started with, and usually work as expected, which has led
| to broad adoption.
| BurningFrog wrote:
| How many more of these until governments step in and take over
| "critical infrastructure"?
| GuB-42 wrote:
| Another one that worries me is Let's Encrypt.
|
| It is not as bad as Cloudflare or AWS because certificates will
| not expire the instant there is an outage, but considers that:
|
| - It serves about 2/3 of all websites
|
| - TLS is becoming more and more critical over time. If
| certificates fail, the web may as well be down
|
| - Certificate lifetimes are becoming shorter and shorter, now
| 90 days, but Let's Encrypt is now considering 6 days, with 47
| days being planned as a minimum
|
| - An outage is one thing, but should a compromise happen, that
| would be even more catastrophic
|
| Let's Encrypt is a good guy now, but remember that Google used
| to be a good guy in the 2000s too!
| seniorThrowaway wrote:
| Agree, I've thought about this one too. The history of
| SSL/TLS certs is pretty hacky anyway in my opinion. The main
| problem they are solving really should have been solved at
| the network layer with ubiquitous IPsec and key distribution
| via DNS since most users just blindly trust whatever root CAs
| ship with their browser or OS, and the ecosystem has been
| full of implementation and operational issues.
|
| Let's Encrypt is great at making the existing system less
| painful, and there are a few alternatives like ZeroSSL, but
| all of this automation is basically a pile of workarounds on
| top of a fundamentally inappropriate design.
| rtkwe wrote:
| There's not really a way around the initial trust problem
| with consumer oriented certs though. Yours could reduce the
| number of initially trusted down to one I think but not any
| further.
| kazen44 wrote:
| its a shame DANE never took off. If we actually got around
| to running a trusted DNSSEC based DNS system and allowed
| clients to create certificates thanks to DANE, we would be
| in a far more resilient setup compared to what we are now.
|
| But DNSSEC was hard according to some, and now we are
| running a massive SPOF in terms of TLS certificates.
| b00ty4breakfast wrote:
| Google was always a for-profit operation. Let's Encrypt/ISRG
| could still go rotten but there are less incentives for them
| to do so as a non-profit.
| phasmantistes wrote:
| (Disclaimer: I am tech lead of Let's Encrypt software
| engineering)
|
| I'm also concerned about LE being a single point of failure
| for the internet! I really wish there were other free and
| open CAs out there. Our goal is to encrypt the web, not to
| perpetuate ourselves.
|
| That said, I'm not sure the line of reasoning here really
| holds up? There's a big difference between this three-hour
| outage and the multi-day outage that would be necessary to
| prevent certificate renewal, even with 6-day certs. And
| there's an even bigger difference between this sort of
| network disruption and the kind of compromise that would be
| necessary to take LE out permanently.
|
| So while yes, I share your fear about the internet-wide
| impact of total Let's Encrypt collapse, I don't think that
| these situations are particularly analogous.
| phendrenad2 wrote:
| Because DDoS is a fact of life (and even if you aren't targeted
| by DDoS, the bot traffic probing you to see if you can be made
| part of the botnet is enough to take down a cheap $5 VPS). So
| we have to ask - why? Personally, I don't accept the hand-wavy
| explanation that botnets are "just a bunch of hacked IoT
| devices". No, your smart lightbulb isn't taking down Reddit. I
| slightly believe the secondary explanation that it's a bunch of
| hacked home routers. We know that home routers are full of
| things like suspicious oopsie definitely-not-government
| backdoors.
| kahrl wrote:
| When there is an accident on the interstate we should blame the
| centralization of traffic and advocate for no more highways.
|
| Very worrying indeed.
| bawolff wrote:
| The same reason we have centralization across the economy.
| Economies of scale is how you make a big business succesful,
| and once you are on top its hard to dislodge you.
| cyanydeez wrote:
| CAPITALISM
|
| Are people really this confused?
| peacebeard wrote:
| A lot of products use AWS because "we could build redundancy
| and multi-region if we need it" and then never build it.
| rtkwe wrote:
| I think some of the issues in the last outage actually
| affected multiple regions. IIRC internally some critical
| infrastructure for AWS depends on us-east-1 or at least it
| failed in a way that didn't allow failover.
| giantrobot wrote:
| People use CloudFlare because it's a "free" way for most sites
| to not get exploited (WAF) or DDoSed (CDN/proxy) regularly. A
| DDoS can cost quite a bit more than a day of downtime, even
| just a thundering herd of legitimate users can explode an
| egress bill.
|
| It sucks there's not more competition in this space but
| CloudFlare isn't widely used for no reason.
|
| AWS also solves real problems people have. Maintaining
| infrastructure is expensive as is hardware service and
| maintenance. Redundancy is even harder and more expensive. You
| can run a fairly inexpensive and performant system on AWS for
| years for the cost of a single co-located server.
| 0xbadcafebee wrote:
| It's not really. People are just very bad at putting the things
| around them into perspective.
|
| Your power is provided by a power utility company. They usually
| serve an entire state, if not more than one (there are smaller
| ones too). That's "centralization" in that it's one company,
| and if they "go down", so do a lot of businesses. But actually
| it's not "centralized", in that 1) there are actually many
| different companies across the country/world, and 2) each
| company "decentralizes" most of its infrastructure to prevent
| massive outages.
|
| And yes, power utilities have outages. But usually they are
| limited in scope and short-lived. They're so limited that most
| people don't notice when they happen, unless it's a giant
| weather system. Then if it's a (rare) large enough impact,
| people will say "we need to reform the power grid!". But later
| when they've calmed down, they realize that would be difficult
| to do without making things worse, and this event isn't common.
|
| Large internet service providers like AWS, Cloudflare, etc, are
| basically internet utilities. Yes they are large, like power
| utilities. Yes they have outages, like power utilities. But the
| fact that a lot of the country uses them, isn't any worse than
| a lot of the country using a particular power company. And
| unlike the power companies, we're not really that dependent on
| internet service providers. You can't really change your power
| company; you can change an internet service provider.
|
| Power didn't used to be as reliable as it is. Everything we
| have is incredibly new and modern. And as time has passed, we
| have learned how to deal with failures. Safety and reliability
| has increased throughout critical industries as we have learned
| to adapt to failures. But that doesn't mean there won't be
| failures, or that we can avoid them all.
|
| We also have the freedom to architect our technology to work
| around outages. All the outages you have heard about recently
| could be worked around, if the people who built on them had
| tried:
|
| - CDN goes down? Most people don't absolutely need a CDN. Point
| your DNS at your origins until the CDN comes back. (And
| obviously, your DNS provider shouldn't be the same as your
| CDN...)
|
| - The control plane goes down on dynamic cloud APIs? Enable a
| "limp mode" that persists existing infrastructure to serve your
| core needs. You should be able to service most (if not all) of
| your business needs without constantly calling a control plane.
|
| - An AZ or region goes down? Use your disaster recovery plan:
| deploy infrastructure-as-code into another region or AZ.
| Destroy it when the az/region comes back.
|
| ...and all of that just to avoid a few hours of downtime per
| year? It's likely cheaper to just take the downtime. But that
| doesn't stop people from piling on when things go wrong,
| questioning whether the existence of a utility is a good idea.
| neop1x wrote:
| It's not only centralization in the sense your website will be
| down if they are down but it is also a centralized MITM proxy.
| If you transfer sensitive data like chats over
| cloudflare-"protected" endpoints, you also allow CF to
| transparently read and analyze it in plain-text. It must be
| very easy for state agencies to spy on the internet nowadays,
| they woukd just ask CF to redirect traffic to them.
| thenthenthen wrote:
| ...sneaks in Azure
| paulddraper wrote:
| Because bots are a real thing.
|
| And it's hard to protect against DDoS without something like
| Cloudflare.
|
| Look at the posts here.
|
| Even the meager HN "hug of death" will take things down
| butlike wrote:
| It's weird to think about so bear with me. I don't mean this
| sardonically or misanthropically. But, it's "just the
| internet." It's just the internet. It dones't REALLY matter in
| a large enough macro view. It's JUST the internet.
| ronald_petty wrote:
| Consider joining the Internet Society. An entire group of
| people who care!
| chasing0entropy wrote:
| Agreed. More worrying is that it appears standard practice or
| separation between domain and nameserver administration has
| been lost to one-stop-shop marketing.
| ridgeguy wrote:
| Short-term economic forces, probably. Centralization is often
| cheaper in the near term. The cost of designing in single-point
| failure modes gets paid later.
| gist wrote:
| There is this tendency to phrase questions (or statements) as
| "when did 'we' ".
|
| These decision are made individually not centrally. There is no
| process in place (and most likely there will never be) that
| will be able to control and dictate if people decide one way of
| doing things is the best way to do it. Even assuming they
| understand everything or know of the pitfalls.
|
| Even if you can control individually what you do for the site
| you operate (or are involved in) you won't have any control on
| parts of your site (or business) that you rely on where others
| use AWS or Cloudflare.
| poemxo wrote:
| For most services it's safer to host from behind Cloudflare,
| and Cloudflare is considered more highly available than a
| single IaaS or PaaS, at least in my headcanon.
| glitchc wrote:
| All decentralized systems tend to centralization over time.
| worldsavior wrote:
| Almost every site I'm trying to connect to is down. The internet
| is way too centralized.
| philiproberts wrote:
| Is this why x.com isn't working right?
| dcjdfvk wrote:
| Even pornhub is down becuase it use cloudflare.
| pc2g4d wrote:
| Does requiring proof-of-work in order to connect accomplish 99%
| of what Cloudflare does?
| mirzap wrote:
| Not even 1%
| SallysField wrote:
| Puup but, anyone else?
| 0xblinq wrote:
| All bets on DNS being the root cause.
| 6d6b73 wrote:
| Half of the internet is down. That's what you get for giving up
| the control of the service that suppose to be decentralized to
| one company. Good, maybe if it costs companies few billions they
| will not put all eggs in one basket.
| keithluu wrote:
| Some CDNs are down too, for example cdn.tailwindcss.com And
| apparently I can't log into Hackernews?
| asim wrote:
| DigitalOcean + Gandi means nothing I run is down. Amazing. We
| depend far too greatly on centralised services where we deem the
| value of reputation and convenience exceeds the potential
| downsides and then the world pays for it. I think we have to feel
| a lot more of this pain before regulation kicks in to change
| things because the reality is people don't change. The only thing
| you can personally do is run a lot of your own stuff for things
| you can.
| luckylion wrote:
| Individually, for you, what's the difference?
|
| You use a service provider, if that service provider is down,
| your site is down. Does it matter to you that others are also
| down in that instance?
| yreg wrote:
| Might even be better to go down at the same time as everyone
| else, because customers might be more lenient on you.
| The_President wrote:
| DigitalOcean is indeed having issues.
|
| > Application error: a client-side exception has occurred while
| loading www.digitalocean.com (see the browser console for more
| information).
|
| Yellow flags on status.digitalocean.com *
| asim wrote:
| Still nothing is down...
| dcjdfvk wrote:
| God my favourite website pornhub.com is also down why onn earth
| cloudflare i just now came from school.
| SallysField wrote:
| Puup but, how about yous guys
| mhuesler wrote:
| how are we still down? it's been 2 hours lol
| pc2g4d wrote:
| Meanwhile my Wordpress blog on DigitalOcean is up. And so is
| DigitalOcean.
|
| My ISP is routing public internet traffic to my IPs these days.
| What keeps me from running my blog from home? Fear of exposing a
| TCP port, that's what. What do we do about that?
| doikor wrote:
| > What keeps me from running my blog from home?
|
| Depending on the contract it might not be allowed to run public
| network services from your home network.
|
| I had a friend doing that and once his site got popular the ISP
| called (or sent a letter? don't remember anymore) with "take
| this 10x more expensive corporate contract or we will block all
| this traffic".
|
| In general why the ISPs don't want you to do that (in addition
| to way more expensive corporate rates) is the risk of someone
| DDoS that site which could cause issues to large parts of their
| domestic customers (and depending on the country be liable to
| compensate those customers for not providing a service they
| paid for)
| trollbridge wrote:
| Yeah, DigitalOcean and Dreamhost are both up. I actually self-
| host on 2Gig fibre service, and all my stuff is up, except I
| park everything behind Cloudflare since there is no way I could
| handle a DDoS attack.
| The_President wrote:
| > Our Engineering team is actively investigating an issue
| impacting multiple DigitalOcean services caused by an upstream
| provider incident. This disruption affects a subset of Gen AI
| tools, the App Platform, Load Balancer, Spaces and provisioning
| or management actions for new clusters. Existing clusters are
| not affected. Users may experience degraded performance or
| intermittent failures within these services.
|
| > We acknowledge the inconvenience this may cause and are
| working diligently to restore normal operations. Signs of
| recovery are starting to appear, with most requests beginning
| to succeed. We will continue to monitor the situation closely
| and provide timely updates as more information becomes
| available. Thank you for your patience as we work towards full
| service restoration.
|
| It's not down for you, but for others.
| wslh wrote:
| Garmin site not working for example, and they removed the export
| option from the mobile application though.
| tyjen wrote:
| Waking up to chaos. Nice.
| anshumankmr wrote:
| ChatGPT is Down. What will LinkedIn posters ever do?
| robwwilliams wrote:
| And Anthropic Claude
| re5i5tor wrote:
| For me right now, Claude.ai is down, but Claude Code
| (terminal, extension) seems to be up and happy. Suggests that
| API is probably up.
| sysstemlord wrote:
| They will embark on a journey of synergy, and be agile.
| anshumankmr wrote:
| Incoming Posts on LinkedIn:
|
| what ChatGPT and Claude being down taught me about b2b SaaS
| bob1029 wrote:
| I can now imagine a scenario where everyone has become so
| dependent on the AI tool that it going down could turn into an
| unanticipated black start event for the entire internet.
| anshumankmr wrote:
| https://tenor.com/view/obiwan-kenobi-disturbance-in-the-
| forc...
|
| I sense a great disturbance in the force... As if millions of
| cringefluencers suddenly cried out in terror cause they had
| to come up with an original thought.
| mrcnkoba wrote:
| For anyone reading this who desperately needs their website up,
| you can try this: If you manage to get to your Cloudflare DNS
| settings and disable the "Proxy status (Proxied)" feature (the
| orange cloud), it should start working again.
|
| Be aware that this change has a few immediate implications:
|
| - SSL/TLS: You will likely lose your Cloudflare-provided SSL
| certificate. Your site will only work if your origin server has
| its own valid certificate.
|
| - Security & Performance: You will lose the performance benefits
| (caching, minification, global edge network) and security
| protections (DDoS mitigation, WAF) that Cloudflare provides.
| kobalsky wrote:
| also the API was working fine while the dash was down.
|
| if you don't have the keys make sure to grab them for the next
| one.
| progmetaldev wrote:
| Unfortunately, this will also expose your IP address, which may
| leave you vulnerable even when the WAF and DDoS protections
| come back up (unless you take the time to only listen for
| Cloudflare IP address ranges, which could still take a beefy
| server if you're having to filter large amounts of traffic).
| creatonez wrote:
| This will also reveal your backend internal IP addresses.
| Anyone can find permanent logs of public IP addresses used by
| even obscure domain names, so potential adversaries don't
| necessarily have to be paying attention at the exact right time
| to find it.
| wrathofmonads wrote:
| Gemini and other agents are now failing when they search for
| something on the web. ChatGPT can't even be accessed.
| shreyrey wrote:
| The privacy kingpin in india, has been caught and most of the
| network are affected. We will be resuming the servers from Sweden
| shortly.
| paraboul wrote:
| All trains are stuck in south of France for << broken
| signalisation >>. Wonder how related this is.
|
| Edit: it was related
|
| https://www.laprovence.com/article/region/83645099971988/pan...
|
| Edit2: They edited the article stating it wasn't related.
| misswaterfairy wrote:
| If they do use Cloudflare... why in the everlasting name of
| Hell did they connect a railway control and signalling system
| to the Internet?!!!
| amenghra wrote:
| Because javascript programmers are cheaper/easier/whatever to
| hire? So everything becomes web-centric. (I'm hoping for this
| comment to be sarcastic but I wouldn't be surprised if it
| turns out not to be)
| mghackerlady wrote:
| Don't say that, you might hurt the JS devs feelings
| oxy1tumn wrote:
| Update: Cloudflare has announced they will be sacrificing their
| CEO at the alter in penance
| longsword wrote:
| used a down-detector site to check if cloudflare is down, but the
| site is running on cloudflare, so i couldnt check if cloudflare
| was down for anyone else, because cloudflare was down
| ropable wrote:
| There's something maliciously satisfying about seeing your own
| self-hosted stuff working while things behind Cloudflare or AWS
| are broken. Sure, they have like four more nines that me, but
| right now I'm sitting pretty.
| ArcHound wrote:
| My (s)crappy personal site was up during the AWS outage, Azure
| outage and now Cloud flare outage. And I have it for 2 months
| only! Maybe I can add a tracker somewhere, might be fun.
| FinnKuhn wrote:
| Can recommend Uptime Kuma for this purpose:
| https://github.com/louislam/uptime-kuma
| dizhn wrote:
| Where would you put it? :) (I have one on the oracle free
| tier)
| ArcHound wrote:
| I actually have exactly this, paired with ntfy. I was
| thinking more like badges: "was up during AWS outage in
| 2025" etc.
| nosferath wrote:
| How do you deal with DNS? I'm hosting something on a Raspberry
| Pi at home, and I had recently moved the DNS to Cloudflare.
| It's quite funny seeing my small personal website being down,
| although quite satisfying seeing both the browser and host with
| a green tick while Cloudflare is down.
| gucci-on-fleek wrote:
| > How do you deal with DNS?
|
| DNS is actually one of the easiest services to self-host, and
| it's fairly tolerant of downtime due to caching. If you want
| redundancy/geographical distribution, Hurricane Electric has
| a free secondary/slave DNS service [0] where they'll
| automatically mirror your primary/master DNS server.
|
| [0]: https://dns.he.net/
| sdoering wrote:
| I don't have experience with a dynDNS setup like you
| describe, hosting from (probably) home. But my domains are on
| a VPS (and a few other places here and there) and DNS is done
| via my domain reseller's DNS settings pages.
|
| Never had an issue hosting my stuff, but as said - don't yet
| have experience hoting something from home with a more
| dynamic DNS setup.
| kristofferR wrote:
| Pangolin is awesome. It's like self-hosted Cloudflare Tunnels
|
| https://github.com/fosrl/pangolin
| kilroy123 wrote:
| My selfhosted sites are down because of the cloudflares proxy.
| Ugh
| rubatuga wrote:
| Only my 'www' is affected ie my blog. Other self hosted like
| jellyfin and vaultwarden work fine
| dizhn wrote:
| I was just able to save a proxied site. Then the dashboard went
| down again. I didn't even know it was still on. It's really not
| doing anything for performance because the traffic is quite
| low.
| jim180 wrote:
| just a couple of days ago, I've moved my self hosted stuff from
| Cloudflare :)
| dnw wrote:
| This is a real problem for some some "old-school enterprise"
| companies that use Oracle, SAP, etc. along with the new AWS/CF
| based services. They are all waiting around for new apps to
| come back up while their Oracle suite/SAP are still
| functioning. There is a lesson here for some of these new
| companies selling to old-school companies.
| ekianjo wrote:
| Cloudfare completely breaking the internet...
| DASD wrote:
| Interesting(unnerving?) to see a number of domain registrars that
| offer their own DNS services utilize at least some kind of
| Cloudflare service for at least their own web fronts. Did a check
| on 6 registrar sites I currently interact with and half were
| down(Namecheap/Spaceship, Name, Dynadot) and up(Porkbun, Gandi,
| GoDaddy).
| madjam002 wrote:
| I just considered moving from Namecheap to Porkbun as Namecheap
| is down, but Porkbun use Cloudflare for their CAPTCHA meaning
| I'm unable to signup and I assume log in as well, so also no
| good.
| haarolean wrote:
| Porkbun also uses Cloudflare for their NS servers.
| 26thCreator wrote:
| I was reading some novels today and then Bam! a cliff hanger
| now...
| MaKey wrote:
| Maybe this incident will make people rethink putting Cloudflare
| blindly in front of every website.
| senfiaj wrote:
| In theory even a single company service could be distributed,
| so only a fraction of websites would be affected, thus it's not
| a necessity to be a single point of failure. So I still don't
| like this argument "you see what happens when over half of the
| internet relies on Cloudflare". And yes, I'm writing this as a
| Cloudflare user whose blog is now down because of this.
| Cloudflare is still convenient and accessible for many people,
| no wonder why it's so popular.
|
| But, yeah, it's still a horrible outage, much worse than the
| Amazon one.
| mallets wrote:
| The "omg centralized infra" cries after every such event kind
| of misses the point. Hosting with smaller companies (shared,
| vps, dedi, colo whatever) will likely result in far worse
| downtimes, individually.
|
| Ofc the bigger perception issue here is many services going
| out at the same time, but why would (most) providers care if
| their annual downtime does or doesn't coincide with others?
| Their overall reliability is no better or worse had only
| their service gone down.
|
| All of this can change ofc if this becomes a regular thing,
| the absolute hours of downtime does matter.
| senfiaj wrote:
| Exactly.
| uni_baconcat wrote:
| I'm glad I have kindle in my bag today. Websites down but not
| much we can do.
| roversx wrote:
| glad to see hacker news is still working.
| gynecologist wrote:
| ChatGPT was down so I couldn't work, go to lichess, turns out,
| it's down too now what do i do?
| delichon wrote:
| Touch grass.
| mghackerlady wrote:
| your username makes me wonder what exactly your job is, and
| what you need ChatGPT for
| samuelokoroafor wrote:
| This centralisation is worrisome. Single points of failures have
| always been a bad idea, especially when that point of failure is
| out of your control.
|
| PS:Someone really doesn't want Gemini 3 to get air time today
| gku wrote:
| Anyone seeing a link between AI-generated infra code and this
| year's wave in popular service outages?
| neop1x wrote:
| Your origin servers are protected now as no one can access them.
| Thanks for choosing CloudFlare's MITM "protection".
| 0xcb0 wrote:
| So they broke the internet. Nice! Never seen so many sites not
| working. Never seen so many desktop app suddenly stop working. I
| don't want to be the person responsible for this. And this again
| has thought me it's better to no rely on external services. Even
| though they seem to big to fail.
| arend321 wrote:
| I sometimes question my business decision to have a multi-cloud,
| multi-region web presence where it is totally acceptable to be
| down with the big boys.
| fbrchps wrote:
| That was something we discussed at my workplace.
|
| Prior hosting provider was a little-known company with decent
| enough track record, but because they employed humans, stuff
| would break. When it did break, C-suite would panic about how
| much revenue is lost, etc.
|
| The number of outages was "reasonable" to anyone who understood
| the technical side, but non-technical would complain for weeks
| after an outage about how we're always down, "well BigServiceX
| doesn't break ever, why do we?", and again lost revenue.
|
| Now on Azure/Cloudflare, we go down when everyone else does,
| but C-Suite goes "oh it's not just us, and it's out of our
| control? Okay let us know when it fixes itself."
|
| A great lesson in optics and perception, for our junior team
| members.
| souvlakee wrote:
| Many services have just disabled the CF proxy and use only DNS.
| If your end server has SSL and can handle some traffic, it
| might work for a while.
| krauses wrote:
| What do we actually lose going from cloud back to ground?
|
| The mass centralization is a massive attack vector for organized
| attempts to disrupt business in the west.
|
| But we're not doing anything about it because we've made a
| mountain at of a molehill. Was it that hard to manage everything
| locally?
|
| I get that there's plenty of security implications going that
| route, but it would be much harder to bring down t large portions
| of online business with a single attack.
| jillesvangurp wrote:
| > What do we actually lose going from cloud back to ground?
|
| A lot of money related to stuff you currently don't have to
| worry about.
|
| I remember how shit worked before AWS. People don't remember
| how costly and time consuming this stuff used to be. We had
| close to 50 people in our local ops team back in the day when I
| was working with Nokia 13 years ago. They had to deal with data
| center outages, expensive storage solutions failing, network
| links between data centers, offices, firewalls, self hosted
| Jira running out of memory, and a lot of other crap that I
| don't spend a lot of time about worrying with a cloud based
| setup. Just a short list of stuff that repeatedly was an issue.
| Nice when it worked. But nowhere near five nines of uptime.
|
| That ops team alone cost probably a few million per year in
| salaries alone. I knew some people in that team. Good solid
| people but it always seemed like a thankless and stressful job
| to me. Basically constant firefighting while getting people
| barking at you to just get stuff working. Later a lot of that
| stuff moved into AWS and things became a lot easier and the
| need for that team largely went away. The first few teams doing
| that caused a bit of controversy internally until management
| realized that those teams were saving money. Then that quickly
| turned around. And it wasn't like AWS was cheap. I worked in
| one of those teams. That entire ops team was replaced by 2-3
| clued in devops people that were able to move a lot faster.
| Subsequent layoff rounds in Nokia hit internal IT and ops teams
| hard early on in the years leading up to the demise of the
| phone business.
| dust-jacket wrote:
| Yeah, people have such short memories for this stuff. When we
| ran our own servers a couple of jobs ago, we had a rota of
| people who'd be on call for events like failing disks. I
| don't want to ever do that again.
|
| In general, I'm much happier with the current status of "it
| all works" or "it's ALL broken and its someone else's job to
| fix it as fast as possible"!
|
| Not saying its perfect but neither was on-prem/colocation
| Shuddown wrote:
| Wanted to check if it was DNS again but https://isitdns.com/ is
| also down...
| souvlakee wrote:
| We definitely need a version without CF.
| philipwhiuk wrote:
| Then the page will just be down for load reasons.
| mystifyingpoi wrote:
| This was geniuinely funny, thanks for that.
| Aldipower wrote:
| This sentence is slowly getting boring after all those recent
| outages: My web app hosted on Hetzner and BunnyCDN still works.
|
| That shows, the distributed nature of the internet is still
| there. It is a problem though, if everything is funneled through
| one provider.
| ndhikaa wrote:
| hahaha this is nuts, can we connect to each other without
| cloudflare?
| ivan-ivanov wrote:
| interesting that HN is not on Cloudflare but the YC website is
| behind Cloudflare so it's also down
| jordanb wrote:
| Like AWS I can't help but think we're going to get more and more
| of these as the tech industry continues to DOGE its workforce.
| bamboozled wrote:
| Vibe coding too
| dpacmittal wrote:
| Didn't realize Twitter uses cloudflare. It seems to be down as
| well
| philipwhiuk wrote:
| I believe it used to be AWS and they switched.
| rhim wrote:
| It's so crazy and scary that Cloudflare is the single point of
| failure for the internet.
| souvlakee wrote:
| But this decision is not determined by CF. It's how the devs
| decided.
| pohl wrote:
| Trying to figure out if this observation was intended to
| frame it so that it's less|same|more scary. The effect is
| more, but it sounds like the intention was less.
| poszlem wrote:
| The common pasture.
| GeorgeWoff25 wrote:
| I can't login in to Cloudflare itself because it is itself
| affected, holy bs
| petecooper wrote:
| https://news.ycombinator.com/user?id=jgrahamc
|
| >I was Cloudflare's CTO.
|
| A gentle reminder to not take any CF-related frustrations out on
| John today.
| philipwhiuk wrote:
| He's now on the Board so not left.
|
| Not that I think blaming individuals on forums who are already
| under stress is a good strategy anyway.
| tapanjk wrote:
| His personal website is down too.
| theideaofcoffee wrote:
| Oh no, we can't take a (former) executive to task about what
| they've wrought with their influence!!! That would be wrong.
|
| If anything, he should be the first to be blamed for the
| greater and greater effect this tech monster has on internet
| stability, since, you know, his people built it.
| ph1l337 wrote:
| Luckily for everyone including Guilhermo he can't dunk on the
| situation since x.com is down as well.
| kelvinzero wrote:
| Jakarta
| linhns wrote:
| Well, was reading the docs for Express, and shouted wtf a couple
| of times, before seeing this post on HN.
| HYUNGYUN wrote:
| Seoul down
| ck2 wrote:
| is there any way to get past challenges.cloudflare.com with
| tokens or something?
|
| so stupid there is no fallback and can take down 50% of the
| internet
|
| adding:looks like even Cloudflare's Silk Privacy Pass with
| challenge tokens is broken
|
| such a great idea to put half the web behind a single fail point
| without fallover
| arpit_mudgal wrote:
| BLR Down
| saanville wrote:
| cursed at me ISP for absolutely nothing
| tacker2000 wrote:
| Even ChatGPT.com is down! Wow!
| joeiq wrote:
| Seriously, bookmarking this site and checking it first next time
| instead of disabling all my ad blockers.
| makeavish wrote:
| NPM also seems to be down due to this! https://status.npmjs.org
| hoshikihao wrote:
| I was so scared, I thought my VPN had failed.
| fatty_patty89 wrote:
| I hecking love depending on big corpo
| bakigul wrote:
| This is a nightmare situation, we can't get in anywhere
| navigate8310 wrote:
| How come cloudflare.com is still working, do they not trust their
| own orange proxy service?
| davidmurdoch wrote:
| Speaking of 5 9s, how would _you_ achieve 5 9s for a basic CRUD
| app that doesn 't need to scale, but still be globally
| accessible? No auth, micro services, email or 3rd party services.
| Just a classic backend connected to a db (any db tech, hosted
| wherever), that serves up some html.
| christophilus wrote:
| Stock VPS somewhere like OVH or Hetzner, with a replica in a
| different provider?
| davidmurdoch wrote:
| Doesn't Hetzner carry the risk if getting kicked off on a
| whim? Only time I hear about them is when someone gets kicked
| out.
| kjuulh wrote:
| It depends on the infrastructure you're running on. There was a
| post yesterday going fairly into depth how you do such
| calculations https://authress.io/knowledge-
| base/articles/2025/11/01/how-w...
|
| You probably cannot achieve this with a single node, so you'll
| at least need to replicate it a few times to combat the normal
| 2-3 9s you get from a single node. But then you've got load
| balancers and dns, which can also serve as single point of
| failure, as seen with cloudflare.
|
| Depending on the database type and choice, it varies. If you've
| got a single node of postgres, you can likely never achieve
| more than 2-3 9s (aws guarantees 3 9s for a multi-az RDS). But
| if you do multi-master cockroach etc, you can maybe achieve 5
| 9s just on the database layer, or using spanner. But you'll
| basically need to have 5 9s which means quite a bit of
| redundancy in all the layers going to and from your app and
| data. The database and DNS being the most difficult.
|
| Reliable DNS provider with 5 9s of uptime guarantees -> multi-
| master load balancer each with 3 9s, -> each load balancer
| serving 3 or more apps each with 3 9s of availability, going to
| a database(s) with 5 9s.
|
| This page from google shows their uptime guarantees for big
| tables, 3 9s for a single region with a cluster. 4 9s for multi
| cluster and 5 9s for multi region
|
| https://docs.cloud.google.com/architecture/infra-reliability...
|
| In general it doesn't matter really what you're running, it is
| all about redundancy. Whether that is instances, cloud vendor,
| region, zone etc.
| misswaterfairy wrote:
| Part of the up-time solution is keeping as much of your app and
| infrastructure within your control, rather than being at the
| behest of mega-providers as we've witnessed in the past month:
| Cloudflare, and AWS.
|
| Probably:
|
| - a couple of tower servers, running Linux or FreeBSD, backed
| up by a UPS and an auto-run generator with 24 hours worth of
| diesel (depending on where you are, and the local areas
| propensity for natural disasters - maybe 72 hours),
|
| - Caddy for a reverse proxy, Apache for the web server,
| PostgreSQL for the database;
|
| - behind a router with sensible security settings, that also
| can load-balance between the two servers (for availability
| rather than scaling);
|
| - on static WAN IPs,
|
| - with dual redundant (different ISPs/network provider) WAN
| connections,
|
| - a regular and strictly followed patch and hardware
| maintenance cycle,
|
| - located in an area resistant to wildfire, civil unrest, and
| riverine or coastal flooding.
|
| I'd say that'd get you close to five 9s (no more than ~5
| minutes downtime per year), though I'd pretty much guarantee
| five 9s (maybe even six 9s - no more than 32 seconds downtime
| per year) if the two machines were physically separated from
| each other by a few hundred kilometres, each with their own
| supporting infrastructure above, sans the load balancing (see
| below), through two separate network routes.
|
| Load balancing would become human-driven in this 'physically
| separate' example (cheaper, less complex): if your-site-1.com
| fails, simply re-point your browser to your-site-2.com which
| routes to the other redundant server on a different network.
|
| The hard part now will be picking network providers that don't
| use the same pipes/cables, i.e. they both use Cloudflare, or
| AWS...
|
| Keep the WAN IPs written down in case DNS fails.
|
| PostgreSQL can do master-master replication, but it's a pain to
| set up I understand.
| vivzkestrel wrote:
| what if you could create a super virtual server of sorts.
| imagine a new cloud provider like vercel but called something
| else. what this provider does is when you create a server on
| their service, they create 3 services, one on aws, one on gcp
| and one on azure. behind the scenes they are 3 separate servers
| but to the end user they are a single server. the end user gets
| to control how many cloud providers are involved. when aws goes
| down, no worries, it switches to the part with gcp on
| endoacustica wrote:
| Same problem here in Italy. Website up and down again.
| Mattia2001 wrote:
| Rome, Palermo, Milan, Catania Down
| skor wrote:
| why do I always get "Server Error" and not an explanation that
| Cloudflare is having problems? This makes me look bad in front of
| my customers.
| Bro1309 wrote:
| Is it down again??
| sergiotapia wrote:
| X is down, and many many other sites. This is not the web I grew
| up on. Do not centralize people.
| bezier-curve wrote:
| I'm really surprised by the sheer scale of how many websites this
| outage is affecting. We really need to decentralize all of these
| monolith clouds.
| I_am_tiberius wrote:
| scaleway.com is down as well. I'm really wondering how a CSP can
| use cloudflare...
| abdulwahidgul wrote:
| Singapore is down as well in Asia
| Mattia2001 wrote:
| Probably the IDF trying a mass network attack to go and Occupy
| the Holy See
| j0nimost wrote:
| africa down as well
| danaos wrote:
| dl.acm.org down, arxiv.org has my back!
| dvcoolarun wrote:
| There was an article on HN a few days back about how companies
| like this are influencing the overall freedom of the web (I
| missed the source) and their own way of doing things. Other
| examples of influence I see similarly are of Vercel, like with
| enterprise. Even a few days back, we saw AWS.
| spacecadet wrote:
| Testing the fences... clever girl...
| Aeolun wrote:
| Why the hell is my claude saying "Please unblock
| challenges.cloudflare.com to proceed."
|
| And then still failing anyway? Why do I need CloudFlare to access
| claude.io? Wtf?
| philipwhiuk wrote:
| AI companies don't want AI to use AI.
| burnt-resistor wrote:
| Dallas CF is down, so basically every app and website is down.
| rgilton wrote:
| What would the Internet's architecture have to look like for
| DDOS'ing to be a thing of the past, and therefore Cloudflare to
| not be needed?
|
| I know there are solutions like IPFS out there for doing
| distributed/decentralised static content distribution, but that
| seems like only part of the problem. There are obviously more
| types of operation that occur via the network -- e.g.
| transactions with single remote pieces of equipment etc, which by
| their nature cannot be decentralised.
|
| Anyone know of research out there into changing the way that
| packet-routing/switching works so that 'DDOS' just isn't a thing?
| Of course I appreciate there are a lot of things to get right in
| that!
| smt88 wrote:
| It's impossible to stop DDoS attacks because of the first "D".
|
| If a botnet gets access through 500k IP addresses belonging to
| home users around the world, there's no way you could have
| prepared yourself ahead of time.
|
| The only real solution is to drastically increase regulation
| around security updates for consumer hardware.
| rgilton wrote:
| Maybe that's the case, but it seems like this conclusion is
| based on the current architecture of the internet. Maybe
| there are ways of changing it that mean these issues are not
| a thing!
| smt88 wrote:
| It's not an architectural problem. It's a fundamental issue
| with trust and distributed systems. The same issues occur
| in physical spaces, like highways.
|
| The core issue is that hackers can steal the "identity" of
| internet customers at scale, not that the internet allows
| unauthenticated traffic.
| venusenvy47 wrote:
| Do the IP addresses botnet members get logged? Could those IP
| addresses be automatically blocked by DNS until they fix
| their machine?
| Maxion wrote:
| Lets say your samsung fridge gets hacked and is now a
| member of a botnet. How do you detect that before the
| botnet does something?
| Woodi wrote:
| Why fridge need to have rights to initiate connection to
| something on internet ?
|
| Why fridge need to even be reachable from the internet ??
| You should have some AI agent for managing your "smart"
| home. At least it's how sci-fi movies/games show it, eg.
| Iron man or Starcraft II ;)
| cuu508 wrote:
| > Why fridge need to have rights to initiate connection
| to something on internet ?
|
| So you can access it from a phone app even when outside
| your home network.
| venusenvy47 wrote:
| I was thinking of a reaction to a DDOS event, so those
| devices are flagged as being infected. You could prevent
| future attacks if those devices are ignored until they
| get fixed.
| smt88 wrote:
| IP addresses aren't unique or stable. You can't use them to
| identify individual devices.
| sentrysapper wrote:
| What would that look like? A network with built-in rate &
| connection limiting?
|
| The closest thing I can think of is the Gemini protocol
| browser. It uses TOFU for authentication, which requires a
| human to initially validate every interaction.
| tempfile wrote:
| Built it into the protocol that you must provide bandwidth in
| order to have your requests served. A bit like forcing people
| to seed torrents.
| rgilton wrote:
| Works for static content and databases, but I don't think it
| works for applications where there is by necessity only one
| destination that can't be replicated (e.g. a door lock).
| pezgrande wrote:
| Something like a mega-transnational-parent ISP authority and
| give tech giants LaLiga kind of power.
| t1234s wrote:
| These big cloud providers are turning into giant off-switches for
| the internet
| greenie_beans wrote:
| thought i'd been hacked, was ready to throw in the towel for this
| career and my saas
| davneet4u wrote:
| It's outrageous that it hasn't been fixed since 2 hours
| CuriousRose wrote:
| Cloudflare runs a high demand service, and the centralisation
| does deserve scrutiny. I think a good middle ground I'll adopt is
| self hosting critical services and then when they have an outage
| redirect traffic to a Cloudflare outage banner.
| monstertank wrote:
| Gemini is up, I asked it to explain what's going on in cave man
| speak:
|
| YOU: Ask cave-chief for fire.
|
| CAVE-CHIEF (Cloudflare): Big strong rock wall around many other
| cave fires (other websites). Good, fast wall!
|
| MANY CAVE-PEOPLE: Shout at rock wall to get fire.
|
| ROCK WALL: Suddenly... CRACK! Wall forgets which cave has which
| fire! Too many shouts!
|
| RESULT: Your Shout: Rock wall does not hear
| you, or sends you to wrong cave. Other Caves (like
| X, big games): Fire is there, but wall is broken. Cannot get to
| fire. ME (Gemini): My cave has my own wall! Not rock
| wall chief! So my fire is still burning! Good!
|
| BIG PROBLEM: Big strong wall broke. Nobody gets fire fast. Wall
| chief must fix strong rock fast!
| rzerowan wrote:
| One of the things that i didnt like about cloudflare MITM as a
| service is their requirement if you want SSL/CDN that you must
| use their DNS. Overconcentration of infra within one single pint
| of disruption with no easy outs when the stack tips over. Sadly i
| dont see any changes or rethink to be more decentralised even
| after this outage.
| dust-jacket wrote:
| to be clear, that's just a limitation on their free service. If
| you pay, you can keep your own DNS
| rzerowan wrote:
| Yeah they keep re-inforcing bad vendor lockin practices. id
| guess the number of free users surpass the paying ones , and
| situations like these leave them all unable to recover.
| jonathanlydall wrote:
| Their paid "professional" plan also has this limitation, only
| "enterprise" and up does not.
| dust-jacket wrote:
| ah, didn't realise that, thanks
| realityking wrote:
| The $199/month/zone Business plan already offers this
| feature.
| andy_ppp wrote:
| I honestly think people should practice more chaos engineering
| themselves and switch off services at random like Cloudflare and
| have failure plans.
| thinkindie wrote:
| For some reason linear.app is working but according to their
| headers they should be behind Cloudflare.
| sambaumann wrote:
| According to the status page services are being restored
| thinkindie wrote:
| I still find a lot of websites/applications (including my
| own) affected.
| Aeolun wrote:
| Hey, this is fun, all my websites are still up! I wonder how that
| happened? I don't even have to worry about my docker registry
| being down because I set up my own after the last global outage.
| pqdbr wrote:
| I had a lot of fun like you as well, until I got my first DDoS
| and bot attacks. There's a reason Cloudflare has 20% of
| internet traffic.
| chanux wrote:
| Does it cost you a lot?
|
| One of my other worries is having fight bots over a couple
| hobby sites while I have other fires to put out (generally in
| life).
| stupidityonmaxx wrote:
| and when i tell people they dont need cf for most of their sites
| they laugh at me. look whos laughing now. theyre down, im up
| Mattia2001 wrote:
| ahahahhaahah
| jonathanlydall wrote:
| Whenever I try collapse root threads on this page it locks up the
| browser for 5+ seconds.
|
| Windows 11, latest Edge browser, 64GB of RAM, 13th Gen i7.
| igleria wrote:
| Does not happen to me under ubuntu, FWIW
| mghackerlady wrote:
| >Windows 11, latest Edge browser
|
| There's your issue
| Telemakhos wrote:
| AWS, Azure, now Cloudflare, all within a month, are hit with
| configuration errors that are definitely neither signs of more
| surveillance gear being added by government agencies nor attacks
| by hostile powers. It's a shame that these fine services that
| everyone apparently needs and that worked so well for so long
| without a problem suddenly all have problems at the same time.
| strict9 wrote:
| Most or all of these lost significant institutional knowledge
| through layoff after layoff and jobs moved to lower cost
| countries.
|
| Maybe a coincidence or maybe not.
| lc64 wrote:
| AWS was not a configuration error, it was a race condition on
| their load balancer's automated DNS record attribution that
| caused empty DNS records. As that issue was being fixed, it
| cascaded into further, more complex issues overloading EC2
| instance provisioning.
| Iulioh wrote:
| I don't get how you ran to that conclusion...
| ElectricSpoon wrote:
| Those football playoffs are really getting out of hand...
|
| Ref: https://news.ycombinator.com/item?id=43157000
| TheJuli wrote:
| Good one!
| mulhoon wrote:
| I know this is bad, and some people's livelihood and lives rely
| on critical infrastructure, but when these things happen, I
| sometimes think GOOD!, let's all just take a breather for a
| minute yeh? Go outside.
| pogue wrote:
| Why are we seeing AWS, then Azure, then Cloudflare all going down
| just out of the blue? I know they go down occasionally, but it's
| typically not major outages like this...
| Diddylovesparty wrote:
| Its ironic that downdetector is down as well..
| Mattia2001 wrote:
| downdetector is up in italy
| m00dy wrote:
| I think Companies are firing wrong people that we get these
| downtimes so often.
| BeFlatXIII wrote:
| Time to check Hacker News instead of work. Even my usual
| procrastination websites are down due to this.
| yanhangyhy wrote:
| first time know a porn website i use is backed by cloudflare.
| abelanger wrote:
| If anyone needs commands for turning off the CF proxy for their
| domains and happens to have a Cloudflare API token.
|
| First you can grab the zone ID via: curl -X GET
| "https://api.cloudflare.com/client/v4/zones" -H "Authorization:
| Bearer $API_TOKEN" -H "Content-Type: application/json" | jq -r
| '.result[] | "\(.id) \(.name)"'
|
| And a list of DNS records using: curl -X GET
| "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records"
| -H "Authorization: Bearer $API_TOKEN" -H "Content-Type:
| application/json"
|
| Each DNS record will have an ID associated. Finally patch the
| relevant records: curl -X PATCH "https://api.cl
| oudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" -H
| "Authorization: Bearer $API_TOKEN" -H "Content-Type:
| application/json" --data '{"proxied":false}'
|
| Copying from a sibling comment - some warnings:
|
| - SSL/TLS: You will likely lose your Cloudflare-provided SSL
| certificate. Your site will only work if your origin server has
| its own valid certificate.
|
| - Security & Performance: You will lose the performance benefits
| (caching, minification, global edge network) and security
| protections (DDoS mitigation, WAF) that Cloudflare provides.
|
| - This will also reveal your backend internal IP addresses.
| Anyone can find permanent logs of public IP addresses used by
| even obscure domain names, so potential adversaries don't
| necessarily have to be paying attention at the exact right time
| to find it.
| mig4ng wrote:
| Awesome! I did it via the Terraform provider, but for anyone
| else without access to the dashboard this is great. Thank you!
| sam-cop-vimes wrote:
| This is exactly what we've decided we should do next time.
| Unfortunately we didn't generate an API token so we are sitting
| twiddling our thumbs.
|
| Edit: seems like we are back online!
| napsterbr wrote:
| Took me ~30 minutes but eventually I was able to log in, get
| past the 2FA screen and change a DNS record.
|
| I surely missed a valid API token today.
| firecall wrote:
| I'm still trying.
|
| Still can't load the Turnstile JS :-/
| biinjo wrote:
| Turnstile is back up (for now). Go refresh. I just
| managed to make an API key and turn off proxied DNS.
| fragmede wrote:
| install tweak chrome extension and mitm yourself and
| force the js to load from somewhere else
| basch wrote:
| Im able to generate keys right now through warp. Login takes
| forever but it is working.
| duggan wrote:
| Also, for anyone who only has an old global API key lying
| around instead of the more recent tokens, you can set:
| -H "X-Auth-Email: $EMAIL_ADDRESS" -H "X-Auth-Key: $API_KEY"
|
| instead of the Bearer token header.
|
| Edit: and in case you're like me and thought it would be clever
| to block all non-Cloudflare traffic hitting your origin...
| remember to disable that.
| basch wrote:
| If anyone needs the internet to work again (or to get into your
| cf dashboard to generate API keys), if you have Cloudflare WARP
| installed, turning it on appears to fix otherwise broken sites.
| Maybe using 1.1.1.1 does too, but flipping the radio box was
| faster. Some parts of sites are still down, even after
| tunneling into to CF.
| adi_kurian wrote:
| super helpful. thanks!
|
| looks like i can get everywhere i couldn't except my
| cloudflare dash.
| basch wrote:
| Its absurdly slow (like multiple minutes to get the login
| page to fully load for the login button to be pressable,
| due to catchpa...), but I was able to log into the
| dashboard. It's throwing lots of errors once inside, but I
| can navigate around some of it. YMMV.
|
| My profile (including api tokens,) and websites pages all
| work, the accounts tab above website on the left does not.
| jlundberg wrote:
| Good advice!
|
| And no need for -X GET to make a GET request with curl, it is
| the default HTTP method if you don't send any content.
|
| If you do send content with say -d curl will do a POST request,
| so no need for -X then either.
|
| For PATCH though, it is the right curl option.
| JoeOfTexas wrote:
| I would advise against this action. Just ride the crash.
| RKFADU_UOFCCLEL wrote:
| If people knew how to play the 5 hour long game they wouldn't
| have been using Cloudflare in the first place.
| digdugdirk wrote:
| If someone wanted to learn about how the modern infrastructure
| stack works, and why things like this occur, where would be some
| good resources to start?
| _1tan wrote:
| Incident post mortems: https://github.com/danluu/post-mortems
| lilOnion wrote:
| I just wanted to ask the same thing. I have a really basic idea
| of how everything is connected but would love to jump in more
| in depth.
|
| How can such big incidents occur where half of the internet is
| down because of one company and what can be done to prevent
| that?
| pona-a wrote:
| Our national transit agency is apparently a customer.
|
| The departure tables are borked, showing incorrect data, the
| route map stopped updating, the website and route planner are
| down, and the API returns garbage. Despite everything, the
| management will be pleased to know the ads kept on running
| offline.
|
| Why would you put a WAP between devices you control and your own
| infra, God knows.
| cube00 wrote:
| > Why would you put a WAP between devices you control and your
| own infra
|
| Checkbox security says a WAP is required and no CISO will put
| their neck on the line to approve the exemption.
| wslh wrote:
| Welcome to AOL.
| thiago_fm wrote:
| ChatGPT isn't working.
|
| No suicides created by ChatGPT Today. Billions of dollars in GPU
| will sit idle. Sudden drop of Linkedin content...
|
| World is a better place
| djfobbz wrote:
| I'd rather mitigate a DDoS attack on my own servers than deal
| with Cloudflare. Having to prove you're human is the second-worst
| thing on my list, right after accepting cookies. Those two things
| alone have made browsing the web a worse experience than it was
| in the late 90s or early 2000s.
| reassess_blind wrote:
| How do you plan on mitigating a DDoS on your own servers?
| solusipse wrote:
| he'll politely ask them to stop
| schnebbau wrote:
| You wait for it to stop.
| FinnKuhn wrote:
| That is not what mitigating means.
| ramon156 wrote:
| You turn off the screen. They can't hurt you if you don't see
| them
| djfobbz wrote:
| Alright kids, breathe...a DDoS attack isn't the end of the
| world, it's just the internet throwing a tantrum. If you
| really don't want to use a fancy protection provider, you can
| still act like a grown-up: get your datacenter to filter
| trash at the edge, announce a more specific prefix with BGP
| so you can shift traffic, drop junk with strict ACLs, and
| turn on basic rate limiting so bots get bored. You can also
| tune your kernel so it doesn't faint at SYN storms, and if
| the firehose gets too big, pop out a more specific BGP prefix
| from a backup path or secondary router so you can pull
| production away from the burning IP.
| arsome wrote:
| Very quickly you'll find this doesn't work. Your DC will
| just null your IP. You'll switch to a new one and the
| attackers will too, the DC will null that one. You won't
| win at this game unless you're a very sizeable organization
| or are just willing to wait the attackers out, they will
| get bored eventually.
| sedawkgrep wrote:
| > pop out a more specific BGP prefix from a backup path or
| secondary router so you can pull production away from the
| burning IP.
|
| This won't help against carpet bombing.
|
| The only workable solution for enterprises is a combination
| of on-prem and cloud mitigation. Cloud to get all the big
| swaths of mitigation and to keep your pipe flowing, and on-
| prem to mitigate specific attack vectors like state
| exhaustion.
| djfobbz wrote:
| I never claimed to be an enterprise, nor did I suggest
| this was the best option for them. Cheers!
| djfobbz wrote:
| Worrying about a DDoS on your tiny setup is like a brand-new
| dev stressing over how they'll handle a billion requests per
| second...cute, but not exactly a real-world problem for
| 99.99% of you. It's one of those internet boogeyman myths
| people love to panic about.
| marticode wrote:
| There's worse than having to prove (over and over and over
| again) that you are human: having your IP just completely
| blocked by Cloudflare zealous bot-filtering (and I use a plain
| mass market ISP in a developed country and not some shady
| network)
| philipwhiuk wrote:
| Some of the mass-market ISPs are very shady - AT&T's Room
| 641A for example :)
| codegeek wrote:
| As much as this situation sucks, how do you plan to "mitigate a
| DDoS attack on my own servers". The reason I use Cloudflare is
| to use it as a proxy especially for DDOS attacks if they do
| occur. Right now, our services are down and we are getting tons
| of customer support tickets (like everyone else) but it is a
| lot easier to explain the the whole world is down vs its just
| us.
| djeastm wrote:
| >it is a lot easier to explain the the whole world is down vs
| its just us.
|
| Makes sense. The ability to pass the buck like this is 95% of
| the reason Cloudflare exists in the first place. Not being
| snarky, either.
| redbell wrote:
| This _Internet_ thing is steadily becoming the most fragile
| surface attack out there. No need for nuclear weapons anymore,
| just hit Cloudflare and AWS and we are back to the stone age.
| IAmGraydon wrote:
| I host everything on Linode (have for over a decade) and am never
| caught up in these outages.
| nicolaslem wrote:
| Linode has been rock solid for me. I wanted to back this
| comment with uptime numbers, unfortunately the service I use
| for that, Uptime Robot, is down because of Cloudflare...
| mef wrote:
| aisuru?
| medusae wrote:
| still down in Australia
| porker wrote:
| Is there any way to remove every SPOF?
|
| Currently I have multi-region loadbalanced servers. DNS and WAF
| (and the load balancer) on Cloudflare.
|
| Moving DNS elsewhere is step 1 so I'm not locked out - but then I
| can't use Cloudflare full stop (without enterprise pricing).
|
| Multi-provider DNS and WAF - okay I could see how that works.
|
| But what about the global load balancer, surely that has to
| remain a single point of failure?
| dangoodmanUT wrote:
| No? The point of cloudflare is that they remove the spof for
| you, but I guess we can say they didn't do it quite perfectly
| easternherald wrote:
| The Eastern Herald news website is down. Easternherald.com
| zeroq wrote:
| the day the earth stood still
| 1970-01-01 wrote:
| Looking forward to seeing their RCA. I'm guessing it's going to
| be glossy in terms of actual customer impact. "We didn't go
| offline, we just had 100% errors. For 60 minutes."
| thekid314 wrote:
| This NYTimes article makes it sound like the problem is fixed,
| but I'm not seeing any improvement yet.
|
| https://www.nytimes.com/2025/11/18/business/cloudflare-down-...
| philipwhiuk wrote:
| Latest is:
|
| Update - The team is continuing to focus on restoring service
| post-fix. We are mitigating several issues that remain post-
| deployment. Nov 18, 2025 - 15:40 UTC
| nish__ wrote:
| Cloudflare has always been a shit company.
| DeathArrow wrote:
| My theory is that people's skills are getting worse. Attention
| spans are diminishing, memory is shrinking. People age and
| retire, new less skilled generations are replacing them. There
| are studies about declining IQ in the last decades. Probably
| mobile phones and social media are to blame.
|
| We see the signs with Amazon and Cloudflare going down, Windows
| Update breaking stuff. But the worse is yet to come, and I am
| thinking about airport traffic control, nuclear power plants,
| surgeons...
| andsoitis wrote:
| > There are studies about declining IQ in the last decades.
| Probably mobile phones and social media are to blame.
|
| It is much more nuanced than that.
|
| The long-term rise (Flynn Effect) of IQs in the 20th century is
| widely believed to be driven by environmental factors more than
| genetics.
|
| Plateau / decline is context-dependent: The reversal or
| slowdown isn't universal, like you suggest. It seems more
| pronounced in certain countries or cohorts.
|
| Cognitive abilities are diversifying: As people specialize more
| (education, careers, lifestyles), the structure of intelligence
| (how different cognitive skills relate) might be changing.
| Dquiroga wrote:
| Since when does critical infrastructure fail weekly?! One week is
| AWS, then azure + AWS, now cloudfare...
|
| Time to go back to on prem. AWS and co are too expensive anyways
| brandon272 wrote:
| A lot of people are "on prem" but use CloudFlare to proxy
| traffic for DDoS attack mitigation, among other reasons.
| speckx wrote:
| I wish the "pause CF" button would work via API or via any other
| way, even if there is an outage like this.
| cromka wrote:
| Wow, with outage of a scale like this, it must be measurable as a
| loss in global GDP
| senfiaj wrote:
| This is worse than than the Amazon outage. I couldn't even login
| to Cloudflare.
| fpauser wrote:
| SPOF
| tovej wrote:
| Is anybody keeping statistics on the frequency of these big
| global internet outages? It seems to be happening extremely
| frequently as of late, but it would be nice to have some data on
| that.
| amukbils wrote:
| OpenAI & chatGPT also down from this
| an-allen wrote:
| What?! No way! Keen to see the post mortem on this. Its always
| DNS.
| yusufmalikul wrote:
| I just comment here to be part of the history.
| kevinkatzke wrote:
| Is it me or do these outages happen pretty often lately?
| dnw wrote:
| Our doctor's office can't make appointments because their "system
| is down."
| Jamie452 wrote:
| Give me back ChatGPT pls
| kordlessagain wrote:
| Man in the middle!
| alexpadula wrote:
| Oh boy
| elcapithanos wrote:
| I'm genuinely curious how much of the web depends on cloudflare
| and AWS. This centralisation sucks though
| erichocean wrote:
| Cloudflare captchas don't work, which has taken down both Claude
| and Perplexity for me.
|
| Lovely.
| theoldgreybeard wrote:
| I miss the old internet where 1 company having an outage didn't
| take down most of it.
| dSebastien wrote:
| Phew, my latest 3h30 workshop about Obsidian was saved. I
| recorded it this morning, not knowing about the Cloudflare issue
| (probably started while I was busy). I'm using Circle.so and
| they're down (my community site is now inaccessible). Luckily,
| they probably use AWS S3 or similar to host their files, so that
| part is still up and running.
|
| Meanwhile all my sites are down. I'll just wait this one out,
| it's not the end of the world for me.
|
| My GitHub actions are also down for one of my project because
| some third-party deps go through Cloudflare (Vulkan SDK). Just
| yesterday I was thinking to myself: "I don't like this dependency
| on that URL...". Now I like it even less
| lucky-Scarlett wrote:
| I've been waiting for hours, it looks like I can finally take a
| day off today.
| TimCTRL wrote:
| at least with Cloudflare, we may have the postmortem report
| tommorow
| haritha-j wrote:
| makes you realise, if cloudflare or one of these large
| organisations decides to (/ gets ordered by a deranged US
| president to) block your internet access, that's a whole lot of
| internet you're suddenly cut off from. Yes, i know there are
| circumventions, but its still a owrrying thought.
| fatchan wrote:
| Once again vindicated by running my own CDN and not living with
| the irrational belief that everything needs cloudflare.
| cheerioty wrote:
| I swear they all sleeping and that one guy on call is like
| "sheeeeet"...
| ghoshbishakh wrote:
| For cf tunnel alternatives are available, I maintain pinggy.io
| brubsby wrote:
| I can't rebuild my NixOS image because of this lol. (chrome
| install not working)
| herbst wrote:
| I am paying for this shit service and this is my longest downtime
| I had in years. Can anyone recommend any other bottleneck to be
| annoyed with in future?
| r721 wrote:
| Quote from The Guardian's story:
|
| >A spokesperson for Cloudflare said: "We saw a spike in unusual
| traffic to one of Cloudflare's services beginning at 11.20am.
| That caused some traffic passing through Cloudflare's network to
| experience errors. While most traffic for most services continued
| to flow as normal, there were elevated errors across multiple
| Cloudflare services.
|
| >"We do not yet know the cause of the spike in unusual traffic.
| We are all hands on deck to make sure all traffic is served
| without errors. After that, we will turn our attention to
| investigating the cause of the unusual spike in traffic."
|
| https://www.theguardian.com/technology/2025/nov/18/cloudflar...
| Jamie452 wrote:
| Sounds like it may have been a cyber attack...
| lm28469 wrote:
| That's not what I'm hearing from insiders
| delfinom wrote:
| "Unusual spike of traffic" can just be errant
| misconfiguration that causes traffic spikes just from TCP
| retries or the like. Jumping to "cyber attack" is eating up
| Hollywood drama.
|
| In most cases, it's just cloud services eating shit from a
| bug.
| catholiccs wrote:
| PEBKAC error
| mrP1p1 wrote:
| Gpt and perplexity still down
| mrP1p1 wrote:
| Germany
| endoacustica wrote:
| Right now is working in Italy: https://www.endoacustica.com/ I
| hope will be not down again.
| eruci wrote:
| This is ridiculous:
|
| They just posted:
|
| Update We've deployed a change which has restored dashboard
| services. We are still working to remediate broad application
| services impact Posted 2 minutes ago. Nov 18, 2025 - 14:34 UTC
|
| but,.. I'm stuck at the captcha that does not work:
| dash.cloudflare.com Verifying you are human. This may take a few
| seconds.
|
| dash.cloudflare.com needs to review the security of your
| connection before proceeding.
| dzink wrote:
| It's back.
| HocusLocus wrote:
| That must have been a really big backhoe...
| shridharxp wrote:
| It's interesting to see hacker news response time reaching almost
| 2 seconds for this post.
| robotfelix wrote:
| > A fix has been implemented and we believe the incident is now
| resolved. We are continuing to monitor for errors to ensure all
| services are back to normal. Posted 3 minutes ago. Nov 18, 2025 -
| 14:42 UTC
|
| Seems like they think they've fixed it fully this time!
| gpi wrote:
| Close! They just updated their states and it's back to working
| on a fix
|
| Update - Some customers may be still experiencing issues
| logging into or using the Cloudflare dashboard. We are working
| on a fix to resolve this, and continuing to monitor for any
| further issues. Nov 18, 2025 - 14:57 UTC
| psim1 wrote:
| Later today or tomorrow there's going to be a post on HN pointing
| to Cloudflare's RCA and multitudes here are going to praise CF
| for their transparency. Let's not forget that CF sucks and took
| half the internet down for four hours. Transparency or no, this
| should not be happening.
| rikafurude21 wrote:
| Alot of things shouldnt be happening. Fact is that no one
| forced half the internet to make CF their point of failure. The
| internet should ask themselves if that was the right call
| fasteo wrote:
| The whole damn internet now depends on them. I guess I am bullish
| for $NET
| hoppp wrote:
| Cloudflare is a central point of failure.
| NooneAtAll3 wrote:
| monopoly causes problems for all customers?
|
| oh no, anyway
| renegat0x0 wrote:
| This reminds me that I really like self-hosting. While it is true
| that many of things do not work, all my services do work. It has
| some tradeoffs of course.
| telepromptereye wrote:
| I didn't see anyone comment this directly, but something these
| recent outages made me wonder, having spent a good chunk of my
| career in 24/7 tech support, is that I can't even fathom the
| amount of people who have been:
|
| - restarting their routers and computers instead of taking their
| morning shower, getting their morning coffee, taking their
| medication on time because they're freaking out, etc. - calling
| ISPs in a furious mood not knowing it's a service in the stack
| and not the provider's fault (maybe) - being late for work in
| general - getting into arguments with friends and family and
| coworkers about politics and economics - being interrupted making
| their jerk chicken
| koakuma-chan wrote:
| My static website hosted on cloudflare works :/
| Oopsin wrote:
| should we think about Akamai
| gadsnprch wrote:
| This seems to corroborate the recent controversial claims that
| American workers do not possess the aptitudes needed to succeed
| in the 21st century. If only we could have gotten more children
| to learn to code. Sigh.
| DaSHacka wrote:
| Very true, clearly all these issues that started cropping up
| after fortune 500 companies started offshoring indicates they
| didn't offshore fast enough
| gadsnprch wrote:
| Accelerate the eschaton.
| dpacmittal wrote:
| Can't wait to read their post-mortem report
| Stasshe wrote:
| in Japan, now, it's alive.
|
| how about your location?
| delduca wrote:
| Pretty sure they used Vibe Coding...
| acc_297 wrote:
| If a cloud vendor with 1 million users experiences a long term
| outage: the vendor has a serious problem. If a cloud vendor with
| 1 billion users experiences a long term outage: the internet has
| a serious problem. Yada-yada-yada xkcd/2347 but it's the big
| block in the middle which crumbled
| octoberfranklin wrote:
| What part of "single point of failure" do people not understand?
|
| Stop. Using. Cloudflare.
| aavshr wrote:
| > In short, a latent bug in a service underpinning our bot
| mitigation capability started to crash after a routine
| configuration change we made. That cascaded into a broad
| degradation to our network and other services. This was not an
| attack.
|
| From the CTO, Source:
| https://x.com/dok2001/status/1990791419653484646
| imdsm wrote:
| Configuration changes are dangerous for CF it seems, and
| knocked down $NET almost 4% today. I wonder what the industry
| wide impact is for each of these outages?
| sammy2255 wrote:
| Pre market was red for all tech stocks today before the
| outage even happened
| __turbobrew__ wrote:
| It still astounds me that the big dogs still do not phase
| config rollouts. Code is data, configs are data, they are one
| and the same. It was the same issue with the giant crowdstrike
| outage last year, they were rawdogging configs globally and a
| bad config made it out there and everything went kaboom.
|
| You NEED to phase config rollouts like you phase code rollouts.
| JohnMakin wrote:
| In a company I am no longer with I argued much the same when
| we rolled out "global CI/CD" on IAC. You made one change,
| committed and pushed, wham it's on 40+ server clusters
| globally. I hated it. The principal was enamored with it,
| "cattle not pets" and all that, but the result was things
| slowed down considerably because anyone working with it
| became so terrified of making big changes.
| wbl wrote:
| Then you get customer visible delays.
| siegecraft wrote:
| I think it's uncharitable to jump to the conclusion that just
| because there was a config-based outage they don't do phased
| config rollouts. And even more uncharitable to compare them
| to crowdstrike.
| cyberpunk wrote:
| It seem fairly logical to me? If a config change causes
| services to crash then rollout stops ... at least in every
| phased rollout system i've ever built...
| __turbobrew__ wrote:
| I have read several cloudflare postmortems and my
| confidence in their systems is pretty low. They used to run
| their entire control plane out of a single datacenter which
| is amateur hour for a tech company that has over $60
| billion in market cap.
|
| I also don't understand how it is uncharitable to compare
| them to crowdstrike as both companies run critical systems
| that affect a large number of people's lives, and both
| companies seem to have outages at a similar rate (if
| anything, cloudflare breaks more often than crowdstrike).
| crazygringo wrote:
| The big dogs absolutely do phase config rollouts as a general
| rule.
|
| There are still two weaknesses:
|
| 1) Some configs are inherently global and cannot be phased.
| There's only one place to set them. E.g. if you run a webapp,
| this would be configs for the load balancer as opposed to
| configs for each webserver
|
| 2) Some configs have a cascading effect -- even though a
| config is applied to 1% of servers, it affects the other
| servers they interact with, and a bad thing spreads across
| the entire network
| creatonez wrote:
| > Some configs are inherently global and cannot be phased
|
| This is also why "it is always DNS". It's not that DNS
| itself is particularly unreliable, but rather that it is
| the one area where you can really screw up a whole system
| by running a single command, even if everything else is
| insanely redundant.
| __turbobrew__ wrote:
| I don't believe that there is anything necessarily which
| requires DNS configs to be global.
|
| You can shard your service behind multiple names:
|
| my-service-1.example.com
|
| my-service-2.example.com
|
| my-service-3.example.com ...
|
| Then you can create smoke tests which hit each phase of
| the DNS and if you start getting errors you stop the
| rollout of the service.
| creatonez wrote:
| Sure, but that doesn't really help for user-facing
| services where people expect to either type a domain name
| in their browser or click on a search result, and end up
| on your website every time.
|
| And the access controls of DNS services are often (but
| not always) not fine-grained enough to actually prevent
| someone from ignoring the procedure and changing every
| single subdomain at once.
| __turbobrew__ wrote:
| > Sure, but that doesn't really help for user-facing
| services where people expect to either type a domain name
| in their browser or click on a search result, and end up
| on your website every time.
|
| It does help. For example, at my company we have two
| public endpoints:
|
| company-staging.com company.com
|
| We roll out changes to company-staging.com first and have
| smoke tests which hit that endpoint. If the smoketests
| fail we stop the rollout to company.com.
|
| Users hit company.com
| cowsandmilk wrote:
| That doesn't help with rolling out updates to the DNS for
| company.com which is the point here. It's always DNS
| because your pre-production smoke tests can't test your
| production DNS configuration.
| immibis wrote:
| Because adversaries adapt quickly, they have a system that
| deploys their counter-adversary bits quickly without phasing
| - no matter whether they call them code or configs. See also:
| Crowdstrike.
| himinlomax wrote:
| You can't protect against _latent bugs_ with phased rollouts.
| JohnMakin wrote:
| Wish this could rocket to the top of the comment thread,
| digging through hundreds of comments speculating about a
| cyberattack to find this felt silly
| gk1 wrote:
| Aaaaand it's down again.
| fasteo wrote:
| > a routine configuration change we made
|
| Every.Single.Time
|
| [1] https://x.com/dok2001/status/1990791419653484646
| sidcool wrote:
| The status is still Red on their dashboard.
| tmaly wrote:
| What are the odds this is a human configuration error related to
| DNS?
| abhisek wrote:
| This is crazy. The internet has so much direct and transitive
| dependency on Cloudflare today. Pretty much the #1 dev slacking
| excuse today is no longer code compiling but cloudflare is down.
| asimpletune wrote:
| Email workers of all things seem to have slowed down
| dramatically, although they're not down completely.
| eth0up wrote:
| I happened to be working with Claude when this occurred. Having
| no idea what exactly what the cause was, I jumped over to GPT and
| observed the same. I did a dig challenges.cloudflare.com and by
| the time I'd figured out _kind of_ what was happening, it seemed
| to have... resolved itself
|
| I must say I'm astonished, as naive as it may be, to see the
| number of separate platforms affected by this. And it has been a
| bit of a learning experience too.
| ramon156 wrote:
| is anyone's hackernews lagging when loading the comments? ive
| seen posts with 2k comments before but for some reason this took
| longer
| catholiccs wrote:
| PEBKAC error
| bigbuppo wrote:
| Has anyone else noticed a major drop in email spam with this
| cloudflare outage?
| omani wrote:
| one way to mitigate DDoS is to enforce source IP checks on the
| way OUT of a datacenter (egress).
|
| sure there are botnets, infected devices, etc that would conform
| to this but where does the sheer power of a big ddos attack come
| from? including those who sell it as a service. they have to have
| some infrastructure in some datacenter right?
|
| make a law that forces every edge router of a datacenter to check
| for source IP and you would eliminate a very big portion of DDoS
| as we know it.
|
| until then, the only real and effective method of mitigating a
| DDoS attack is with even more bandwidth. you are basically a
| black hole to the attack, which cloudflare basically is.
| vlovich123 wrote:
| The biggest attacks literally come from botnets. There's not a
| lot coming from infrastructure services precisely because these
| services are incentivized to shut that shit down. At most it
| would be used as the control plane which is how people attempt
| to shut down the botnets.
| kazen44 wrote:
| alright, what you are proposing is kind of hard to do. Source
| routing is not easy, and source validations is even harder.
|
| and what prevents me, as a abuse hoster or "bad guy" from just
| announcing my own IP space directly on a transit or IXP?
|
| You might say, the IXP should do source checking aswell, but
| what if ipspace is distributed/anycasted across multiple ASN's/
| on the IXP?
|
| Also, if you add multiple egress points distributed across
| different routing domains, it gets complicated fast.
|
| Does my transit upstream need to do source validation of my IP
| space? What about their upstream? Also, how would he know which
| IPspace belongs to which ASN's considering the allocation of
| ASN numbers and IP space is distributed across different
| organisations across the globe. (some of which are more
| malicious/non function than others[0]). Source routing becomes
| extremly complex because there is no single, universal mapping
| between IP space and ASN's they belong too.
|
| [0]https://afrinic.net/notice-for-termination-of-the-
| receiversh...
| tethys wrote:
| This incident has been resolved. Posted 4 minutes ago. Nov 18,
| 2025 - 19:28 UTC
|
| Better link for chroniclers, since the incident is now buried
| pretty far down on the status page:
| https://www.cloudflarestatus.com/incidents/8gmgl950y3h7
| dang wrote:
| Thanks! We've switched to that from
| https://www.cloudflarestatus.com/?t=1 at the top.
| crest wrote:
| Oh the nuclear bomb proof network is unusable because someone
| sneezed over at Cloudflare.
| gorgoiler wrote:
| I would love to see a competition for the most banal thing that
| went wrong as a result of this. For example, I'm pretty sure the
| reason my IKEA locker wouldn't latch shut was because the OS had
| hung while talking to a Cloudflare backend.
___________________________________________________________________
(page generated 2025-11-18 23:00 UTC)