[HN Gopher] The Cloudflare outage might be a good thing
___________________________________________________________________
The Cloudflare outage might be a good thing
Author : radeeyate
Score : 234 points
Date : 2025-11-24 03:04 UTC (19 hours ago)
(HTM) web link (gist.github.com)
(TXT) w3m dump (gist.github.com)
| charcircuit wrote:
| >It's ironic because the internet was actually designed for
| decentralisation, a system that governments could use to
| coordinate their response in the event of nuclear war
|
| This is not true. The internet was never designed to withstand
| nuclear war.
| chasing0entropy wrote:
| Arpanet absolutely was designed to be a physically resilient
| network which could survive the loss of multiple physical
| switch locations.
| anonym29 wrote:
| ARPANET was literally invented during the cold war for the
| specific and explicit purpose of networked communications
| resilience for government and military in the event major
| networking hubs went offline due to one or more successful
| nuclear attacks against the United States
| charcircuit wrote:
| It literally wasn't. It's an urban myth.
|
| >Bob Taylor initiated the ARPANET project in 1966 to enable
| resource sharing between remote computers.
|
| >The ARPANET was not started to create a Command and Control
| System that would survive a nuclear attack, as many now
| claim.
|
| https://en.wikipedia.org/wiki/ARPANET
| anonym29 wrote:
| The stated research goals are not necessarily the same as
| the strategic funding motivations. The DoD clearly
| recognized packet-switching's survivability and dynamic
| routing potential when the US Air Force funded the
| invention of networked packet switching by Paul Baran six
| years earlier, in 1960, for which the explicit purpose
| _was_ "nuclear-survivable military communications".
|
| There is zero reason to believe ARPA would've funded the
| work were it not for internal military recognition of the
| utility of the underlying technology.
|
| To assume that the project lead was told EVERY motivation
| of the top secret military intelligence committee that was
| responsible for 100% of the funding of the project takes
| either a special kind of naivete or complete ignorance of
| compartmentalization practices within military R&D and
| procurement practices.
|
| ARPANET would never have been were it not for ARPA funding,
| and ARPA never would've funded it were it not for the
| existence of packet-switched networking, which itself was
| invented and funded, again, six years before Bob Taylor
| even entered the picture, for the SOLE purpose of "nuclear-
| survivable military communications".
|
| Consider the following sequence of events:
|
| 1. US Air Force desires nuclear-survivable military
| communications, funds Paul Baran's research at RAND
|
| 2. Baran proves packet-switching is conceptually viable for
| nuclear-survivable communications
|
| 3. His specific implementation doesn't meet rigorous Air
| Force deployment standards (their implementation partner,
| AT&T, refuses - which is entirely expectable for what was
| then a complex new technology that not a single AT&T
| engineer understood or had ever interacted with during the
| course of their education), but the concept is now proven
| and documented
|
| 4. ARPA sees the strategic potential of packet-switched
| networks for the explicit and sole purpose of nuclear-
| survivable communications, and decides to fund a more
| robust development effort
|
| 5. They use academic resource-sharing as the
| development/testing environment (lower stakes, work out the
| kinks, get future engineers conceptually familiar with the
| underlying technology paradigms)
|
| 6. Researchers, including Bob Taylor, genuinely focus on
| resource sharing because that's what they're told their
| actual job is, even though that's not actually the true
| purpose of their work
|
| 7. Once mature, the technology gets deployed for it's
| originally-intended strategic purposes (MILNET split-off in
| 1983)
|
| Under this timeline, the sole true reason for ARPA's
| funding of ARPANET is nuclear-survivable military
| communication, Bob Taylor, being the military's R&D pawn,
| is never told that (standard compartmentalization
| practice). Bob Taylor can credibly and honestly state that
| he was tasked with implementing resource sharing across
| academic networks, which is true, but was never the actual
| underlying motivation to fund his research.
|
| ...and the myth of "ARPANET wasn't created for nuclear
| survivability" is born.
| oidar wrote:
| Per interviews, the initial impetus wasn't to withstand a
| nuclear attack - but after it was first set up, it most
| certainly a major part of the thought process in design. ht
| tps://web.archive.org/web/20151104224529/https://www.wired.
| ..
| charcircuit wrote:
| >but after it was first set up
|
| Your link is talking about work Baran did before ARPANET
| was created. The timeline doesn't back your point. And
| when ARPANET was created after Baran's work with Rand:
|
| >Wired: The myth of the Arpanet - which still persists -
| is that it was developed to withstand nuclear strikes.
| That's wrong, isn't it?
|
| >Paul Baran: Yes. Bob Taylor1 had a couple of computer
| terminals speaking to different machines, and his idea
| was to have some way of having a terminal speak to any of
| them and have a network. That's really the origin of the
| Arpanet. The method used to connect things together was
| an open issue for a time.
| oidar wrote:
| Read the whole article. And peruse the oral history here:
| https://ethw.org/Oral-History:Paul_Baran - the genesis
| was most definitely related to the cold war.
|
| "A preferred alternative would be to have the ability to
| withstand a first strike and the capability of returning
| the damage in kind. This reduces the overwhelming
| advantage by a first strike, and allows much tighter
| control over nuclear weapons. This is sometimes called
| Second Strike Capability."
| bblb wrote:
| Perhaps. Perhaps not. But it will survive it. It will survive a
| complete nuclear winter. It's too useful to die, and will be
| one the first things to be fixed after global annihilation.
|
| But Internet is not hosting companies or cloud providers.
| Internet does not care if they don't build their systems
| resilient enough and let the SPOFs creep up. Internet does it's
| thing and the packets keep flowing. Maybe BGP and DNS could use
| some additional armoring but there are ways around both of them
| in case of actual emergency.
| chasing0entropy wrote:
| Spot on article, but without a call to action. What can we do to
| combat the migration of society to a centralized corpro-
| government intertwined entity with no regard for unprofitable
| privacy or individualism?
| DANmode wrote:
| Learn how to host anything, today.
| imsurajkadam wrote:
| Even if you learn to Host, there are many other services that
| are going to get relied on those centralised platforms, so if
| you are thinking to Host, every single thing on your own,
| then it is going to be more work than you can even imagine
| and definitely super hard to organise as well
| DANmode wrote:
| Anything.
| randallsquared wrote:
| Have you tried that? I gave up on hosting my own email server
| seven or eight years ago, after it became clear that there
| would be an endless fight with various entities to accept my
| mail. Hosting a webserver without the expectation that you'll
| need some high powered DDOS defense seems naive, in the
| current day, and good luck doing that with a server or two.
| IgorPartola wrote:
| I have never hosted my own email. It took me roughly a day
| to set it up on a vanilla FreeBSD install running on
| Vultr's free tier plan and it has been running flawlessly
| for nearly a year. I did not use AI at all, just the
| FreeBSD, Postfix, and Dovecot's handbooks. I do have a fair
| bit of Linux admin and development experience but all in
| all this has been a weirdly painless experience.
|
| If you don't love this approach, Mail-in-a-box works
| incredibly well even if the author of all the Python code
| behind it insists on using tabs instead of spaces :)
|
| And you can always grab a really good deal from a small
| hosting company, likely with decades of experience in what
| they do, via LowEndBox/LowEndTalk. The deal would likely
| blow AWS/DO/Vultr/Google Cloud out of the water in terms of
| value. I have been snagging deals from there for ages and I
| lost a virtual host twice. Once was a new company that
| turned out to be shady and another was when I rented a VPS
| in Cairo and a revolution broke out. They brought
| everything back up after a couple of months.
|
| For example I just bought a lifetime email hosting system
| with 250GB of storage, email, video, full office suite,
| calendar, contacts, and file storage for $75. Configuration
| here is down to setting the DNS records they give you and
| adding users. Company behind it has been around for ages
| and is one of the best regarded in the LET community.
| dmoy wrote:
| It's not insurmountable to set up initially. And when you
| get email denied from whatever org (your lawyer, your
| mom, some random business, whatever), each individual one
| isn't insurmountable to fix. It does get old after
| awhile.
|
| It also depends on how much you are emailing, and who. If
| it's always the same set of known entities, you might be
| totally fine with self hosting. Someone else who's
| regularly emailing a lot of new people or businesses
| might incur a lot of overhead. At least worth more than
| their time than a fastmail or protonmail subscription or
| whatever.
| randallsquared wrote:
| I ran my own mail server from 1998 through 2019, and set
| up a FreeBSD mail server as one of my first contract jobs
| in 1998 or 1999. I used Sendmail, Exim, Postfix, and
| qmail at various times. I switched to mail-in-a-box in
| 2014, and contributed a few minor fixes, then (which I'd
| forgotten about until I idly looked to see, just now).
|
| Throughout 20 years of running my own mail server for
| companies, friends, and myself, the additional effort to
| get commercially-run mail servers to accept mail was both
| annoying and random ("oh, look, hosted Outlook has
| started rejecting our mail again..."), and sometimes they
| don't even send a standard response but just "accept" and
| blackhole the email. Eventually you find out that someone
| else in the /24 you're in at Rackspace or DigitalOcean is
| happily running an open relay, and that's why your IP is
| having problems. Or any of a dozen similar things.
|
| In 2019, having gotten very tired of this, I gave up and
| moved my mail handling to Amazon Workmail and SMS, and
| after setting it up properly once, it's been trouble-free
| and maintenance-free for half a decade. Compared to some
| solutions, it's expensive, but not in absolute terms.
| rurban wrote:
| If you host you are running on my cPanel SW. 70% of the
| internet is doing that. Also a kinda centralized point of
| failure, but I didn't hear of any bugs in the last 14 years.
| card_zero wrote:
| We could quibble about the premise.
| adrianN wrote:
| Individuals are unlikely to be able to do something about the
| centralization problem except vote for politicians that want to
| implement countermeasures. I don't know of any politicians
| (with a chance to win anything) that have that on their agenda.
| turtletontine wrote:
| That's called antitrust, and is absolutely a cause you can
| vote for. Some of the Biden administration's biggest
| achievements were in antitrust, and the head of the FTC for
| Biden has joined Mamdani's transition team.
| teiferer wrote:
| There is a crucial step between having an opinion and voting.
| It's conversations within society. That's what makes
| democracy and facilitates change. If you only take your
| opiniom, isolated from everybody else, and vote from that,
| there isn't much democracy going on and your chance for
| change is slim. It's when there is broad conversations
| happening when movements have an impact.
|
| And that step is here on HN. That's why it's very relevant to
| observe that that HN crowd is increasingly happy to support a
| non-free internet. Be it walled gardens, geofencing, etc.
| theideaofcoffee wrote:
| > They [outages] can force redundancy and resilience into
| systems.
|
| They won't until either the monetary pain of outages becomes
| greater than the inefficiency of holding on to more systems to
| support that redundancy, or, government steps in with clear
| regulation forcing their hand. And I'm not sure about the latter.
| So I'm not holding my breath about anything changing. It will
| continue to be a circus of doing everything on a shoestring
| because line must go up every quarter or a shareholder doesn't
| keep their wings.
| morshu9001 wrote:
| That's ok though, not every website needs 5 9s
| krick wrote:
| It would be a good thing, if it would cause anything to change.
| It obviously won't. As if a single person reading this post
| wasn't aware that the Internet is centralized, and couldn't name
| specifically a few sources of centralization (Cloudflare, AWS,
| Gmail, Github). As if it's the first time this happens. As if
| after the last time AWS failed (or the one before that, or one
| before...) anybody stopped using AWS. As if anybody _could_
| viably stop using them.
| captainkrtek wrote:
| > It would be a good thing, if it would cause anything to
| change. It obviously won't.
|
| I agree wholeheartedly. The only change is internal to these
| organizations (eg: CloudFlare, AWS) Improvements will be made
| to the relevant systems, and some teams internally will also
| audit for similar behavior, add tests, and fix some bugs.
|
| However, nothing external will change. The cycle of pretending
| like you are going to implement multi-region fades after a
| week. And each company goes on continuing to leverage all these
| services to the Nth degree, waiting for the next outage.
|
| Not advocating that organizations _should /could_ do much, it's
| all pros/cons. But the collective blast radius is still
| impressive.
| chii wrote:
| the root cause is customers refusing to punish these
| downtime.
|
| Checkout how hard customers punish blackouts from the grid -
| both via wallet, but also via voting/gov't. It's why they are
| now more reliable.
|
| So unless the backbone infrastructure gets the same flak,
| nothing is going to change. After all, any change is
| expensive, and the cost of that change needs to be worth it.
| MikeNotThePope wrote:
| Is a little downtime such a bad thing? Trying to avoid some
| bumps and bruises in your business has diminishing returns.
| aaron_m04 wrote:
| Depends on the business.
| krige wrote:
| What's "a little downtime" to you might be work ruined
| and day wasted for someone else.
| fragmede wrote:
| It's 2025. That downtime could be be difference between
| my cat pics not loading fast enough, or someone's
| teleoperated robot surgeon glitching out.
| bloppe wrote:
| I remember a Google cloud outage years ago that happened
| to coincide with one of our customers' massively
| expensive TV ads. All the people who normally would've
| gone straight to their website instead got 502. Probably
| a 1M+ loss for them all things considered.
|
| We got an _extremely_ angry email about it.
| cactusplant7374 wrote:
| I have a lot of bad days every year. More than I can
| count. It's just part of living.
| Xelbair wrote:
| Even more so when most of the internet is also down.
|
| What are customers going to do? Go to competitor that's
| also down?
|
| It is extremely annoying, will ruin your day, but as
| movie quote goes - if everyone is special, no one is.
| immibis wrote:
| They could go to your competitor that's up. If you choose
| to be up, your competitor's customers could go to you.
| dewey wrote:
| If it's that easy to get the exact same service / product
| as another vendor the maybe your competitive advantage
| isn't so high. If Amazon would be down I'd just wait a
| few hours as I don't want to sign up on another site.
| MikeNotThePope wrote:
| I agree. These days it seems like everything is a micro-
| optimization to squeeze out a little extra revenue.
| Eventually most companies lose sight of the need to offer
| a compelling product that people would be willing to wait
| for.
| throwaway0352 wrote:
| I think you're viewing the issue from an office worker's
| perspective. For us, downtime might just mean heading to
| the coffee machine and taking a break.
|
| But if a restaurant loses access to its POS system (which
| has happened), or you're unable to purchase a train
| ticket, the consequences are very real. Outages like
| these have tangible impacts on everyday life. That's why
| there's definitely room for competitors who can offer
| reliable backup strategies to keep services running.
| mallets wrote:
| Those are examples where they shouldn't be using public
| cloud in the first place. Should build those services to
| be local-first.
|
| Using a different, smaller cloud provider doesn't improve
| reliability (likely makes it worse) if the architecture
| itself wrong.
| wongarsu wrote:
| Do any of those competitors actually have meaningfully
| better uptime?
|
| From a societal level, having everything shut down at
| once is an issue. But if you only have one POS system
| targeting only one backend URL (and that backend has to
| be online for the POS to work) then cloudflare seems like
| one of the best choices
|
| If the uptime provided by cloudflare isn't enough then
| the solution isn't a cloudflare competitor, it's the
| ability to operate offline (which many POS have,
| including for card purchases) or at least multiple
| backends with different DNS, CDN, server location etc.
| mopsi wrote:
| Downtimes happen one way or another. The upside of using
| Cloudflare is that bringing things back online is their
| problem and not mine like when I self-host. :]
|
| Their infrastructure went down for a pretty good reason
| (let the one who has never caused that kind of error cast
| the first stone) and was brought back within a reasonable
| time.
| whatevaa wrote:
| Grid reliability depends on where you live. In some places,
| UPS or even a generator is a must have. So it's a bad
| example, I would say.
| LoganDark wrote:
| > Checkout how hard customers punish blackouts from the
| grid - both via wallet, but also via voting/gov't.
|
| What? Since when has anyone ever been free to just up and
| stop paying for power from the grid? Are you going to pay
| $10,000 - $100,000 to have another power company install
| lines? Do you even have another power company in the area?
| State? Country? Do you even have permission for that to
| happen near your building? Any building?
|
| The same is true for internet service, although personally
| I'd gladly pay $10,000 - $100,000 to have literally
| anything else at my location, but there are no proper other
| wired providers and I'll die before I ever install any sort
| of cellular router. Also this is a rented apartment so I'm
| fucked even if there were competition, although I plan to
| buy a house in a year or two.
| heartbreak wrote:
| The hyperscalers definitely vote with their wallets.
| tjwebbnorfolk wrote:
| > the root cause is customers refusing to punish these
| downtime.
|
| ok how do I punish cloudflare -- build my own globally-
| distributed content-delivery network just for myself so
| that I can be "decentralized"?
|
| Or should I go to one of their even-larger competitors like
| AWS or GCP?
|
| What exactly do you propose?
| ehhthing wrote:
| With the rise in unfriendly bots on the internet as well as
| DDoS botnets reaching 15 Tbps, I don't think many people have
| much of a choice.
| swiftcoder wrote:
| The cynic in me wonders how much blame the world's leading
| vendor of DDoS prevention might share in the creation of that
| particularly problem
| immibis wrote:
| They provide free services to DDoS-for-hire services and do
| not terminate the services when reported.
| zamadatix wrote:
| Not that I doubt examples exist (I've yet to be at a
| large place with 0 failures on responding to such issues
| over the years), but it'd be nice if you'd share the
| specific examples you have in mind if you're going to
| bother commenting about it. It helps people understand
| how much is a systemic problem to be interested in vs
| having a comment which more easily falls into many other
| buckets instead. I'd try to build trust off the user
| profile as well, but it proclaims you're shadowbanned for
| two different reasons - despite me seeing your comment.
|
| One related topic I've seen brought up is Workers abuse
| https://www.fortra.com/blog/cloudflare-pages-workers-
| domains..., but that goes against this claim they do
| nothing when reported.
| sjamaan wrote:
| Same with the big Crowdstrike fail of 2024. Especially when
| everyone kept repeating the laughable statement that these guys
| have their shit in order, so it couldn't possibly be a simple
| fuckup on their end. Guess what, they don't, and it was. And
| nobody has realized the importance of diversity for resilience,
| so all the major stuff is still running on Windows and using
| Crowdstrike.
| c0l0 wrote:
| I wrote https://johannes.truschnigg.info/writing/2024-07-impe
| nding_g... in response to the CrowdStrike fallout, and was
| tempted to repost it for the recent CloudFlare whoopsie. It's
| just too bad that publishing rants won't change the darned
| status quo! :')
| graemep wrote:
| People will not do anything until something really
| disastrous happens. Even afterwards memories can fade.
| Cloudstrike has not lost many customers.
|
| Covid is a good parallel. A pandemic was always possible,
| there is always a reasonable chance of one over the course
| of decades. However people did not take it seriously until
| it actually happened.
|
| A lot of Asian countries are a lot better prepared for a
| tsunami then they were before 2004.
|
| The UK was supposed to have emergency plans for a pandemic,
| but it was for a flu variant, and I suspect even those
| plans were under-resourced and not fit for purpose. We are
| supposed to have plans for a solar storm but when another
| Carrington even occurs I very much doubt we will deal with
| it smoothly.
| testdelacc1 wrote:
| If anything, centralisation shields companies using a
| hyperscaler from criticism. You'll see downtime no matter where
| you host. If you self host and go down for a few hours,
| customers blame you. If you host on AWS and "the internet goes
| down", then customers treat it akin to an act of God, like a
| natural disaster that affects everyone.
|
| It's not great being down for hours, but that will happen
| regardless. Most companies prefer the option that helps them
| avoid the ire of their customers.
|
| Where it's a bigger problem is when a critical industry like
| retail banking in a country all choose AWS. When AWS goes down
| all citizens lose access to their money. They can't pay for
| groceries or transport. They're stranded and starving, life
| grinds to a halt. But even then, this is not the bank's problem
| because they're not doing worse than their competitors. It's
| something for the banking regulator and government to worry
| about. I'm not saying the bank shouldn't worry about it, I'm
| saying in practice they don't worry about it unless the
| regulator makes them worry.
|
| I completely empathise with people frustrated with this status
| quo. It's not great that we've normalised a few large outages a
| year. But for most companies, this is the rational thing to do.
| And barring a few critical industries like banking, it's also
| rational for governments to not intervene.
| DeathArrow wrote:
| >If anything, centralisation shields companies using a
| hyperscaler from criticism. You'll see downtime no matter
| where you host. If you self host and go down for a few hours,
| customers blame you.
|
| What if you host on AWS and only you go down? How does
| hosting on AWS shield you from criticism?
| testdelacc1 wrote:
| This discussion is assuming that the outage is entirely out
| of your control because the underlying datacenter you
| relied on went down.
|
| Outages because of bad code do happen and the criticism is
| fully on the company. They can be mitigated by better
| testing and quick rollbacks, which is good. But outages at
| the datacenter level - nothing you can do about that. You
| just wait until the datacenter is fixed.
|
| This discussion started because companies are actually fine
| with this state of affairs. They are risking major outages
| but so are all their competitors so it's fine actually. The
| juice isn't worth the squeeze to them, unless an external
| entity like the banking regulator makes them care.
| graemep wrote:
| > If anything, centralisation shields companies using a
| hyperscaler from criticism. You'll see downtime no matter
| where you host. If you self host and go down for a few hours,
| customers blame you.
|
| Not just customers. Your management take the same view. Using
| hyperscalers is great CYA. The same for any replacement of
| internally provided services with external ones from big
| names.
| testdelacc1 wrote:
| Exactly. No one got fired for using AWS. Advocating for
| self-hosting or a smaller provider means you get blamed
| when the inevitable downtime comes around.
| BlackFly wrote:
| I think this really depends on your industry.
|
| If you cannot give a patient life saving dialysis because you
| don't have a backup generator then you are likely facing some
| liability. If you cannot give a patient life saving dialysis
| because your scheduling software is down because of a major
| outage at a third party and you have no local redundancy then
| you are in a similar situation. Obviously this depends on
| your jurisdiction and probably we are in different ones, but
| I feel confident that you want to live in a district where a
| hospital is reasonably responsible for such foreseeable
| disasters.
| testdelacc1 wrote:
| Yeah I mentioned banking because of what I was familiar
| with but medical industry is going to be similar.
|
| But they do differ - it's never ok for a hospital to be
| unable to dispense care. But it is somewhat ok for one bank
| to be down. We just assume that people have at least two
| bank accounts. The problem the banking regulator faces is
| that when AWS goes down, all banks go down simultaneously.
| Not terrible for any individual bank, but catastrophic for
| the country.
|
| And now you see what a juicy target an AWS DC is for an
| adversary. They go down on their own now, but surely Russia
| or others are looking at this and thinking "damn, one
| missile at the right data Center and life in this country
| grinds to a halt".
| stingraycharles wrote:
| It's just a function of costs vs benefits. For most people,
| building redundancy at this layer costs far too much than the
| benefits.
|
| If Cloudflare or AWS go down, the outage is usually so big that
| smaller players have an excuse and people accept that.
|
| It's as simple as that.
|
| "Why isn't your site working?" "Half the internet is down, here
| read this news article: ..." "Oh, okay, let me know when it's
| back!"
| tcfhgj wrote:
| > As if anybody could viably stop using them.
|
| You can, and even save money.
| fragmede wrote:
| > It obviously won't.
|
| Here's where we separate the men from the boys, the women from
| the girls, the Enbys from the enbetts, and the SREs from the
| DevOps. If you went down when Cloudflare went do, do you go
| multicloud so that can't happen again, or do you shrug your
| shoulders and say "well, everyone else is down"? Have some
| pride in your work, do better, be better, and strive for
| greatness. Have backup plans for your backup plans, and get out
| of the pit of mediocrity.
|
| Or not, shit's expensive and kubernetes is too complicated and
| "no one" _needs_ that.
| rkomorn wrote:
| You make the appropriate cost/benefit decision for your
| business and ignore apathy on one side and dogma on the
| other.
| markus_zhang wrote:
| It's too few and far between. It's gonna make some changes if
| it's a monthly event. If businesses start to lose connection
| for 8 hours every month, maybe the bigger ones are going to run
| for self hosting or at least some capacity of self hosting.
| mkornaukhov wrote:
| Yeah, agree. But even in case of 8 hour downtime (it's almost
| 99% SLA) it isn't beneficial for really small firms.
| GuB-42 wrote:
| Same idea with the Crowdstrike bug, it seems like it didn't
| have much of on effect on their customers, certainly not with
| my company at least, and the stock quickly recovered, in fact
| doing very well. For me, it looks like nothing changed, no
| lessons learned.
| beanjuiceII wrote:
| what do you mean no lesson learned? seems like you haven't
| been paying attention..there's always a lesson learned
| peaseagee wrote:
| I believe they mean that Crowdstrike learned that they
| could screw up on this level and keep their customers....
| thewebguyd wrote:
| That's true of a lot of "Enterprise" software. Microsoft
| enjoys success from abusing their enterprise customers
| what seems like daily at this point.
|
| For bigger firms, the reality is that it would probably
| cost more to switch EDR vendors than the outage itself
| cost them, and up to that point, CrowdStrike was _the_
| industry standard and enjoyed a really good track records
| and reputation.
|
| Depending on the business, there are long term contracts
| and early termination fees, there's the need to run your
| new solution along side the old during migration, there's
| probably years of telemetry and incident data that you
| need to keep on the old platform, so even if you switch,
| you're still paying for CrowdStrike for the retention
| period. It was one (major) issue over 10+ years.
|
| Just like with CloudFlare, the switching costs are higher
| than outage cost, unless there was a major outage of that
| scale multiple times per year.
| beanjuiceII wrote:
| that IS the lesson! there are a million questions i can
| ask myself about those incidents. What dictates they
| can't ever screw up? sure it was a big screw up, but
| understanding the tolerances for screw ups is important
| to understanding how fast and loose you can play it. AWS
| has at least a big outage a year, whats the breaking
| point? risk and reward etc.
|
| I've worked places where every little thing is yak
| shaved, and places where no one is even sure if the
| servers are up during working hours. Both jobs paid
| well.. both jobs had enough happy customers
| ectospheno wrote:
| I'm pretty cloudflare centric. I didn't start that way. I had
| services spread out for redundancy. It was a huge pain. Then
| bots got even more aggressive than usual. I asked why I kept
| doing this to myself and finally decided my time was worth
| recapturing.
|
| Did everything become inaccessible the last outage? Yep.
| Weighed against the time it saves me throughout the year I call
| it a wash. No plans to move.
| 0x073 wrote:
| The outage wasn't a good thing, since nothing is changing as a
| result. (How many outages does cloud flare had?)
| timenotwasted wrote:
| "Embrace outages, and build redundancy." -- It feels like back in
| the day this was championed pretty hard especially by places like
| Netflix (Chaos Monkey) but as downtime has become more expected
| it seems we are sliding backwards. I have a tendency to rely too
| much on feelings so I'm sure someone could point me to some data
| that proves otherwise but for now that's my read on things.
| Personally, I've been going a lot more in on self-hosting lots of
| things I used to just mindlessly leave on the cloud.
| stroebs wrote:
| The problem is far more nuanced than the internet simply becoming
| too centralised.
|
| I want to host my gas station network's air machine
| infrastructure, and I only want people in the US to be able to
| access it. That simple task is literally impossible with what we
| have allowed the internet to become.
|
| FWIW I love Cloudflare's products and make use of a large amount
| of them, but I can't advocate for using them in my professional
| job since we actually require distributed infrastructure that
| won't fail globally in random ways we can't control.
| Fnoord wrote:
| Literally impossible? On the contrary; Geofencing is easy. I
| block all kind of nefarious countries on my firewall, and I
| don't miss them (no loss not being able to connect to/from a
| mafia state like Russia). Now, if I were to block FAMAG... or
| Cloudflare...
| stroebs wrote:
| Yes, literally impossible. The barrier to entry for anyone on
| the internet to create a proxy or VPN to bypass your
| geofencing is significantly lower than your cost to prevent
| them.
| Aurornis wrote:
| I don't even understand where this line of reasoning is
| going. Did you want a separate network blocked off from the
| world? A ban on VPNs? What are we supposed to believe could
| have been disallowed to make this happen?
| Dylan16807 wrote:
| I don't understand why you want to allow any random guy
| anywhere in the US but not people country hopping on VPNs.
| For your air machine infrastructure.
|
| It's a bit weird that you can't do this simple thing, but
| what's the motivation for this simple thing?
| Joel_Mckay wrote:
| Actually, the 140k Tor exit nodes, VPNs, and compromised
| proxy servers have been indexed.
|
| It takes 24 minutes to compile these firewall rules, but
| the black-list along with tripwires have proven effective
| at banning game cheats. Example, dropping connections from
| TX with a hop-count and latency significantly different
| from their peers.
|
| Preemptively banning all bad-reputation cloud IP ranges
| except whitelisted hosts has zero impact on clients. =3
| Fnoord wrote:
| I don't have a filter list for compromised proxy servers
| and VPNs. Do you have a link? I'd be interested in
| logging such. For Tor, I use [1] (formats in json, txt,
| md) on OPNsense, but I've also been able to indeed simply
| parse ASNs (which I currently use for "Twitter, Inc.").
|
| > Preemptively banning all bad-reputation cloud IP ranges
| except whitelisted hosts has zero impact on clients. =3
|
| This. There's outbound and inbound, and it is very
| unlikely your print server requires connections from
| Russia or China (to name an example). You're probably
| better off making a whitelist, jumphost, or using a VPN
| with proper authentication to access your services.
|
| Outbound, now that is more difficult to assess. On a
| desktop, I like a personal firewall for that purpose.
| Little Snitch on macOS and Open Snitch on Linux have
| helped me a lot here, but ultimately your hardware
| firewall is probably lenient on outgoing connections,
| when you should ask yourself does my network require
| this, or are they better off with only a HTTP(S) proxy by
| default?
|
| [1] https://github.com/7c/torfilter
| Aurornis wrote:
| > and I only want people in the US to be able to access it.
| That simple task is literally impossible with what we have
| allowed the internet to become.
|
| Is anyone else as confused as I am about how common anti-
| openness and anti-freedom comments are becoming on HN? I don't
| even understand what this comment wants: Banning VPNs? Walling
| off the rest of the world from US internet? Strict government
| identity and citizenship verification of people allowed to use
| the internet?
|
| It's weird to see these comments get traction after growing up
| in an internet where tech comments were relentlessly pro
| freedom and openness on the web. Now it seems like every day I
| open HN and there are calls to lock things down, shut down
| websites, institute age (and therefore identify) verification
| requirements. It's all so foreign and it feels like the vibe
| shift happened overnight.
| dmoy wrote:
| > Is anyone else as confused as I am about how common anti-
| openness and anti-freedom comments are becoming on HN?
|
| In this specific case I don't think it's about being anti-
| open? It's that a business with only physical presence in one
| country selling a service that is only accessible physically
| inside the country.... doesn't.... have any need for selling
| compressed air to someone who isn't like 15 minutes away from
| one of their gas stations?
|
| If we're being charitable to GP, that's my read at least.
|
| If it was a digital services company, sure. Meatspace in only
| one region though, is a different thing?
| vpribish wrote:
| you're being obtuse, GP clearly wants a locked down
| internet
| teiferer wrote:
| > In this specific case I don't think it's about being
| anti-open? It's that a business with only physical presence
| in one country selling a service that is only accessible
| physically inside the country.... doesn't.... have any need
| for selling compressed air to someone who isn't like 15
| minutes away from one of their gas stations?
|
| But that person might be physically further away at the
| time they want to order something or gather information
| etc. Maybe they are on holidays in Spain and want to access
| their account to pay a bill. Maybe they are in Mexico on a
| work trip and want to help their aunt back home to use some
| service for which they need to log in from abroad.
|
| The other day I helped a neighbor (over here in Europe)
| prepare for a trip to Canada where he wanted to make
| adjustments to a car sharing account. The website always
| timed out. It was geofenced. I helped him set up a VPN.
| That illustrated how locked in this all has become,
| geofencing without thinking twice.
| dmoy wrote:
| I guess GP didn't provide enough info, but to me it
| looked like it was the underlying infra that is networked
|
| That is I'm assuming:
|
| 1. Customers are meatspace only, never use any computer
| interface 2. The network access is for administration
| only 3. That administration is exclusively in the US
| Dylan16807 wrote:
| That's the most obvious answer but if that's the case
| then restricting to "US" is _way_ too wide in the general
| case and also too narrow if an employee takes a trip to
| another country and tries to check in. That simple task
| is fundamentally flawed to the point it 's not worth
| worrying about.
| tensegrist wrote:
| "only need US customers to be able to" vs "want non-US
| customers to be unable to"
| Aurornis wrote:
| > In this specific case I don't think it's about being
| anti-open?
|
| The anti-open part was the mention of "allowed to become",
| as if we needed to disallow something to achieve this
| unstated goal.
| thewebguyd wrote:
| > It's all so foreign and it feels like the vibe shift
| happened overnight.
|
| The cultural zeitgeist around the internet and technology has
| changed, unfortunately. But it definitely didn't happen
| overnight. I've been witnessing it happen slowly over the
| past 8-10 years, with it accelerating rapidly only in the
| last 5.
|
| I think it's a combination of special interest groups &
| nation states running propaganda campaigns, both with bots
| and real people, and a result of the internet "growing up."
| Once it became a global, high-stakes platform for finance and
| commerce, businesses took over, and businesses are
| historically risk averse. Freedom and openness is no longer a
| virtue but a liability (for them).
| zrm wrote:
| > I want to host my gas station network's air machine
| infrastructure, and I only want people in the US to be able to
| access it. That simple task is literally impossible with what
| we have allowed the internet to become.
|
| That task was never simple and is unrelated to Cloudflare or
| AWS. The internet at a fundamental level only knows where the
| next hop is, not where the source or destination is. And even
| if it did, it would only know where the machine is, not where
| the person writing the code that runs on the machine is.
| teiferer wrote:
| And that is a good thing and we should embrace it instead of
| giving in to some idiotic ideas from a non-technical C-suite
| demanding geofencing.
| asimovDev wrote:
| not a sysadmin here. why wouldn't this be behind a VPN or some
| kind of whitelist where only confirmed IPs from the offices /
| gas stations have access to the infrastructure?
| yardstick wrote:
| In practice, many gas stations have VPNs to various services,
| typically via multiple VPN links for redundancy. There's no
| reason why this couldn't be yet another service going over a
| VPN.
|
| Gas stations didn't stop selling gas during this outage. They
| have planned for a high degree of network availability for
| their core services. My guess is this particular station is
| an independent or the air pumping solution not on anyone's
| high risk list.
| Joel_Mckay wrote:
| Client side SSL certificates with embedded user account
| identification are trivial, and work well for publicly exposed
| systems where IPsec or Dynamic frame sizes are problematic
| (corporate networks often mangle traffic.)
|
| Accordingly, connections from unauthorized users is effectively
| restricted, but is also not necessarily pigeonholed to a single
| point of failure.
|
| https://www.rabbitmq.com/docs/ssl
|
| Best of luck =3
| Xelbair wrote:
| Genuine question - why are you spending time and effort on
| geofencing when you could spend it on improving your
| software/service?
|
| It takes time and effort for no gain in any sensible business
| goal. People outside of US won't need it, bad actors will spoof
| their location, and it might inconvenience your real customers.
|
| And if you want a secure communication just setup zero-trust
| network.
| WJW wrote:
| > bad actors will spoof their location
|
| Isn't that exactly the point? Why are North Korean hackers
| even allowed to connect to the service, and why is spoofing
| location still so easy and unverifiable?
|
| Nobody is expected to personally secure their physical
| location against hostile state actors. My office is not
| artillery proof, nor does it need to be: hostile actions
| against it would be an act of war and we have the military to
| handle those kind of things. But with cybersecurity suddenly
| everyone is expected to handle everyone from the script
| kiddie next door to the Mossad. I see the point in OPs post:
| perhaps it would be good if locking down were a little easier
| than "just setup zero-trust network".
| Xelbair wrote:
| you can as easily get attackers from within your own
| networks, you're falling for fallacy that everything on the
| 'inside' is secure.
| WJW wrote:
| Just because one group of attackers is (/might be) inside
| your network doesn't mean you also have to let all other
| groups in. There is zero reason to let (say) North
| Koreans interact with your gas pump API, other than that
| the internet is set up so that it is virtually impossible
| to prevent unfriendly parties from contacting your
| servers.
| Aurornis wrote:
| > Why are North Korean hackers even allowed to connect to
| the service,
|
| Asking why some group is "allowed" to use the internet is
| equivalent to demanding either strict verification or that
| we cut off some entire country where they reside from the
| entire internet.
|
| Either that, or someone doesn't understand basic
| fundamentals of networking and thinks there's some magic
| solution to this problem.
|
| A common variation of this comment is "why do we allow kids
| to access <insert topic here>" with demands that something
| be done about it. Then when something is done about it,
| there is shock and outrage upon realizing that you can't
| filter out children without forcing identity verification
| upon everyone. Similar vibes here, just replace age with
| demographic.
| WJW wrote:
| It wouldn't surprise me at all if mandatory online ID
| verification will become a thing within the next century
| or so.
| Dylan16807 wrote:
| North Korea in particular is weird because of sanctions,
| but pick any country in Europe instead: The user might be a
| past or future visitor to the gas station and need to
| access the system even if they're outside the US right now.
| Or maybe they're actually at the gas station but their
| phone's data is based in Europe.
|
| Even accurate country tracking is flawed in most
| situations.
|
| If the goal is specifically "is at the gas station right
| now" then maybe there's a gap in functionality here, but
| you could make them connect to the wifi.
|
| Also country-sponsored hackers can easily get a real
| presence in the US. If country level geoblocking became
| perfect, they wouldn't be slowed down for more than a week.
| notepad0x90 wrote:
| Is Cloudflare having more outages than aws, gcp or azure?
| Honestly curious, I don't know the answer.
| nananana9 wrote:
| Definitely not.
|
| I was a bit shocked when my mother called me for IT help and
| sent me a screenshot of a Cloudflare error page with
| Cloudflare being the broken link and not the server. I
| assumed it's a bug in the error page and told her that the
| server is down.
| L-four wrote:
| It's a tragedy of the commons. Even if you don't use Cloudflare
| does it matter if no one can pay for your products.
| tonyhart7 wrote:
| I don't like this argument since you can applied this argument to
| google,microsot,aws,facebook etc
|
| Tech world is dominated by US company and what is alternative to
| most of these service???? its a lot fewer than you might think
| and even then you must make a compromise in certain areas
| throwaway81523 wrote:
| Now just wait til every country on earth really does replace most
| of its employees with ChatGPT... and then OpenAI's data center
| goes offline with a fiber cut or something. All work everywhere
| stops. Cloudflare outage is nothing compared to that.
| DeathArrow wrote:
| That's why it's better to have redundancy. Hire Claude and
| Deepseek, too.
| teiferer wrote:
| > goes offline with a fiber cut
|
| If a fiber cut brings your network down then you have
| fundamental network design issues and need to change hiring
| practices.
| delaminator wrote:
| That was this outage. ChatGPT and Claude are both behind
| Clouflare's bot detection. You couldn't log into either Web
| frontends.
|
| And the error message said you were blocking them. We had
| support tickets coming in demanding to know why ChatGPT was
| being blocked.
|
| We also couldn't log into our supplier's B2B system to place
| our customer orders.
|
| So all the advice of "just self host" is moot when you're in a
| food web.
| oidar wrote:
| I wonder what would life without cloudflare look like? What
| practices would fill the gaps if a company didn't - or wasn't
| allowed to -- satisfy the the concerns that cloudflare fills.
| immibis wrote:
| Pretty much exactly like it does now but with less captchas.
|
| Fun fact: Headless browsers can easily pass cloudflare captchas
| automatically. They're not actually captchaing - they're just a
| placebo. You just need to be coming from a residential IP
| address and using a real browser.
| hombre_fatal wrote:
| > Pretty much exactly like it does now but with less
| captchas.
|
| This just isn't true. e.g. I saw a 30x increase in traffic on
| my forum due to AI bots that I had to use CF to block.
|
| CF is mainly empowered by the naive ideals of the internet's
| design that never built-in countermeasures against bad
| actors. You're expected to just deal with it yourself
| somehow. And that means outsourcing it, especially as
| residential IP address botnets on unlimited ISP data plans
| become cheaper and cheaper.
|
| Just ask yourself why web hosting providers themselves can't
| offer services at CF's level. It's because it's too hard of a
| problem even for them.
| immibis wrote:
| You didn't _have to_ use CF to block them. You chose to use
| CF to block them. How was your experience with Anubis or
| https://git.gammaspectra.live/git/go-away?
|
| Or you could simply... serve the requests. If your normal
| traffic is only, like, 1 request per minute, then 30x that
| is still pretty low and there's no actual reason to worry
| about it.
|
| Web hosting providers don't offer bot blockers because
| first, they have no reason to care, and second, they can
| serve the requests, and third, some of them want to upsell
| you on bandwidth (you should prefer the ones with unmetered
| bandwidth).
|
| BTW AFAIK there's still zero evidence that the massive DDoS
| wave has anything at all to do with AI. It could be, say,
| one of Russia's many small avenues of trying to break the
| West, or Cloudflare trying to get more business, or the NSA
| trying to make Cloudflare get more business because it's
| tapped into Cloudflare.
| zie1ony wrote:
| My friend wasn't able to do RTG during the outage. They had to
| use ultrasound machine on his broken arm to see inside.
| Aurornis wrote:
| > My friend wasn't able to do RTG during the outage.
|
| What is RTG?
| soni96pl wrote:
| X-ray
| digestives wrote:
| X-ray, in some languages (like Polish) the abbreviation comes
| from https://en.wikipedia.org/wiki/Roentgen_(unit)
| teiferer wrote:
| Wilhelm Rontgen, Nobel Prize in 1901, experimentally
| discovered X-rays.
| Surac wrote:
| The thing I learned from the incident is that rust offer a unpack
| function. It puzzles me why the hell they build such a function
| in the first place.
| aw1621107 wrote:
| > It puzzles me why the hell they build such a function in the
| first place.
|
| One reason is similar to why most programming languages don't
| return an Option<T> when indexing into an
| array/vector/list/etc. There are always tradeoffs to make,
| especially when your strangeness budget is going to other
| things.
| almosthere wrote:
| how many people are still on us-east-1
| mcny wrote:
| My old employer used azure. It irritated me to no end when they
| said we must rename all our resources to match the convention
| of naming everything US East as "eu-" because (Eastern United
| States I guess)
|
| A total clown show
| 0xbadcafebee wrote:
| Centralization has nothing to do with the problems of society and
| technology. And if you think the internet is _all_ controlled by
| just a couple companies, you don 't actually understand how it
| works. The internet is _wildly_ decentralized. Even Cloudflare
| is. It offers tons of services, all of which are completely
| optional and can be used individually. You can also stop using
| them at any time, and use any of their competitors (of which
| there are many).
|
| If, on the off chance, people just get "addicted" to Cloudflare,
| and Cloudflare's now-obviously-terrible engineering causes
| society to become less reliable, then people will respond to
| that. Either competitors will pop up, or people will depend on
| them less, or governments will (finally!) impose some regulations
| around the operation of technical infrastructure.
|
| We have actually _too much_ freedom on the Internet. Companies
| are free to build internet systems any way they want - including
| in very unreliable ways - because we impose no regulations or
| standards requirements on them. Those people are then free to
| sell products to real people based on this shoddy design, with no
| penalty for the products falling apart. So far we haven 't had
| any gigantic disasters (Great Chicago Fire, Triangle Shirtwaist
| Factory Fire, MGM Grand Hotel Fire), but we have had major
| disruptions.
|
| We already dealt with this problem in the rest of society.
| Buildings have building codes, fire codes, electrical codes. They
| prescribe and require testing procedures, provide standard
| building methods to ensure strength in extreme weather, resist a
| spreading fire long enough to allow people to escape, etc. All
| measures to ensure the safety and reliability of the things we
| interact with and depend on. You can build anything you want -
| say, a preschool? - but you aren't allowed to build it in a
| shoddy manner. We have that for physical infrastructure; now we
| need it for virtual infrastructure. A software building code.
| DeathArrow wrote:
| Centralization means having a single point of failure for
| everything. If your government, mobile phone or car stops
| working, it doesn't mean all governments, all cars and all
| mobile phones stop working.
|
| Centralization makes mass surveillance easier, makes
| selectively denying of service easier. Centralization also
| means that once someone hacks into the system, he gains access
| to all data, not just a part of it.
| vasco wrote:
| I'll die on the hill that centralization is more efficient than
| decentralization and that rare outages of hugely centralized
| systems that are otherwise highly reliable are much better than
| full decentralization with much worse reliability.
|
| In other words, when AWS or Cloudflare go down it's catastrophic
| in the sense that everyone sees the issues at the same time, but
| smaller providers usually have much more ongoing issues, that
| just happen to be "chronic" vs "acute" pains.
| Xelbair wrote:
| >I'll die on hill that hyperoptimized systems are more
| efficient than anti-fragile.
|
| Of course they are, the issue is what level of failure were
| going to accept.
| GeneralMaximus wrote:
| Efficient in terms of what, exactly?
|
| There are multiple dimensions to this problem. Putting
| everything behind Cloudflare might give you better uptime,
| reliability, performance, etc. but it also has the effect of
| centralizing power into the hands of a single entity. Instead
| of twisting the arms of ten different CXOs, your local
| politician now only needs to twist the arm of a single CXO to
| knock your entire business off the internet.
|
| I live in India, where the government has always been hostile
| to the ideals of freedom of speech and expression. Complete
| internet blackouts are common in several states, and major ISPs
| block websites without due process or an appeals mechanism.
| Nobody is safe from this, not even Github[1]. In countries like
| India, decentralization is a preventative measure.
|
| [1] https://en.wikipedia.org/wiki/Censorship_of_GitHub#India
|
| And I'm not even going to talk about abuse of monopoly power
| and all that. What happens when Cloudflare has their Apple
| moment? When they jack up their prices 10x, or refuse to serve
| customers that might use their CDNs to serve "inappropriate"
| content? When the definition of "inappropriate" is left fuzzy,
| so that it applies to everything from CSAM to political
| commentary?
|
| No thanks.
| vasco wrote:
| The fix to government censorship must be political, not
| technical.
| torginus wrote:
| And the irony is that people are pushing for decentralization
| like microservices and k8s - on centralized platforms like AWS.
| rzerowan wrote:
| So were going backwards to a world where there are basically 5
| computers running everything and everyone is basically accessing
| the world through a dumb terminal.Even though the digital slab in
| our pockets has more compute than a roomful of the early gen
| devices. Hopefully critical infrashifts back to managed metal or
| private clouds - dont see it though with the last decades of
| cloud evangalism to move all legacy systems to the cloud doesnt
| look like reversing anytime soon.
| zwnow wrote:
| I agree considering all the Cloudflare AWS Azure apologists I
| see all around... Learning AWS already is the #1 tip on social
| media to "become employed as a dev in 2025 guaranteed" and I
| always just sigh when seeing this. I wouldnt touch it with a
| stick.
| fragmede wrote:
| Yeah it's crazy to realize it takes a room of electronics for
| me to get my (g)mail. The more things change, the more they
| stay the same, eh?
| tomschwiha wrote:
| For me personally I didn't notice the downtime in the first hour
| or so. When using some website assets were not loading, but
| that's it. Turnstile outage maybe impacted me most. Could be
| because I'm EU based and Cloudflare is not "so" widespread here
| as in other parts of the world.
| notepad0x90 wrote:
| Does the author of this post not see the irony of posting this
| content on Github?
|
| My counter argument is that "centralization" in a technical sense
| isn't about what company owns things but how services are
| operated. Cloudflare is very decentralized.
|
| Furthermore, I've seen regional outages caused by things like
| anchors dropped by ships in the wrong place, a shark eating a
| cable. Regional power outages caused by squirrels,etc... outages
| happen.
|
| If everyone ran their own server from their own home, AT&T or
| Level3 could have an outage and still take out similar swathes of
| the internet.
|
| With CDNs like cloudflare, if Level3 had an outage, your website
| won't be down because your home or VPS host's upstream transit
| happens to be Level3 (or whatever they call themselves these
| days) because your content (at least static) is cached globally.
|
| The only real reasonable alternative is something like ipfs, web3
| and similar talk.
|
| Cloudflare has always called itself a content transport provider,
| think of it as such. But also, Cloudflare is just one player,
| there are several very big players. Every big cloud provider has
| a competing product, not to mention companies like Akamai.
|
| People are rage posting about cloudflare, especially because it
| has made CDNs accessible to everyone. You can easily setup a free
| cloudflare account and be on your merry way. This isn't something
| you should be angry about. You're free to pay for any number of
| other cdns, many do.
|
| If you don't like how Cloudflare has so much market share, then
| come up with a similarly competitive alternative and profit. Just
| this HN thread alone is enough for me to think there is a market
| for more players. Or, just spread the word about the competition
| that exists today. Use frontdoor, cloudfront, netlify, flycdn,
| akamai,etc... It's hardly a monopoly.
| jcattle wrote:
| "The Cloudflare outage was a good thing [...] they're a warning.
| They can force redundancy and resilience into systems."
|
| - he says. On Github.
| Afforess wrote:
| Thanks for doing the meme! https://knowyourmeme.com/memes/we-
| should-improve-society-som...
|
| You are very intelligent!
| jcattle wrote:
| That's fair. However I don't think I would have wrote that if
| those thoughts were shared on a blogging platform.
|
| Most blogging platforms do not qualify as critical
| infrastructure. GitHub with all its CI/CD and supply chain
| attacks does.
|
| There is a certain particular irony of this being written on
| critical (centralized) infrastructure without any apparent
| need.
|
| Maybe it was intended, maybe not, in any case I found it
| funny.
| rkomorn wrote:
| I agree. I think the whole point is someone like TFA author
| has a pretty broad choice of places they can choose to
| publish this and choosing GitHub is somewhat ironic.
|
| Reminds me of the guy who posted an open letter to Mark
| Zuckerberg like "we are not for sale" on LinkedIn, a place
| that literally sells access to its users as their main
| product.
| nicman23 wrote:
| i hate that i cannot just scrape things for me usage and i have
| to use things like camufox instead of curl
| torginus wrote:
| What happens if you don't use Cloudflare and just host everything
| on a server?
|
| Can't you run a website like that if you don't host heavy
| content?
|
| How common are DDOS attacks anyway, and aren't there local (to
| the server), that analyze user behavior to a decent accuracy (at
| least it can tell they're using a real browser and behaving more
| or less like a human would, making attacks expensive).
|
| Can't you buy a list of ISP ranges from a GeoIP provider (you
| can), at least then you'd know which addresses belong to real
| humans.
|
| I don't think botnets are that big of a problem (maybe in some
| obscure places of the world, but you can temp rangeban a certain
| IP range, if there's a lot of suspicious traffic coming from
| there).
|
| If lots of legit networks (as in belonging to people who are
| paying an ISP for their network connections) have botnets, that's
| means most PCs are compromised, which is a much more severe
| issue.
| dijit wrote:
| Yeah, you can.
|
| Lots of people use raspberry pi's for this, which is a smidge
| anaemic for some decent load (HN Hug Of Death)- even an Intel
| N100 is more grunt, for context.
|
| This makes people think that their self hosting setup can
| _never_ handle HN load; because when they see people talking
| about self hosting the site goes down.
| rainonmoon wrote:
| Most people shouldn't use a Pi because most people can't
| configure a web server securely. A VPS would be a better
| option for just about everybody trying to "self-host" whether
| they put Cloudflare in front of it or not.
| dijit wrote:
| in both cases you're setting up a webserver.
|
| I guess you're concerned about lateral network movement?
| Justified, but as long as it's patched it's going to be
| just as secure.
| rainonmoon wrote:
| You're right, but with an asterisk. I don't care if my DO
| droplet gets popped with an RCE. I do care if someone
| establishes persistence in my home.
| 1718627440 wrote:
| You can have different networks in your physical home.
| rainonmoon wrote:
| And?
| 1718627440 wrote:
| Meaning your internal network and your publicly hosted
| services need to not to be in the same network.
| dewey wrote:
| Botnets use real residential connections not just data centers.
| So your static list of "real people" doesn't really make a
| difference.
| justsomehnguy wrote:
| > What happens if you don't use Cloudflare and just host
| everything on a server?
|
| It works.
|
| > Can't you run a website like that if you don't host heavy
| content?
|
| Even with a heavy content - question is how many visitors do
| you have. If there is one once an hour you would suffice on a
| 100Mbit/Unlim connection.
|
| > How common are DDOS attacks anyway
|
| Extremely rare. 99% of _sites_ never experience it, 1% do have
| _some_ trouble because somebody nearby is being DDoS 'ed.
|
| > and aren't there local (to the server), that analyze user
| behavior to a decent accuracy (at least it can tell they're
| using a real browser and behaving more or less like a human
| would, making attacks expensive).
|
| No point, you can't do anything anyway - it's a _denial_ of
| service so there are gigabytes of trash flowing your way.
|
| > Can't you buy a list of ISP ranges from a GeoIP provider (you
| can), at least then you'd know which addresses belong to real
| humans.
|
| No point. If you are not being DDoS'ed then you just spent
| money and time (ie money) on useless preventive measure you
| never use. And when (if) it would come you can't do anything
| anyway, because it's a distributed denial of service attack.
|
| > I don't think botnets are that big of a problem (maybe in
| some obscure places of the world, but you can temp rangeban a
| certain IP range, if there's a lot of suspicious traffic coming
| from there).
|
| It's not a DDoS if you can filter at the endpoint.
| miki123211 wrote:
| I don't know how many times I need to say this, but I will die on
| this hill.
|
| Centralized services don't decrease redundancy. They're usually
| far more redundant than whatever homegrown solution you can come
| up with.
|
| The difference between centralized and homegrown is mostly
| psychological. We notice the outages of centralized systems more
| often, as they affect everything at the same time instead of
| different systems at different times. This is true even if, in a
| hypothetical world with no centralization, we'd have more total
| outage time than we do now.
|
| If your gas station says "closed" due to a problem that only
| affects their own networks, people usually go "aah they're
| probably doing repairs or something", and forget about the
| problem 5 minutes later. If there's a Cloudflare outage...
| everybody (rightly) blames the Cloudflare outage.
|
| Where this becomes a problem is when correlated failures are
| actually worse than uncorrelated ones. If Visa goes down, it's
| better if Mastercard stays up, because many customers have both
| and can use the other when one doesn't work. In some ways, it's
| better to have 30 mins of Visa outages today and 30 mins of
| Mastercard outages tomorrow, than to have just 15 mins of
| correlated outages in one day.
| dgan wrote:
| > Centralized services don't decrease redundancy
|
| Alright, but it creates a failure correlation where previously
| there was none
| silvestrov wrote:
| Have you ever heard of the "sendmail worm", aka Morris Worm ?
|
| https://en.wikipedia.org/wiki/Morris_worm
|
| You can definitely have failure correlation without having
| centralized services.
| masfuerte wrote:
| In my experience services aren't failing due to a lack of
| redundancy but due to an excess of complexity. With the move to
| the cloud we are continually increasing both redundancy and
| complexity and this is making the problem worse.
|
| I have a cheap VPS that has run reliably for a decade except
| for a planned hour of downtime. Which was in the middle of the
| night when no-one cared. Amazon is more reliable in theory. My
| cheap VPS is more reliable in practice.
| lloeki wrote:
| "redundancy" might not be there correct word. If we had a
| single worldwide mega-entity serving 100% of the internet it
| would be both a monopoly and would have tons of redundant
| infrastructure.
|
| But it would also be quite unified; the system, while full of
| redundancies, as a whole is a unique one operated the same way
| end to end; by virtue of it being a single system handled in a
| uniform way, a single glitch could bring it all down. There is
| no diversity in the system's implementation, the monoculture
| itself makes it vulnerable.
| freeplay wrote:
| The problem is creating a single point of failure.
|
| There's no doubt a VM in AWS is exponentially more redundant
| than my VM running on a couple of Intel NUCs in my closet.
|
| The difference is, when I have a major outage, my blog goes
| down.
|
| When EC2 has a major outage, all of the blogs go down. Along
| with Wikipedia, Starbucks, and half the internet.
|
| That single point of failure is the issue.
| YetAnotherNick wrote:
| Single point of failure means exactly opposite of what you
| think it means. If my work depends on 5 services to be up,
| each service would be a single point of failure, and
| correlation of failure is good for probability that I can do
| my work.
| smj-edison wrote:
| This is a really interesting point, because I could see a
| situation where your application requires integration with
| say 10 services. If they all run on AWS, they either all go
| down or all run together. If they're all self-hosted,
| there's a good chance that at any time one of the ten is
| down, and so your service can't run.
| freeplay wrote:
| I see what you're saying but I have to push back.
|
| "If one thing I need is going to be down, everything might
| as well be down."
|
| If I have a product with 5 dependencies and one of them is
| down, there's things I can do to partially mitigate. A
| circuit breaker would allow my thing to at least stay up
| and responsive. Maybe I could get a status message up and
| turn off a feature flag to disable what calls that
| dependency.
|
| On the other hand, if all my dependencies are down _AND_
| the management layer is down _AND_ the AWS portal is not
| functioning correctly, I 'm pretty much SOL.
|
| Massive centralization is never, ever a good thing for
| anyone other than the ones who are doing the centralizing.
| chrisjj wrote:
| True title: The Cloudflare outage was a good thing
| joeblubaugh wrote:
| meta: why are we rewriting such anodyne titles? "was" -> "might
| be" undermines the author's point
| mrasong wrote:
| Yeah, when it went down, a bunch of the sites I use every day
| just stopped working.
|
| That's when I realized it's basically one of the backbone pieces
| of the entire internet.
| ovo101 wrote:
| Outages like this highlight just how much of the internet's
| resilience depends on a single provider. In a way, it's a healthy
| reminder: if one company's hiccup can take down half the web,
| maybe we've over-centralized. A "good thing" only if it sparks
| more serious conversations about redundancy, multi-provider
| strategies, and reducing monoculture risk. Otherwise, we'll just
| keep repeating the same failure modes at larger scales.
| rafaelcosta wrote:
| I don't get why this applies on the Cloudflare outage but not on
| the AWS ones... I'd argue that the big cloud providers are WAY
| more impactful when they go down than Cloudflare. The only
| difference is that the average techie uses Cloudflare more and
| sees the impact more, but this point was already there before...
| SirMaster wrote:
| If these systems are as important as they say, it's surprising to
| me that they are not built with backups and redundancies in place
| like other mission critical things are engineered and built with.
| tjwebbnorfolk wrote:
| Every HN comment seems to say the same thing: downtime is
| inexcusable and the centralization of these services is ruining
| the internet.
|
| I still don't see the big deal. 12 hours of downtime once every
| couple years isn't the end of the world. So people can't log into
| their bank website for a few hours -- banks used to only be open
| for like 4 hours a day and somehow we all survived. Twitter is
| down? Oh what a tragedy. Customers get some refunds, Cloudflare
| fixes the issue, and people move on with life.
|
| Cars still break down occasionally after 100+ years of
| engineering for reliability and safety. The power still goes out
| every now and then. Cook on the stove. The cost of making
| everything perfect all the time just isn't worth it.
|
| I run my own servers on my own network and do not use Cloudflare.
| My stuff goes down too. And it's "decentralized" in the way you
| think the internet "should" be, which entails its own risks. So
| what do you all want, exactly? A public lashing of every
| developer at Cloudflare who pushes a bug to prod? A congressional
| investigation? I just don't understand the outrage here.
|
| Stuff breaks occasionally. Get used to it, and design
| accordingly.
| joshuamcginnis wrote:
| From a consumers perspective, that makes sense. From a
| business's perspective, downtime can mean significant loss of
| revenue or new business opportunity.
| tjwebbnorfolk wrote:
| The costs of perfection are much, much greater. Are you
| willing to pay 2-3x the cost of everything to go from 99.999%
| to 100.0000000% uptime?
|
| Probably the only thing in existence with 100.00% uptime are
| our nuclear missile command and control systems. Like, even
| my pen runs out of ink sometimes. It's just crazy how hard it
| is to have stuff work all the time.
| ahepp wrote:
| I wonder if consolidation actually makes this less of an
| issue for businesses?
|
| If my website is down, but my competitors' isn't, I might
| lose business to them. If my competitor's website is also
| down, where are the customers gonna go?
| rigrassm wrote:
| > So people can't log into their bank website for a few hours,
| banks used to only be open for like 4 hours a day and somehow
| we all survived.
|
| 1. I believe it's payment processing systems not functioning
| properly that causes real problems for people and not simply
| bank websites being down. Especially given...
|
| 2. Banks being closed so much back when cash/checks were
| actually widely used wasn't an issue because you could just pop
| over to an ATM or whip out a checkbook. In today's system,
| every single purchase you make requires communication between
| the merchant, your bank, and any number of middlemen via the
| internet.
|
| Yeah, cash is still used today but I've been noticing even
| things like school sports events have stopped taking cash all
| together and simply post a QR code to buy from your phone.
|
| That is unless the school has crap cell reception (with no
| public Wi-Fi either!), Cloudflare shits the bed, Visa thinks
| you're buying porn, you locked your debit card and now can't
| unlock it cuz the website is down, or any one of the million
| things that break all the time. Replace school sports event
| with literally every single things that requires a financial
| transaction and it's easy to see how even a short outage can
| lead to actual harm being realized.
| YmiYugy wrote:
| It's worth considering the counter factual. Let's say there would
| be a few dozen semi popular DDoS services. Would that be better?
| Some assumptions: The services would be slightly less effective
| and also have worse downtimes. You could argue that Cloudflare is
| coasting on a monopoly and that competition would drive them to
| improve, but I'm pretty confident that DDoS protection it one of
| those things were having a large network to absorb attacks and a
| large team to monitor them if very valuable. I submit as evidence
| that Cloudflare has been doing well despite the 3 big cloud
| providers offering DDoS protection.
|
| So what would be the result of a highly decentralized but
| slightly worse and less reliable DDoS protection? I'd argue that
| for a lot of things this wouldn't be an improvement. Cloudflare
| being so dominant means lot's of things go down simultaneously.
| But that only matters for fungible services, e.g. if a schools
| education portal goes down, it doesn't matter if all the other
| education portals are also down. There are cases where it matters
| like the tyre pumps. I'd argue that these devices have no reason
| to be reliant on an online connection to begin with. I think
| cloud services as a whole have massively improved the reliability
| of internet services. In almost all cases reducing the overall
| amount of outages is a higher priority than preventing outage
| correlations.
___________________________________________________________________
(page generated 2025-11-24 23:01 UTC)