[HN Gopher] Cloudflare Is Having Issues
___________________________________________________________________
Cloudflare Is Having Issues
Author : Kuinox
Score : 332 points
Date : 2023-06-12 14:59 UTC (8 hours ago)
(HTM) web link (www.cloudflarestatus.com)
(TXT) w3m dump (www.cloudflarestatus.com)
| bookofjoe wrote:
| Why not do what Orwell advised and call them "problems?"
| nektro wrote:
| hope they stay down
| voyager1 wrote:
| We use Cloudflare primarily for CDN, but they are having issues a
| few times a year and I am looking for an alternative. Any
| suggestions?
| rs999gti wrote:
| Amazon Cloudfront with WAF
|
| Azure CDN with Front Door
|
| etc.
|
| etc.
| seydor wrote:
| reddit, hn, cloudflare down. This is a bad day for the internet
| anaganisk wrote:
| Lol what? TikTok, Facebook, Google, YouTube, AWS seem to be
| doing just fine.
| josefresco wrote:
| Sorting comments by date would be helpful on large outage threads
| like this that are now hours (4) old.
| jgrahamc wrote:
| I suppose this is at the top of HN because of Reddit having an
| outage and Hacker News being slow. But neither Reddit nor HN use
| Cloudflare. And if you look at the status page there this isn't
| our core CDN offering, it's certain products that are affected.
| freedomben wrote:
| Given the innaccurate and clickbaity title "Cloudflare is
| having Issues" (when in reality it's just a few services, not
| the main stuff Cloudflare is largely known for), I suspect a
| lot of people are upvoting and not actually following the link.
| A comprehensive Cloudflare issue would be huge news. The truth
| here is not huge news IMHO.
| k8sToGo wrote:
| The statement "Cloudflare is having issues" is true as soon
| as count(issues) > 0. So what's so wrong about the title?
|
| And how do you know how many websites use R2 etc. so that you
| can jump to the conclusion that it's not huge news?
| HWR_14 wrote:
| > The statement "Cloudflare is having issues" is true as
| soon as count(issues) > 0.
|
| Pedantically, it's true as soon as "count(issues) > 1"
| freedomben wrote:
| Exactly. The headline isn't _untrue_ , but it is
| misleading. Most people hear "Cloudflare is having
| issues" and assume Cloudflare as a whole, and this is not
| an unreasonable assumption.
|
| Reductio ad absurdum can be used to demonstrate the logic
| flaw. Let's assume that one person using Dynamo DB got a
| 500 error response from a two successive API calls (so an
| error rate of ~4.0e-09). It would technically be true for
| the headline to say only "AWS is having issues." A
| headline like that is going to rocket to the top of HN,
| and it's not going to provide many people with useful
| information.
|
| It's also silly. There's plenty of room in the headline
| to instead of "Cloudflare is having issues" to say
| "Cloudflare R2, Stream Live, and others are having
| issues."
| ChrisMarshallNY wrote:
| I'm downloading a large app update (MAMP Pro).
|
| It's screaming along at 47KBsec.
|
| I'll have it all in about two and a half hours.
|
| Not sure if that is related.
| arek_nawo wrote:
| I've been experiencing occasional slow-downs and issues on HN
| since last week. I would presume that it's due to more load
| (with WWDC last week and now Reddit black-out) but I don't
| really see this fully-reflected in the comment or up-vote count
| on the main page (except the recent Vision Pro thread).
| [deleted]
| teaearlgraycold wrote:
| An ex admin of Reddit thinks the protest broke the internal
| cache layer performance:
| https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...
| jrockway wrote:
| That was my guess, not knowing anything about Reddit's
| architecture. The "this subreddit is private" message
| probably used to be really rare, so it probably does the auth
| check for every request. Now every Google result links to
| "this subreddit is private", and so traffic to that endpoint
| went up by orders of magnitude. The result is an outage.
|
| If I were an SRE at Reddit, the day I got wind of the "we're
| making all subreddits private" thing, I'd double check that
| code-path to see what we were in for on Day 0. However, I am
| not.
| dylan604 wrote:
| Or maybe, the SREs at Reddit decided this is their
| contribution to the protest to let it burn
| jrockway wrote:
| In this economy?
| dylan604 wrote:
| They don't have to let it burn to the ground, but they
| also don't have to bust their asses to allow a point to
| be made.
| fragmede wrote:
| At their paygrade?
|
| Looking at Meta/Google/Apple's layoff packages of 6
| months of severance, and factoring in $170k-$389k annual
| income at Reddit (per levels.fyi), I would hope they have
| enough savings to live off of for months if not years, to
| enable thtem to protest, should they so desire.
| datavirtue wrote:
| Humans spend the money they have available. All of it.
| rat9988 wrote:
| at this income level, it's not worth losing my job in
| this economy, not even in a booming economy.
| DaiPlusPlus wrote:
| 6 months severance is exceptional - 3 months is what you
| get if you've been there at least a few years and in
| good-standing.
|
| And the twentysomething kids on $200k on the west coast
| won't be saving that money (if the number of Teslas on
| the road in Redmond is anything to go by): they have no
| reason to believe they won't make the same kind of TC at
| the next job they apply for.
| EamonnMR wrote:
| Whoa tildes are a reddit alternative? I thought it was all
| about, like, public access unix systems.
| progval wrote:
| tildes.net seems completely unrelated to the ones you are
| thinking of (the tildeverse)
| EamonnMR wrote:
| Ah, thanks.
| DiggyJohnson wrote:
| Is there a known feasible way of receiving an invite at the
| present time?
| teaearlgraycold wrote:
| Huh?
|
| It's not quite an alternative. It's good though and I'd
| recommend joining once there's another invite wave.
| dom96 wrote:
| Anyone have an invite to share for Tildes?
| ecliptik wrote:
| Same here. Looking at the list of topics, it hits a lot of
| my interests and isn't 100% focused on tech. My contact
| details are on my website in my profile.
| teaearlgraycold wrote:
| Sent!
| teaearlgraycold wrote:
| Looks like the email address on your keybase page is
| unreachable?
| intelVISA wrote:
| Would love one as well.
| teaearlgraycold wrote:
| Sent!
| buildbot wrote:
| Would love one too...
| teaearlgraycold wrote:
| Can I get your email address?
| buildbot wrote:
| tilde at maxg.io :)
| anikom15 wrote:
| This is at the top of HN because Cloudflare is not delivering a
| product people pay them for.
| capableweb wrote:
| I suppose this is at the top of HN as CloudFlare is quickly
| become a centralized point of failure for big parts of the
| internet, even without counting the CDN, so large swaths are
| affected regardless if it's just a "minor" outage or not. Hence
| it's interesting enough to land at the top.
| cyanydeez wrote:
| BUT THE INTERNET IS SUPPOSE TO ROUTE AROUND TTHIS STUFF.
|
| Remember when crypto promised a decentralized utopia of
| distributed systems that any interference wouldnt work, but
| slowly but surely it all congregated into a couple of
| oligarchical companies?
|
| Surely by now the technology of the internet has matured
| enough that these conglomerates are going to do the same but
| with a bit less fraud involved.
| supriyo-biswas wrote:
| The internet does work around issues at the inter-
| networking layer through BGP and similar protocols, though
| the same resilience is sadly absent at the higher layers.
| dangerlibrary wrote:
| https://en.wikipedia.org/wiki/Border_Gateway_Protocol
|
| BGP was standardized in 1989 and has been in use since
| 1994. The technology of the internet has matured in many
| ways, and remains almost identical in many others.
| Sometimes Microsoft is right, and backwards compatibility
| is the killer feature.
| sph wrote:
| WTF has crypto got to do with centralisation of the
| Internet? The Internet is well-decentralised, but if every
| Joe Blogg _has to_ put their website behind Cloudflare, it
| 's not the Internet's fault, let alone crypto (?).
|
| Stop putting every-bloody-thing behind Cloudflare, and the
| problem solves itself. I don't know whether to laugh or cry
| when I read of someone on HN seriously saying that they
| need a CDN for their personal website, or that they really
| need to use AWS or GCP for that matter. We tech workers
| have lost the plot, and it's our fault it's all in the
| hands of the few.
| goodpoint wrote:
| I think you are confusing different decades. The early
| Internet was way more decentralized and democratic.
|
| Cryptocurrencies promised many things but only wanted to
| replace oligarchical companies with oligarchical miners.
| sophacles wrote:
| I think they're confusing layers. The internet is more
| than just http servers. The IP traffic is routed around
| problems all the time. If the HTTP server is down, that
| doesn't stop the packets from arriving there (having been
| routed around dozens of down links and broken routers at
| any given moment).
| NicoJuicy wrote:
| There's also the perception issue that if the host fails, it
| most likely isn't cloudflare although cloudflare gives a
| warning.
|
| Note: experienced it from the first row, cloud had an issue
| in a region with SSL...
| [deleted]
| MichaelZuo wrote:
| Is there some sort of backup/failsafe mechanism for sites
| that use Cloudflare?
| viraptor wrote:
| Depends how badly they fail. To take the full advantage of
| CF you need to keep your DNS with them. That means if you
| can't configure any changes, you can't quickly move to
| another provider either. Your only solution at that point
| is to transfer the whole domain, but that may also require
| CF's assistance. (Unless they don't handle your apex
| domain)
|
| Effectively unless they're down for more than a day, you're
| better off taking the hit and waiting for them to resolve
| everything.
| ta1243 wrote:
| > Effectively unless they're down for more than a day,
| you're better off taking the hit and waiting for them to
| resolve everything.
|
| lol. There's me worrying about a 2 minute downtime once
| every 10 years
| remram wrote:
| To use Cloudflare, you switch your domain's DNS server to
| them. If that broke, you probably can't switch out without
| incurring the usual DNS propagation delay on your
| nameserver.
|
| If DNS is up, you can change specific records to point to
| your origin or another provider instead of the proxy/CDN,
| provided that it can handle the load and doesn't need a
| setup similar to Cloudflare (i.e. repointing your
| nameserver).
| ebcase wrote:
| Do they support Secondary DNS, e.g.
| https://support.dnsimple.com/articles/secondary-dns/
|
| ?
| dahfizz wrote:
| Anecdotally: op guessed right - I was only interested because
| of the HN and reddit outages
| robertlagrant wrote:
| I wouldn't say it's a centralised point of failure. Many
| individual sites choose to use it, so it is used by big parts
| of the internet, but that doesn't make it centralised. Maybe
| a single point of failure.
| efdee wrote:
| It's impossible for a single point of failure not to be
| centralised. Otherwise it wouldn't be a _single_ point of
| failure.
| robertlagrant wrote:
| Of course it is. If a million websites are built with
| Flask then a bug in Flask affects all of them. If a
| million websites individually decide to use Cloudflare
| then Cloudflare affects them all.
|
| But that doesn't mean Cloudflare is built in a
| centralised way (I assume it isn't), nor that there's
| something about the internet that is centralised around
| Cloudflare. Rather, people are choosing to include it as
| a dependency in their stack.
| efdee wrote:
| Using a piece of software is not the same as using an
| online service.
|
| If all those websites are using Flask, that would be the
| equivalent of centralising on Flask. The opposite of that
| would be many Flask-compatible yet unrelated other
| frameworks being used in parallel. A bug in Flask would
| not affect those not using that very codebase.
|
| The centralisation people are speaking of here is the
| amount of people all putting their eggs in Cloudflare's
| basket.
| robertlagrant wrote:
| > If all those websites are using Flask, that would be
| the equivalent of centralising on Flask
|
| I agree on the equivalence between this and Cloudflare
| use, but not that this should be described as
| centralisation, which is a particularly potent word on
| the open web. Popularity isn't the same as
| centralisation. Each website could rewrite to remove
| Flask if they wanted.
|
| Another example: if lots of people watch Squid Game, that
| doesn't indicate a centralisation of television
| programmes. No options are removed. People are just
| choosing to do a similar thing, but not in a way that
| centralises anything.
| p4bl0 wrote:
| In your example Flask would be a single point of failure
| and could indeed be called decentralized. For example
| websites can individually patch their Flask installation
| and come back up one by one, without depending on anyone
| else (at least once Flask is fixed).
|
| Here with Cloudflare, a single entity is responsible for
| the fix and will more or less fix the failure for every
| sites that uses it at the same time, by fixing it on
| their side. And website individually cannot do anything
| on their own.
|
| So I would argue it makes sense to call that centralized,
| at least from a structural/operational perspective.
|
| It is at the very least a form of _contraction_ of the
| network.
| robertlagrant wrote:
| From an operational perspective it's just a supplier like
| any other. Lots of suppliers are involved in a business's
| website. That doesn't make it centralised, though?
|
| > It is at the very least a form of contraction of the
| network.
|
| I think it's that at _most_. No one has to use them, as
| they accelerate / enhance open protocols. That is the
| least lock-in one could hope for, so they don't contract
| anything in a negative way.
|
| Contrast with, say, Etsy/Shopify, who actively try to
| replace the open space with closed ones.
| starttoaster wrote:
| It seems like you two are arguing different perspectives
| on the word centralization without conceding that a term
| can be relative to a viewpoint.
| robertlagrant wrote:
| But every word can be relative to that? I don't see where
| we'd go with that. I do think it's how the word relates
| to the topic that's interesting, if that's what you mean,
| but I think we are trying to get at that (-:
| pixl97 wrote:
| >It's impossible for a single point of failure not to be
| centralised.
|
| Is a monoculture centralized? It doesn't have to be, and
| yet it can all fail due to a single fault.
| activiation wrote:
| There are so many websites using it that it is basically
| the same as being centralized.
| sgt wrote:
| So many sites using Flask across the web is centralized?
| r4indeer wrote:
| You're comparing apples to oranges. Flask is software of
| which all instances run independent of each other while
| Cloudflare is SaaS where all its sites depend on the same
| service.
|
| If Cloudflare goes down, so does a significant portion of
| the web. Flask cannot have an outage like Cloudflare. All
| they can do is push a faulty update, but even then you
| can rollback or stay on the old version.
| tetha wrote:
| > If Cloudflare goes down, so does a significant portion
| of the web. Flask cannot have an outage like Cloudflare.
| All they can do is push a faulty update, but even then
| you can rollback or stay on the old version.
|
| This is what was on the tip of my tongue. I'd argue
| you're missing the portion of control in this whole
| discussion, and how much process you can place in front
| of a component change.
|
| If I have a flask dependency, I have a lot of control
| over this dependency. If flask screws up badly, I have
| many options: I can update. I can not update. I can
| downgrade. In fact, I could fork flask internally and fix
| it on my own and either be a good citizen and open up a
| PR, or I could be something else. I can test all of this
| in any number of environments before it hits a customer,
| and even more stages before it hits all customers.
|
| With Cloudflare - or any number of Hosters as well, like
| AWS, Azure, Google Cloud, I have very little control. If
| I use Cloudflare as a CDN and Cloudflare goes down, I
| might not have the capacities at my upstream server to
| handle the load from all my customers, so I am down as
| long as Cloudflare is down. I wouldn't have the footprint
| available necessary to replace AWS in our own private
| DCs, even if we pooled all spare capacities - and then
| I'd still have to find a way to exfil our data from a
| downed AWS. (Which, yes, we have, but it'll take long
| hours)
|
| And no matter how much I test, if my hoster fucks up, I'm
| immediately fucked as well, no matter what processes I
| might have. The only process around this would be
| provider independence, which is really expensive and a
| lot of effort even if you just have a luke-warm standby.
| robertlagrant wrote:
| You can multihost your static site; you don't have to
| only use Cloudflare. You can have more than one CDN.
| Nothing is centralising you or funnelling you somewhere
| you don't want to go.
|
| And you can do these things in parallel, unlike the Flask
| example, which you pretty much have to commit to using
| solely.
| activiation wrote:
| If most sites were using Flask and forced to do broken
| updates, it would be a centralized problem.
| senttoschool wrote:
| Possibly third parties that Reddit/HN rely on are using
| Cloudflare?
| BiteCode_dev wrote:
| True.
|
| But if they fetch a lot of URLs from cloudflare and it takes 30
| seconds to answer with a timeout instead of a http/200 under
| 20ms, then some architectural decisions that were sound under
| the latter case may make the whole system slow in the former
| one.
| lanstin wrote:
| People have started testing network failures but so often
| fail to test slow network failures. All of a sudden your code
| has 500x more pending open sockets, both in and out, and the
| memory spikes and it all goes to hell. Even if the fast fail
| code path is indeed best effort.
| mysterydip wrote:
| Basically the old "I Love Lucy" chocolate factory sketch:
| https://youtu.be/AnHiAWlrYQc
| bombcar wrote:
| I would love to see graphs showing "length of reply that
| eventually succeeded" - I suspect that most networks today,
| if you don't get a response in 5 seconds, you ain't never
| gonna get anything useful.
|
| In other words, I wonder if going to _fail fast_ would help
| the health of the Internet more than _wait forever_
| timeouts. Might reduce DDoS effects, too.
| BiteCode_dev wrote:
| Well a proper system should reduce the timeout for the
| same domain, the more it gets raised, and set it back to
| default once stats are good again.
|
| But it's very complicated and costly to setup, so almost
| nobody does this.
| toast0 wrote:
| If you're building complicated systems, you should
| probably reduce traffic to sites that are failing to
| respond, rather than continuing to send traffic, but
| timeout faster. Depending on what stage in the process
| the request is failing, it might not make a big
| difference (in a typical HTTPS exchange, the costs ramp
| the farther you go, processing a syn < processing a
| client hello < processing a complex request. (If it's a
| simple request, processing the client hello is more
| expensive than the request, of course).
|
| If you send all the same traffic, and probably more
| because of retries with shorter and shorter timeouts,
| chances are you are going to keep the system overloaded,
| never detect success and never return to default
| timeouts. Dropping most of the traffic, and then turning
| it back on when the system recovers can lead to
| oscillation where the system works enough to drive more
| traffic that overloads the system etc, but at least
| you're getting some processing done.
| BiteCode_dev wrote:
| Well, you do need, jitter, exponential backoff, caching,
| black and white listing, a stat base decision tree, etc.
| That's why it's a complicated and costly problem.
|
| But if you are consuming a lot of API content, if you
| have crawlers or if you provide features like "get
| article title/summary/image/thumbs", at some scale it's
| an important decision to make.
| dylan604 wrote:
| >I suspect that most networks today,
|
| I guess that comes with the type of end point in that
| network. A typical website, absolutely, yes, I'd agree.
| An API end point allowing requests of large data pools
| that might take a few seconds to generate but yet not a
| total time out would be acceptable.
| Aardwolf wrote:
| Is HN slow because everyone who was normally posting on reddit
| is now commenting on HN instead?
| starttoaster wrote:
| Time to upgrade from a t2.Micro to a t3.Medium, I suppose.
| doxeddaily wrote:
| I think so. I'm trying to move my time over here as much as
| possible.
| DoctorOW wrote:
| Same here. Even the subreddits that are still open, I'd
| feel dirty reading.
| _zaphod wrote:
| [dead]
| lagniappe wrote:
| Off topic: Cloudflare, like Google, is bad for the internet- not
| just for the way they centralize the internet but for the way
| they begin to conflate themselves _with_ The Internet. Very few
| sites need Cloudflare, and of those with it equipped, an even
| fewer number configure it or use it correctly for their
| application.
| BiteCode_dev wrote:
| I run several services without cloudflare, but it's so
| tempting.
|
| Their product is excellent, and so easy to use. You push a
| button, and suddenly a whole class of difficult problems
| disappears.
|
| It's very hard to fight against the urge to use it.
| waboremo wrote:
| This in combination with the above is a huge issue, in which
| people are too concerned about "don't use Cloudflare" angle
| that they don't spend any effort into creating alternative
| solutions for the problems that exist.
|
| We can dismiss the problems all we want by saying "most
| people don't need it", but if they're using it, clearly they
| see a need and this has to be addressed not ignored. Which is
| why it's so hard to fight against the urge, so go ahead and
| use it if it saves you time and money. Until real
| alternatives exist, don't hinder yourself.
| blibble wrote:
| > You push a button, and suddenly a whole class of problems
| disappears.
|
| and a whole new class of problems appears
|
| (my regular cloudflare outage is when they decide to add new
| "security" that the existing "disable security" page rules
| don't apply to, great for API consumers)
| judge2020 wrote:
| Sites don't need cloudflare, unless:
|
| - They want to save on bandwidth costs
|
| - They have to deal with some level of DDOS or site scrapers
| hitting every page at once
|
| - They want to block IPs, geographies, ASNs, etc. without
| editing a server config
|
| - They want speed & server-sided visitor analytics, email
| routing, security settings, redirect configuration, and DNS all
| in one dashboard
| datavirtue wrote:
| With the cost of popular cloud vendors, Cloudflare is
| essential. The first bullet points is enough to drive people
| there.
|
| You can also manage all of your internet infrastructure
| (domains, dns, etc) under a single dashboard that doesn't
| suck. Pry it from my cold dead fingers.
| tomwheeler wrote:
| You left out this one:
|
| - They want their vendor to arbitrarily deny access to
| customers and prospects
|
| Maybe they have a 100% success rate at blocking actual
| threats but they sure do have a lot of false positives. I get
| blocked or forced into captchas at least three times per
| week.
|
| I used to report it to the site admins, but among the few
| that responded, almost none knew how to fix it. I no longer
| bother, so I suspect that the less clueless admins have no
| idea how many visitors Cloudflare has driven away.
| supriyo-biswas wrote:
| Absolutely. I had a post here about how Cloudflare is
| locking out Linux users[1] (which was momentarily resolved
| and they're back to blocking them again), but it was
| quickly flagged, suppressing any real discussion about it.
|
| I have to assume jgrahamc was one of the flaggers, given
| their indignant comment at the top of this thread.
|
| [1] https://news.ycombinator.com/item?id=36197401
| burnished wrote:
| Could it have been the paragraph you removed for being in
| bad taste?
| adammartinetti wrote:
| Hi there! I work at Cloudflare. Our global performance
| for Linux users on challenge pages looks good at the
| moment, but I'd love to take a closer look.
|
| Could you send me an email with details you have
| available, (rayID, IP address + website, or HAR file) at
| amartinetti at cloudflare.com?
| johnklos wrote:
| How about you actually look at your own logs on your own
| servers, rather than expecting people you exclude from
| doing your homework for you?
| doctor_radium wrote:
| See my comments below. Would appreciate you whitelisting
| the Opera Mini browser globally. I'll then be happy to
| take back some of the things I said.
| neilv wrote:
| I've had a couple HN comment threads recently, about
| Cloudflare blocking me, and I contacted multiple sites
| about this, but so far no improvement.
|
| Maybe the right people are hearing, but they consider it
| a necessary evil of negligible impact (accurately or
| inaccurately). Or is stuck with Cloudflare, who says
| they'll take care of it. Or maybe the message is getting
| lost before it gets to someone who cares.
| bamfly wrote:
| > I used to report it to the site admins, but among the few
| that responded, almost none knew how to fix it. I no longer
| bother, so I suspect that the less clueless admins have no
| idea how many visitors Cloudflare has driven away.
|
| It's an acceptable cost, versus other ways of dealing with
| abusive traffic.
|
| Hell, we (sysadmins, back when that was a term people used)
| used to just blackhole all the IP blocks associated with
| certain countries--made the logs _so very much_ quieter,
| and this was back when the _whole Internet_ was a lot
| quieter to begin with. At least CloudFlare 's less blunt
| than that.
|
| Failing to block abusive traffic can be really expensive.
| Detecting it is always going to cause false positives.
| Admins are OK with those as long as they don't cost more
| than implementing a system with fewer false positives
| would. A half-percent tax on revenue (to pick a number out
| of a hat) in the form of lost customers is a reasonable
| trade-off for a lot of companies. You've got to have pretty
| serious scale before it's worth investing real money to try
| to shave that down by a couple tenths of a percent (you'll
| never get it to zero, and only places like Amazon have the
| kind of scale that make it worth attempting to closely
| approach zero)
| fzeindl wrote:
| > - They want their vendor to arbitrarily deny access to
| customers and prospects
|
| You can turn this behavior off as site owner.
| doctor_radium wrote:
| TL;DR: Also off topic.
|
| 20 years ago I was beta testing a browser nobody's heard of
| and happened upon a mainstream PC accessories seller like
| pcconnection.com or the original cdw.com with a poorly
| designed site: malformed cookies, important information
| hidden inside nonstandard tooltips, etc. Don't recall their
| name, but they had domains for the US and Canada. I emailed
| them at least twice to point this out, but they blew me
| off. I was amused when they went out of business a few
| years later.
|
| When you criticise Cloudflare, the common response is that
| the site owner has chosen to block this or that. I don't
| believe for a moment that Cloudflare offers a checkbox for
| "Spin the busy animation forever, never loading the site
| and never generating an error". This is what happens when a
| company dubs itself the Internet Police while not knowing
| what the f*ck they're doing. In the words of the Joshua AI
| from Wargames
|
| "The only winning move is not to play."
|
| I suggest that it's impossible to do what Cloudflare is
| attempting without false positives, but they haven't
| figured that out yet. And when you complain in their public
| forum, do they apologize and send up the flares? No, you're
| met with arrogance:
|
| https://community.cloudflare.com/t/browser-integrity-
| check-b...
|
| I wasn't caught in this particular dragnet, but had the
| same thing happen to me this past winter with the Iceraven
| browser. I did complain to one of the affected sites, who
| weren't particularly helpful, and the issue disappeared on
| its own about 2 months later. Maybe somebody did start a
| public thread with Cloudflare, but knowing the likely
| response, I held off restarting that fight.
|
| I use the Opera Mini browser sometimes (not recommended),
| which apparently has its worldwide server/endpoint in the
| Netherlands. There used to be a US endpoint but no longer.
| If you're using Cloudflare to block European traffic for
| one reason or another, okay, I get it. If you're using
| Cloudflare to block "unusual traffic" like VPN endpoints
| (no idea how it actually works), it would take them two
| minutes to do a reverse lookup on the IP and see
|
| 109.211.145.82.in-addr.arpa. 13531 IN PTR h18-05-12.opera-
| mini.net
|
| and simply whitelist the IP globally. But they
| haven't...while a little voice nags at me, reminding me
| this is supposed to be their day job.
|
| On some level I'm okay with the status quo and letting my
| first paragraph scenario play out: smart businesses grow
| while idiots go bankrupt. With Chrome, Safari, and Edge
| controlling roughly 90% of the browser market, though, I'm
| not holding my breath. The rest of me considers this
| behavior anti-competitive and a growing civil rights issue.
| Imagine needing a toll road or ferry service to get
| somewhere but the owner tells you they don't like your car
| and please get another one. The average person would
| exclaim, "What? There's nothing wrong with my car!!" Ditto
| for whatever browser I choose. If it supports TLS 1.3 but
| doesn't like the web site's HTML/JavaScript/CSS, then maybe
| I'll consider switching...or maybe I won't. But that should
| be my choice, not theirs, and definitely not the choice of
| a middleman with delusions of grandeur. Web sites designed
| for use by the public need to be accessible by the public.
| yjftsjthsd-h wrote:
| Personally, my favorite was CF emailing me to brag about
| how many robots/malicious actors they block, right after
| blocking me for being a robot/malicious. Like... that's
| sure one way to undermine your numbers...
| RoyGBivCap wrote:
| > _I get blocked or forced into captchas at least three
| times per week._
|
| Same.
|
| I get blocked on websites I have accounts on if I use a
| VPN. I'm not just a visitor, I'm a member, and still get
| blocked by cloudflare. Other times it's a small or local
| business and to me the website just looks _down_. Then, if
| I bother to think about _and_ willing to drop my VPN,
| suddenly the website works fine.
|
| Treating VPN users as hostile is getting really fuckin old.
| jabart wrote:
| VPN users are hostile just not you. VPN is abused by bad
| actors all the time using a shared IP that you are likely
| using too. If you had a dedicated IP I'm sure this would
| be different though makes you less anonymous which is
| part of the reason why you get captchas.
| fullspectrumdev wrote:
| > Maybe they have a 100% success rate at blocking actual
| threats
|
| They don't.
|
| There's a new, working CF WAF bypass published almost every
| day in the bug bounty hunter circles.
| richardw wrote:
| In the beginning of a project I want velocity and progress.
| CF gives me:
|
| Easy SSL. Register domain at cost. Throw up static site on S3
| equivalent (or app on whichever cloud), add Cloudflare and a
| couple redirects (no www to www, no ssl to ssl), done. It's
| like gmail for simple web hosting. It's also like early
| Google, friendly and useful and not trying to squeeze every
| dollar for shareholders and bonuses.
|
| They lend us their very competent team at free or minimal
| cost. I totally get the single point of failure and monopoly
| but compared to AWS, Azure and Google complexity it's a
| simple web app dream until I have time to care about certs
| etc.
|
| Competitors need to build an easy alternative and get as good
| as the CF team. Which is way better than me at futzing with
| config I only touch once every couple years.
| ricardo81 wrote:
| It's also popular in the SEO crowd for artificial link
| building. With such a chunk of the web behind Cloudflare, the
| presumption is that it's one less fingerprint that is
| reasonably scalable.
|
| For the most part, the bandwidth doesn't matter, the sites
| are made for bots.
| mschuster91 wrote:
| > - They have to deal with some level of DDOS or site
| scrapers hitting every page at once
|
| Sadly, that is the norm with _everything_ since Shodan made
| it trivially easy to find out targets for websites. Once your
| hostname or IP ends up there, you _will_ get blasted with
| exploits a couple minutes after a 0-day was published.
| fullspectrumdev wrote:
| I think blaming shodan/etc is incorrect - shodans rise in
| popularity coincided with the rise of cheap, fast bandwidth
| at VPS providers enabling you to scan real fuckin fast.
|
| It's much more efficient to just do "zmap | ./exp" than it
| is to query shodan, get a limited amount of targets, etc.
| pessimizer wrote:
| And Cloudflare seems to be increasingly denying access to sites
| from browsers with spoofed or hidden referers, which is an
| attack on individual privacy.
| fzeindl wrote:
| You can turn this behaviour off, it's called "Browser
| integrity check".
| pessimizer wrote:
| As a site owner, but not as a site visitor. And without
| cloudflare, the site owner wouldn't even have the ability
| to do this.
|
| It also seems to be something they've moved to flipping on
| by default.
| digitallyfree wrote:
| I don't like the centralization of Cloudflare, but it's the
| only way I can host public web pages out of my homelab with a
| DSL connection. I would rather have a CDN than put my content
| on a VPS as the servers remain 100% under my control and I can
| change providers anytime.
| mrits wrote:
| Very few people need seat belts. But if you don't have one on
| when you do you are going to have a bad time.
| AlexAndScripts wrote:
| Cloudflare lets me serve dozens of sites from a single pi 3 on
| 10mbit Internet. Sure, I could get a VPS, but I save a lot of
| money (I'm 18) this way. The only thing clients hit me for is
| the actual HTML.
| arek_nawo wrote:
| No site NEEDS Cloudflare, but many can benefit from their slew
| of products. From their CDN to DDoS protection, Workers, and
| even basic things like DNS management - they provide a really
| good user experience across the board. Sure, it adds another
| potential point of failure to the system, but for many - it's a
| worthy trade-off.
| 22c wrote:
| CloudFlare free-tier is incredibly useful for sticking in front
| of the cheapest bargain basement VPS WordPress host you can
| find and miraculously still ending up with a somewhat
| functional small-business website.
|
| It'd be even better if there were competition in this space,
| but there aren't too many options outside of cloud providers
| who are likely relying on the fact you might accidentally slip
| up one day so they can charge your credit card.
| jbotdev wrote:
| Fortunately there are so many cheap options nowadays for
| fully managed small-business websites which include all the
| CDN/WAF stuff for you (Squarespace, managed WordPress, etc.).
| The setup/maintenance time for running your own web server
| doesn't make sense unless you already have an "IT" person on
| the payroll.
| sph wrote:
| With all the politeness and calm I can muster: you do not
| need Cloudflare for your small-business website.
|
| Again: _you do not need Cloudflare for your small-business
| website._
|
| Do you have any idea of how many requests can a shitty
| unoptimised website serve on commodity hardware? Judging by
| comments like yours, which seem to be the majority, I wonder
| if anyone with less that 15 years of experience is still able
| to write a website serving 10k users a day (1 request every 8
| seconds) on a 2 core VPS without needing a CDN.
|
| Let me spoil the black magic only greybeard seem to know:
| STOP. OVERENGINEERING. You don't need Cloudflare. No one
| cares about DDosing your website, you're not Reddit for
| Heaven's sake.
|
| If you overengineer, at least quit all rushing to give your
| custom to the same company, making Cloudflare a de facto
| monopoly.
| bombcar wrote:
| People stick cloud flare on the front less for the CDN/DDoS
| protection and more for the web application firewall,
| because their small business website is almost certainly
| some incredibly out of date Wordpress install with more
| vulnerabilities than you can shake a stick at.
|
| Wordpress should integrate one of those "Wordpress to
| static site" plugins as the _default_ because that 's all
| 80% of the users need.
| rs999gti wrote:
| > STOP. OVERENGINEERING.
|
| Please tell this to every junior/cloud developer/architect
| when it comes to microservices.
| sophacles wrote:
| On the other side of the coin - launching an attack that
| would overwhelm that vps has never been easier. Even if
| you're serving static pages and your 2 cores can fill the
| 10G nic with compute to spare, the pipe can be filled with
| enough inbound garbage that legit requests have their
| packets dropped. This isn't that hard in 2023. There are
| dozens of booter services for hire, and if you really are
| scraping the bottom of the barrel of hosting, some pissed
| off kid can probably convince their discord buddies to
| generate enough traffic to knock your site over.
| datavirtue wrote:
| Our company was knocked offline by a DDOS attack. It
| continued all day unabated until we signed up for a free
| cloudflare account. That was easy.
|
| Flash forward about 8 years and I find myself using
| cloudflare to stave off the immense suffering of Azure/AWS
| by using R2 and Pages. As soon as more people find out how
| easy Cloudflare is and how much it can lube a product
| deployment, I expect a lot more of the internet to end up
| there by choice.
|
| Cloudflare deployed my app better/easier than Microsoft was
| able to deploy it to Azure (Microsoft owns GitHub and Azure
| and still didn't have their shit ironed out...Cloudflare
| just works).
|
| I had to tweak them both but Azure took hours and
| cloudflare was a quick, cleanly documented change.
|
| I wish they offered a service to host my nodejs APIs.
| mattferderer wrote:
| I like Azure, GitHub, AWS for a lot of things but
| Cloudflare has it's niches that are just better.
|
| I haven't read into why Open AI uses Cloudflare instead
| of Azure services but I find it very interesting
| considering their Microsoft arrangement.
| francisofascii wrote:
| Using a CDN is actually the opposite of overengineering.
| Using a CDN is an easy, low cost way to outsource any
| performance concerns. Overengineering is trying to handle
| all the potential performance issues yourself, with custom
| caching, database indexes, code optimization, better
| hardware, etc. For many sites it is simply an insurance
| policy. Most drivers don't need auto insurance, until that
| one day you do.
| mattferderer wrote:
| Exactly this. The amount of cost & time to make a
| (WordPress/Drupal/random database backed CMS website)
| site run well on a cheap VM is significantly higher than
| Cloudflare's free & paid options. Those aren't my first
| choice in CMS these days but a lot of the small business
| web still run them & migrating isn't cheap from a
| technical or user training perspective.
|
| If you happen to have a popular CMS like WordPress on a
| cheap VM, odds are you are going to get DDoS all the
| time, even if you only have 100 legit views/day.
| Cloudflare will reduce this dramatically.
|
| I would guess any site not using Cloudflare (or someone
| similar) is more likely overengineering. As always
| though, every case is unique & it depends.
| johnklos wrote:
| Absolutely bull poop.
|
| First, setting up anything using some third party service
| like Cloudflare is already too much work and doesn't even
| work for many people in parts of the world Cloudflare has
| determined are undesirable.
|
| Second, I can, have and do host popular CMSes on hardware
| much more modest than Raspberry Pi performance.
|
| Third, "odds are you are going to get DDoS all the time"?
| Are you a Cloudflare shill? This is nothing but wildly
| hyperbolic. In a quarter of a century of hosting, I've
| had to deal with one specific DDoS actor. One.
|
| How are you going to claim that "odds are you are going
| to get DDoS all the time"? Go ahead, provide evidence,
| although I'm sure you can't and won't.
| capableweb wrote:
| Well, you say that a large part of the internet are using
| Cloudflare then by that measure, they are The Internet,
| together with all the rest of the services who are also The
| Internet.
| c00lio wrote:
| "AOL is The Internet"
|
| sorry, just had a flashback...
| ljm wrote:
| As far as I remember it, Cloudflare was one of the first
| services to offer TLS on their free tier which bought an
| incredible amount of good will.
|
| Between Cloudflare and LetsEncrypt I've never had to pay for an
| SSL certificate again.
|
| Of courss, the easy to use interface was also welcome. Using
| their CDN for small scale or hobby projects is total overkill
| though.
| jgrahamc wrote:
| We were. September 2014:
| https://blog.cloudflare.com/introducing-universal-ssl/
| a_subsystem wrote:
| Where else can I set up a DNS proxy and block all but the
| countries I want within about 15 minutes for free?
| dxuh wrote:
| I host a very small blog and I am tempted to use Cloudflare,
| because it seems latency is very bad from some geographic
| regions. And I would guess every website wants reasonable
| latency from a large part of the world, right?
| bombcar wrote:
| Latency matters much less than people think in many cases;
| especially if it's mostly a static site.
|
| Page load times are almost always dominated by the megabytes
| of images and javascript that everyone seems to include now.
| casenmgreen wrote:
| I'm sorry they're having trouble, but speaking just for myself,
| perhaps this means I will be able for a short time to view a very
| considerable number of web-sites which normally are inaccessible
| behind a permanent loop of "we must check the security of your
| connection".
| AmenBreak wrote:
| [flagged]
| klysm wrote:
| I'm always excited when things go down like this because that
| means I get to read another postmortem
| ck2 wrote:
| If a foreign power wanted to amplify chaos online, this seems
| like an easy pre-scheduled time to do it?
|
| But more likely just kids with endless hacked IoT available to
| DDoS whatever they feel like.
|
| Before IPv6 there were "internet weather" services, are they
| still around?
| taubek wrote:
| Is this the reason for access issues to Hacker News at the
| moment?
| marginalia_nu wrote:
| HN's slowness is probably knock-on result of Reddit's outage.
| It's where a lot of people to go to discuss the Reddit shit-
| tornado.
| LinuxBender wrote:
| Probably not, at least not directly. HN is hosted on M5
| Computer Security at the moment and it's DNS is AWS. The load
| issues usually happen when a few popular threads have a really
| high comment count and dang usually has to enable pagination or
| other things to compensate.
| MuffinFlavored wrote:
| > HN is hosted on M5 Computer Security
|
| Do we know if it is just like, one tall MySQL instance and a
| few RubyOnRails API instances? Not sure what would be used
| for "caching" as well.
|
| > The load issues usually happen when a few popular threads
| have a really high comment count and dang usually has to
| enable pagination or other things to compensate.
|
| Why not auto-turn it on at like 500 comments or so?
| LinuxBender wrote:
| _Do we know if it is just like, one tall MySQL instance and
| a few RubyOnRails API instances?_
|
| Two servers, one active and one standby. Both running Arc
| on BSD. No MySQL or Ruby.
|
| _Why not auto-turn it on at like 500 comments or so?_
|
| Best to ask dang directly on that one _hn@ycombinator.com_
| as I have never seen the code and responsible adults don 't
| let me near their code.
| nprateem wrote:
| It always surprises me that a VC that has backed multiple
| tech unicorns can't run a scalable web site.
| fathyb wrote:
| I always felt like scalability and growth aren't goals
| here, which is something I like to be honest.
| nprateem wrote:
| This must be a major funnel for them. The fact they can't
| even run a robust text-only website ought to be an
| embarrassment, but here we are. Every time there's a
| major tech story the site grinds to a halt until one guy
| jumps onto a server and ninjas it or whatever.
| Unprofessional to say the least.
| NicoJuicy wrote:
| It's actually pretty impressive that they are running
| this from one server in their own programming language
| nprateem wrote:
| Sure. Maybe they do the same as VCs. Use limited crappy
| resources in their own non-standard way to cripple
| growth.
| [deleted]
| Macha wrote:
| It's not like they can go grab employees from one of
| their funded companies to go work on HN, and they've
| clearly been happy with the results of their level of
| engineering investment in HN.
|
| (Besides, as Reddit is showing, do you want your
| discussion site run like a tech unicorn? Arguably things
| were better for users in the skeleton crew days)
| dotnet00 wrote:
| Considering all the trash that comes with heading in that
| direction (as has been repeatedly demonstrated by social
| media which has started off similarly to HN, only to turn
| around and shit on the people that helped them get
| going), I'm perfectly fine with a site that slows down a
| bit in special circumstances.
| anamexis wrote:
| HN is written in Arc, an experimental lisp dialect created
| by Paul Graham. I believe it runs on a single instance? Not
| sure though.
|
| I believe pagination does turn on automatically at 500
| comments.
| perihelions wrote:
| I think it's just too many Reddit refugees at once.
|
| PSA: if you browse HN in logged-out mode (e.g. a private tab),
| you can receive cached versions of HN threads that load much
| faster. The logged-in experience is currently worse than the
| logged-out: dynamic content generation for user-account stuff
| seems to be hitting a slow path on the server.
| travisjungroth wrote:
| This isn't the exact same, but there's a similarity in order-
| of-magnitude differences and flow rates: Most professional
| pilots work for the airlines. When you remove military, it's
| a huge majority. Most pilot jobs flow into the airlines. It's
| a pretty cohesive labor market.
|
| So, whatever market forces are going on for pilot labor in
| your little niche job are _completely dominated_ by the
| airlines. Like you'd think banner towing jobs would be mostly
| affected by advertising trends, but they're not. Passenger
| travel trends are way more important. (This is for the job
| market, not the business market)
|
| Similarly, the flow from a way bigger site can overwhelm
| whatever you have going on.
| chaikasumu wrote:
| I think that is pretty unlikely. HN doesn't use Cloudflare at
| all as far as I know, and not to downplay the scope of the
| outage, but Cloudflare's core CDN Offering is fine. Other core
| services such as Cloudflare Pages/Workers/KV are also fine.
| This is mostly addon/developer products, the biggest of which
| (imo) being R2. If you log out, it loads a lot faster (at least
| the main/cached pages)
| scrum-treats wrote:
| Looks like it is AWS. Given the fact that both Reddit and HN
| (and Apple, which is also experiencing issues) are on AWS, it
| looks suspicious.
|
| Also, new account creation in HN is broken at the moment.
| theanonymousone wrote:
| In Switzerland, the national rail company's online services were
| down in the morning and people could not buy tickets.
|
| Then news came out that MasterCard was also having problems
| during the day.
|
| And then Reddit, HackerNews (it was totally down for me, not just
| slow), and now Cloudflare!
| torvald wrote:
| Welcome to the future!
___________________________________________________________________
(page generated 2023-06-12 23:01 UTC)