[HN Gopher] Meta outage
___________________________________________________________________
Meta outage
Author : geocrasher
Score : 641 points
Date : 2024-03-05 15:27 UTC (7 hours ago)
(HTM) web link (metastatus.com)
(TXT) w3m dump (metastatus.com)
| jonnycomputer wrote:
| It logged me out and told me that my credentials were incorrect;
| I thought my credentials had been stolen, so I'm kinda personally
| glad that it seems to be happening to a lot of other people too.
| I know that's a bit selfish, but :shrug:
| MyFirstSass wrote:
| Same, thought all of my friends were getting spammed and i'll
| look like a "boomer" who got phished.
|
| Then i remembered i have a very long and secure password, then
| immediately panicked about someone having access to my Gmail.
|
| The sense of security is more brittle than i thought.
| sandspar wrote:
| On a psychological note, I think the threat detection part of
| our brain doesn't always notify our conscious thought that
| it's actively monitoring for threats. I've often noticed that
| when I'm carefully handling a hot frying pan then my ringing
| phone is more likely to startle me than usual.
| fuzzfactor wrote:
| When you've already got one threat on your hands you're
| less prepared for anything else.
| sandspar wrote:
| That makes sense. I've noticed too that my brain seems to
| have a threat pre-emption module as well as a threat
| reaction module. For example, I'll sometimes be walking
| and texting at the same time, only to stop in my tracks
| and suddenly realize that there's a hidden stair in front
| of me.
| 101008 wrote:
| I panicked the same. Even when I tried to recover my password,
| it said my email address wasn't associated to any account. I
| thought I lost it forever
| marcosdumay wrote:
| That's the natural endgame of the "user-facing services must
| not stop, if something they depend upon stops, they must only
| degrade" philosophy.
| lanstin wrote:
| I heard for a while Netflix would fail open if auth was
| unavailable. Like it's just movies just let em see it.
|
| Facebook data is more sensitive. Not so much the data
| people go there to see, cool memes that their friends
| liked, but the list of friends and interests.
|
| Other places I worked had the ability for Ops to push out a
| change saying the site was down for maintenance. After a
| while we stopped using it and just took the hit of a bunch
| of 5xx errors. Basically when the planned down times became
| shorter than the time to propagate the down setting.
| marcosdumay wrote:
| Failing open is maybe ok. Telling everybody on the world
| their account doesn't exist anymore isn't.
| kortex wrote:
| Same same. I went through the password reset flow (I was
| overdue anyways), it never sent anything to my SMS, so I did it
| again with email, reset the password and went to log in with
| the new password, "Incorrect password" error. Old password,
| also incorrect.
|
| Didn't help that I had just posted a lukewarm spicy take on how
| linguistic prescriptivism is BS.
|
| All the while the website felt like it was unstable, hard to
| describe, but it felt like it was bouncing around between URLs
| too much and reloading a lot.
|
| Definitely feels like a botched update on their end.
|
| E: Instagram is misbehaving as well, banner loads but big
| "Something is wrong" error on the feed.
|
| E: now youtube has "Something went wrong" - WTF. I can't
| believe I'm saying this, but thank goodness for reddit and
| X[itter]???
|
| E: interesting, seeing a big spike across multiple platforms on
| downdetector, including AWS:
| https://downdetector.com/status/aws-amazon-web-services/ I'm
| not able to log in right now, but that could be PEBCAK, I have
| too many saved IDs and I don't want to fail2ban myself
| pratnala wrote:
| The password reset flow was broken too. And I got logged out
| on every device. My friend who works there said they can't
| login either.
| cratermoon wrote:
| Discord is also having issues.
| samaritano wrote:
| downdetector reports has gone down but to me is still bugged
| out, been catching a livestream on youtube all along though,
| meta stocks are back up from the dip so I take it some
| regions are restored to normality
| NikolaNovak wrote:
| Yes. My spidey sense went off and I told my work I'll be off
| for an hour while I redo all my passwords... might still do
| that but glad to know it's not _necessarily_ me getting hacked.
| pratnala wrote:
| don't bother. fb's forgot password flow is broken too.
| nathanaldensr wrote:
| As was the account hijack process. It just loops.
| smcl wrote:
| I called out some comment for being racist a little earlier
| (yeah I know, just report and move on...) and figured they'd
| managed to pwn my account somehow. Good to know it's not just
| me.
| fishnchips wrote:
| Strictly speaking, just because there's an outage does not
| mean you're not pwn'ed.
| darkwater wrote:
| Maybe the outage happened because they used a 0-day to pwn
| smcl
| fishnchips wrote:
| Crazier things have happened.
| smcl wrote:
| In an "anything's possible" sense then yeah. But the fact
| that FB was not letting me login with the credentials I
| knew to be correct was directly attributed to a global
| outage, rather than a me-specific issue. Which I can now
| verify by checking the devices that are authorised to my
| account.
| suyash wrote:
| me too, also Instagram, could be that Facebook got hacked ?
| suyash wrote:
| Pro tip: in chase meta actually got hacked, it would be good
| idea to not use that password on any other websites, change
| them immediately.
| mfrommil wrote:
| A much better UX would be clear error messaging informing users
| that the service is down and there is no problem with their
| individual account.
|
| This would prevent people from panicking they've been hacked
| and/or unnecessarily resetting their password.
| eurekin wrote:
| There was for a brief moment. I got that once
| Kalium wrote:
| You are absolutely correct. That would be a _much_ better
| experience.
|
| That said, getting there strikes me as pretty challenging.
| Automatically detecting a down state is difficult and any
| detection is inevitably both error-prone and only works for
| things people have thought of to check for. The more complex
| the systems in question, the greater the odds of things going
| haywire. At Meta's scale, that is likely to be nearly a daily
| event.
|
| The obvious way to avoid those issues is a manual process.
| Problem there tends to be that the same service disruptions
| also tend to disrupt manual processes.
|
| So you're right, but also I strongly suspect it's a much more
| difficult problem than it sounds like on the surface.
| matsemann wrote:
| But there's something off here. I wouldn't expecting to be
| shown as logged out when the services are down. I'd expect
| calls to fail with something aka 500 and an error showing
| "something happen edited on our side". Not all the apps
| going haywire.
| Kalium wrote:
| At the scale of Meta, "down" is a nuanced concept. You
| are very unlikely to get every piece of functionality
| seizing up at once. What you are likely to get is _some_
| services ceasing to function and other services doing
| error-handling.
|
| For example, if the service that authenticates a user
| stops working but the service that shows the login form
| works, then you get a complex interaction. The resulting
| messaging - and thus user experience - depend entirely on
| how the login page service was coded to handle whatever
| failure the authentication service offered up. If that
| happens to be indistinguishable from a failure to
| authenticate due to incorrect credentials from the
| perspective of the login form service, well, here we are.
|
| At Meta's scale, there's likely quite a few underlying
| services. Which means we could be getting something a
| dozen or more complex interactions away from wherever the
| failures are happening.
| shkkmo wrote:
| > If that happens to be indistinguishable from a failure
| to authenticate due to incorrect credentials from the
| perspective of the login form service, well, here we are.
|
| If you can't distinguish those, then that is bad software
| design.
| sandspar wrote:
| Slightly unrelated question, but just how "Big" is Meta?
| I know it's vast, but as an outsider I have trouble
| grokking the scale of it.
| pixl97 wrote:
| When most people talk about serving thousands and maybe
| millions of requests per second, Meta talks about
| billions of requests per second.
|
| https://read.engineerscodex.com/p/how-facebook-scaled-
| memcac...
| jessriedel wrote:
| Isn't this just the standard problem of reporting useful
| error messages? Like, yes, there are academic situations
| where you can't distinguish between two possible error
| sources, but the vast majority of insufficiently
| informative error messages in the real world arise
| because low effort was applied to doing so.
| Kalium wrote:
| Yes and no.
|
| Yes, with the additions of sheer scale, a vast number of
| services, multiple layers, and the difficulty of defining
| "down" added in. I think the difficulty of reporting
| useful error messages is proportional to the number of
| places an error can reasonably happen and the number of
| connections it can happen over, and by any metric Meta's
| got a lot of those.
|
| No, in that detecting when you should be reporting a
| useful error message is itself a complex problem. If a
| service you call gives you a nonsense response, what do
| you surface to the user? If a service times out, what do
| you report? How do you do all this without confusing,
| intimidating, and terrifying users to whom the phrase
| "service timeout" is technobabble?
| jessriedel wrote:
| > If a service you call gives you a nonsense response,
| what do you surface to the user?
|
| If this occurred during the authentication process, I
| think I would tell the user "Sorry, the authentication
| process isn't working. Try again later." rather than
| "Invalid credentials". And you could include a
| "[technical details]" button that the user could click if
| they were curious or were in the process of
| troubleshooting.
| lanstin wrote:
| Come on use a little imagination. DNS lookup for the db
| holding the shard with the user credentials disappears.
| Code isn't expecting this, throws a generic 4xx because
| security instead of a generic 5xx (plenty of people
| writing auth code will take the stance all failures are
| presented the same as a bad password or non-existing
| username); caller interprets this a login failure.
|
| Same auth system system used to validate logins to the
| bastions that have access to DNS. Voila.
| shkkmo wrote:
| > plenty of people writing auth code will take the stance
| all failures are presented the same as a bad password or
| non-existing username
|
| Those people would be wrong. You can take all unexpected
| errors and stick them behind a generic error message like
| "something went wrong" but you should not lie to your
| users with your error message.
| jtuple wrote:
| It's about not leaking sensitive information.
|
| If you have different messages for invalid username vs
| invalid password, you can exploit that to determine if a
| user has an account at a particular service.
|
| "Invalid credentials" for either case solves this
| problem.
|
| But sure, let's report infra failures different as
| "unexpected error"
|
| Now, what happens if the unexpected error is only when
| checking passwords, but not usernames?
|
| Do you report "invalid credentials" when given an invalid
| username, but "unexpected error" when given a valid name
| but invalid password?
|
| If so, you're leaking information again and I can
| determine valid usernames.
|
| So, safe approach is to report "invalid credentials" for
| either invalid data or partial unexpected errors.
|
| Only time you could safely report "unexpected error" is
| if both username check and password check are failing,
| which is so rare that it's almost not worth handling.
| Esp. at the risk of doing wrong and leaking info again.
| seppel wrote:
| > That said, getting there strikes me as pretty
| challenging. Automatically detecting a down state is
| difficult and any detection is inevitably both error-prone
| and only works for things people have thought of to check
| for. The more complex the systems in question, the greater
| the odds of things going haywire. At Meta's scale, that is
| likely to be nearly a daily event.
|
| Well, in principle, the frontend just has to distinguish
| between HTTP status 500 (something broken in the backend,
| not the fault of the user) and some HTTP status code 4xx
| (the user did something wrong).
| Kalium wrote:
| Yes, assuming the responses are usefully different,
| accurate, and you get responses in a timely manner.
| seppel wrote:
| The "your username/password is wrong" message came in a
| timely manner. So someone transformed "some unforeseen
| error" into a clear but wrong error message.
|
| And this caused a lot of extra trouble on top of the
| incident.
| boring_twenties wrote:
| Well you can't expect to hire engineers with half a brain for
| the pitiful compensation Meta offers, can you?
| barbazoo wrote:
| Not the worst thing that a bunch of Facebook users are
| resetting their passwords.
| sandspar wrote:
| That's the sound of millions of "password" becoming
| "passwordnew"
| mns wrote:
| There are quite some harsh comments here below. You can't
| plan for every possible failure point, who knows what part of
| a system/infra out of everything that they have went down and
| triggered this behaviour. Some things you just can't
| catch/predict. Especially in huge systems like theirs. I
| would expect people here to understand things like these and
| not just call people names for something like this, we all
| know things seem simple/clear from the outside, but the job
| of debugging and fixing something like this take quite some
| effort.
| anigbrowl wrote:
| This is a company with one of the largest digital
| infrastructures in the world. An outage is understandable,
| inability to tell they're having an outage and inform users
| appropriately is not. Stop making excuses for people who
| are literally awash in resources.
| dylan604 wrote:
| It is always better for the company's rep for the issue
| to have been on your end. Admitting fault comes with a
| potential liability. It's gaslighting written as an SLA
| shkkmo wrote:
| You can't plan for every contigency, but you can reserve
| potentially scary message for situations where you know
| they are correct. An unpected error state should NOT result
| in a "invalid credentialiald error".
| shadowgovt wrote:
| This is the nature of credentials errors. The more
| information you give, the more you're telling an
| untrusted and therefore assumed-hostile agent.
|
| I hate it because it's bad UX, but that's the thinking
| behind it.
| shkkmo wrote:
| Pushing people to unnecessarily reset credentials
| increases risk. Not only does it increase acute risk, but
| it also decreases the value of the signal by crying wolf.
|
| The argument here is the kind of nonsense cargo cult
| security that pervades the industry.
| shadowgovt wrote:
| I think this argument falls flat on two axes:
|
| - in general, if the system is broken enough to be giving
| false-negatives on valid credentials, it's broken enough
| that there isn't much planning to be done here because
| the system's not supposed to break. So if they give me
| "Sorry, backend offline" instead of "invalid credential,"
| they've now turned their system into an oracle for
| scanning it for queries-of-death. That's useful for an
| attacker.
|
| - in the specifics of this situation, (a) credential
| reset was offline too so nobody could immediately rotate
| them anyway and (b) as a cohort, Facebook users _could_
| stand to rotate their credentials more often than the
| "never" that they tend to rotate them, so if this outage
| shook their faith enough that they changed their
| passwords after system health was restored... Good? I
| think "accidentally making everyone wonder if their
| Facebook password is secure enough" was a net-positive
| side-effect of this outage.
| jonnycomputer wrote:
| It would be a better UX, but, depending on the outage, that
| might be a really hard behavior to guarantee.
| Twirrim wrote:
| Likewise, started password reset process that won't complete,
| asked my wife to double check my account wasn't compromised and
| posting cryptocurency crap or somesuch.
| KMag wrote:
| Yea, the wife came to me in a bit of a panic that her Facebook
| account got hacked. I tried logging in to FB to check if I had
| been unfriended, and I also got errors indicating my password
| was incorrect. My FB password is 96 bits from /dev/urandom in a
| GPG-based password manager I wrote for myself a couple decades
| ago. So, no my password wasn't wrong, and I'm not a big enough
| target for someone to put enough effort into figuring out how
| to snarf up my password data and crack my GPG passphrase.
|
| Anyway, when FB thought my password was wrong I calmed way
| down. I thought maybe FB corrupted their password DB or
| something, so I just tried to reset my password, got into an
| odd workflow loop, and then quacked "downdetector facebook".
| sigil wrote:
| > My FB password is 96 bits from /dev/urandom in a GPG-based
| password manager I wrote for myself a couple decades ago.
|
| We have the same approach to password management!
| alkonaut wrote:
| Yeah your product should NEVER confuse an auth service being
| down with a failed auth. This is really terrible by FB.
| jph wrote:
| Facebook is throwing up the sign in page, as if your sign in has
| failed or someone's logged you out. Instagram and Messenger also
| seem to be affected.
| TheOtherHobbes wrote:
| And Threads.
| coffedean wrote:
| And the Quest headsets.
| jzw8833 wrote:
| Yes, just checked Instagram and it says "Couldn't refresh feed"
| and is generally not loading. Messenger has also logged me out.
| subtra3t wrote:
| For some reason I couldn't browse the home page but I could
| still access my messages (website).
| corobo wrote:
| Oh good, I thought iOS Firefox was forgetting to load its cookie
| jar again
| greatquux wrote:
| This will be great for productivity!
| cm2187 wrote:
| I thought the average facebook user was of retirement age.
| epcoa wrote:
| Instagram is also affected.
| graemep wrote:
| Not sure about globally, but middle aged, 45 - 65, slightly
| more women.
|
| I would bet those who are not working in full time jobs are a
| lot more active though.
| subtra3t wrote:
| In a lot of developing countries, Facebook is often the
| _only_ app that a significant part of the population uses.
| And that includes young adults as well.
| aranw wrote:
| Instagram has stopped working for me and I've also noticed other
| sites having issues like YouTube?
| MyFirstSass wrote:
| Yes both Youtube and Instagram seems to be down too for me.
| jzw8833 wrote:
| Instagram is down but Youtube seems to be fine for me
| MyFirstSass wrote:
| Youtube seems to be about 1/3 i get "Something went wrong"
| error on the frontpage, so only sporadic.
| MarcellusDrum wrote:
| Maybe because of the sudden increase of traffic?
| zalyh wrote:
| YouTube seems fine
| rvz wrote:
| Oh dear.
|
| Facebook, WhatsApp, Instagram, Threads are all down. and its time
| to contact the CEO of Meta to bring them all back up.
|
| At least Twitter / X is still up, so time to complain about it
| there.
| lebean wrote:
| I'm on it! I WhatsApped Zuckerberg directly. He's aware of the
| situation.
| lsh123 wrote:
| WhatsApp works for me
| afturner wrote:
| [deleted]
| ink_13 wrote:
| Meta doesn't do that. For one thing, "Patch Tuesday" is a
| Windows thing, and 0% of production traffic is served from
| Windows there. For another, they are constantly and always
| redeploying.
|
| More likely that someone bungled a deploy of user auth. No
| doubt they are rolling back as we speak.
| chongli wrote:
| _More likely that someone bungled a deploy of user auth_
|
| How would that affect YouTube?
| bandwidth wrote:
| Surplus load from Meta users flocking to other platforms
| mdrzn wrote:
| FB logs you out and claims "Wrong password" if you try to login.
| IG just doesn't load new posts or comments, but doesn't log me
| out.
| MyFirstSass wrote:
| Anyone have a good guesstimate of how much money is "lost" when
| Youtube/Facebook/Instagram/Whatsapp (more?) are down each minute?
| thehappypm wrote:
| Naively, divide ad revenue by time to get a dollars-per-time.
|
| But thats naive because ad serving isn't totally sold out so
| they can make up for it by increasing the density of ads in the
| next time window. If the outage is short, then the impact is
| small.
|
| But some markets are totally sold out and there's no making up
| for lost impressions.
| MarcellusDrum wrote:
| This is something I've thought about a while back. Like
| Facebook probably has a "maximum number of ads shown to users
| per post" value. So theoretically, they have a ceiling for
| how many ads can be bought in a specific time frame before
| having to increase the ceiling/find new users.
|
| Do they share data about this?
| post-it wrote:
| > before having to increase the ceiling/find new users.
|
| Or raise the prices of ads.
| bmau5 wrote:
| They also do not refund spend during this time typically so
| wouldn't be a 1:1
| sbrother wrote:
| They don't? I've worked in ad tech at some smaller places
| and we absolutely refunded spend during outages.
| bmau5 wrote:
| From my experience you'll receive a partial refund - and
| in some instances like inexplicable overspending, etc. -
| you won't receive anything. This may be an exception
| given a full sitewide outage, though
| chatmasta wrote:
| How is money even being spent if users aren't viewing the
| ads?
| thehappypm wrote:
| Not all ad spend is impression based, there are things
| like takeovers. And, sometimes impression counts arent
| guaranteed, theyre estimated
| lenerdenator wrote:
| That's because they were some smaller places.
|
| When you've more-or-less monopolized a lot of the web's
| content sharing you get to tell your clients to pound
| sand. Where else they gonna go? Twitter? The incel white
| supremacist dollar is not what advertisers call "the good
| dollar".
| bananapub wrote:
| > they can make up for it by increasing the density of ads in
| the next time window
|
| no need to do that, for google search, people will _come back
| later_ to make the search
| thehappypm wrote:
| Yes, but, they'll need to serve X ads in a smaller amount
| of time
| kevincox wrote:
| > they can make up for it by increasing the density of ads in
| the next time window
|
| Not only that but the bigger spenders will have more budget
| so the bidding after a large outage should return higher bids
| on average leading to increased profit per ad slot.
| mfrommil wrote:
| This is rough napkin math, no need to downvote if anyone knows
| the real number and this is way off :)
|
| Meta 2023 ad revenue was $131 billion. To make it easy, let's
| assume an even spread for # of users and ad revenue generation
| per hour/minute of the day and day of the year (which I'm sure
| is not the case).
|
| This would be:
|
| $358 million per day
|
| $15 million per hour
|
| $249k per minute
|
| This also assume a minute down won't be somewhat or totally
| offset by a spike in users when it comes back online.
| fullstop wrote:
| I didn't check your math, but your last number is probably
| per minute, not per hour.
| mfrommil wrote:
| Thanks, you're right . Edited it
| seydor wrote:
| I assume they will run all those campaign views after yhe
| outage, so very little will be actually lost
| zeven7 wrote:
| I can't answer your question, but when I was at Google I made a
| mistake that caused ads serving on Google results to become
| unclickable. For the postmortem they had me calculate (I don't
| think a dollar amount but) the number of ad clicks that would
| have happened during the time it was down. Of course I looked
| up average cost per click rates. Not sure if I could share even
| if I remembered, but it really put things in perspective.
|
| Overall it was a good learning experience. I didn't get
| reprimanded; several months later I got a promotion.
| strunz wrote:
| I can't believe you typed all that and didn't include a
| number, what a tease
| pradn wrote:
| Something like $X million an hour.
| ZiiS wrote:
| A hell of a lot less than when they are up.
| aaomidi wrote:
| This might be the end of Facebook if it lasts long enough.
|
| Invalidating people's sessions on apps? That's a HUGE cost.
| There's a huge % of users that won't be able to get back in.
| AlfredBarnes wrote:
| I hope so.
| dddddaviddddd wrote:
| I've known several people who, when they got signed out,
| couldn't remember their password and access their email, and so
| made a new account. I'm sure Facebook will see a spike of this.
| 00deadbeef wrote:
| It might not have actually invalidated the session
| aaomidi wrote:
| It does on the app. Logs you out.
| issafram wrote:
| Seems like somehow YouTube is also affected? I'm starting to
| think that this is an attack of sorts and not Meta specific
| issafram wrote:
| Google Play Store also affected
| beejiu wrote:
| Given Facebook is like 10% of total Internet bandwidth, I'm
| curious what happens when all that traffic suddenly gets
| "redirected" elsewhere by people going to other sites and
| platforms. Could YouTube cope with a sudden 10% uptick in
| streams for example?
|
| Edit: YouTube seems now to be defaulting to a lower bitrate,
| leading me to guess it's a demand issue.
| ykonstant wrote:
| That seems to be the actual case. Most of the "usual web" is
| struggling. Even HN is struggling for me.
| usrusr wrote:
| For me it happened when my long-dormant messenger lighted up (for
| a perfectly valid reason, no surprise there) and made me go
| through those unbundling options presumably mandated by EU.
| Certainly a coincidence, but it does irrationally strengthen that
| satisfying feeling I get whenever I see the ad giant stumble.
| Euphorbium wrote:
| They recently started demanding that I log in on a phone app, to
| make some choice to use messenger on ipad. That is pretty much
| the end of messenger for me.
| kossTKR wrote:
| What are the chances this is GPT-5 that has escaped and gone
| rogue?
|
| Jest aside, i wonder when/if stuff like that will actually
| happen.
| fishnchips wrote:
| Negligible. It would have gone after Twitter first.
| prepend wrote:
| Comically twitter seems to be working fine and is where I was
| able to find confirmation that it wasn't just me.
| fishnchips wrote:
| Yeah, exactly, so it must still be a good old human
| mistake, not Skynet.
| Woshiwuja wrote:
| You mean X right?
| mihaic wrote:
| I don't think that anyone but Elon ever means X.
| Woshiwuja wrote:
| forgot the /s
| causal wrote:
| Clearly the work of Llama 3
| meindnoch wrote:
| About 95%.
| corytheboyd wrote:
| My shadow is extremely good at mimicking my every move, but I
| don't live in fear that it will make the jump to the third
| dimension, kill me, and assume my identity. Should I?!
| TeMPOraL wrote:
| Well, maybe - if the source of the shadow keeps getting more
| and more powerful, it will eventually _turn you into a
| permanent shadow_.
| 1970-01-01 wrote:
| Which one this time - BGP or DNS?
|
| Either way, global GDP should be up :)
| ototot wrote:
| Given the range. Doesn't look like DNS
| marcosdumay wrote:
| So... No way it could be DNS?
| chrismarlow9 wrote:
| Narrator: it was DNS
| kordlessagain wrote:
| My sense is that it's BGP.
| wenbin wrote:
| Google login seems to be slow just now for 2 of our websites
| synergy20 wrote:
| yes it's all broken now, so I can take a short break here :)
| notsahil wrote:
| Discord is having issues too! https://discordstatus.com/
| montekristooGDB wrote:
| +
| wddkcs wrote:
| Could this be related to Google's log in page change? Seemed
| cosmetic only, but funny timing that Google's page update
| happens the same time all these log in issues pop up.
| chimeracoder wrote:
| > Could this be related to Google's log in page change?
| Seemed cosmetic only, but funny timing that Google's page
| update happens the same time all these log in issues pop up.
|
| That rollout is staggered over time, so not all users receive
| it at the same time. It's unlikely to be related.
| shombaboor wrote:
| They've been previewing that change for weeks, and I
| wondered: why do they need to change it at all? Is it some
| product manager justifying his need to exist?
| alwa wrote:
| Based on their hype banners, I was ready for a major
| overhaul, too. Or at least something obviously different
| about the login flow. It sure looks like somebody just
| clicked the left-align button and spit-shined the typeface
| a tiny bit.
|
| Which, I guess, is the best possible redesign: one that
| freshens up without rocking the boat.
| Shrezzing wrote:
| I think the downtime associated with other services could just
| be people choosing alternative sites for their social media
| time.
|
| Facebook + Insta makes up a huge share of the social media
| market, and when they go offline, it'd be natural for their
| competitors to receive large sudden upticks in trafic they're
| not immediately prepared for on a Tuesday morning.
| happosai wrote:
| Scary idea but could indeed be. Who let the zombies out!
| BaldricksGhost wrote:
| Threads is down too. Given its integration with Instagram that's
| not exactly surprising though.
| monsieurgaufre wrote:
| Also can't logged into my openAI account.
|
| Not saying is related, but timing is weird.
| r4nd0m_jump wrote:
| if you are logging in with facebook account that might be the
| reason
| monsieurgaufre wrote:
| Indeed if it was the case. But i always create an account
| with my email for this kind of situation.
| vincnetas wrote:
| every one panicked that they got hacked. i got a slim hope that i
| also got hacked and that i will not bother to recover my account
| and just roll without FB :) im too week to quit my self.
| revscat wrote:
| Maybe post your password to Twitter or something. Let it escape
| into the wild, where it can be observed in its natural state.
| Liquidor wrote:
| I know your intention is to help, but please don't share your
| FB password (if that wasn't obvious already lol). Letting
| randoms log into your FB account will just have massive
| consequences with your friends and family thinking it's you
| talking to them etc.
| revscat wrote:
| tZ7TuQ)cznNhnimgJVwUuys(uy
| noman-land wrote:
| Send me your password and I'll log in and change it to
| nonsense. Get out of that heck hole.
| avgDev wrote:
| I tried logging in, I couldn't and was like "nice, I guess I
| want to have to look at garbage today".
|
| I really would like a social network just for friends without
| all the garbage bloat.
| dv_dt wrote:
| Well, I've been spitballing 1-3 years for effects of layoffs to
| affect company performance, so this timeframe is about right
| lazide wrote:
| Yup. That it's related to the elections is also predictable,
| due to stress.
|
| Made worse in big corp due to affirmative action + lack of
| enough qualified candidates meeting diversity criteria.
|
| Which is inevitable when you have coarse criteria applied to
| such a large industry this way so quickly, as it takes decades
| for _anyone_ to be qualified for the senior roles, and many
| years for junior /mid level, even if there were no pipeline
| issues, which there are.
|
| And unqualified folks in leadership, and mid level == stupid
| mistakes.
|
| And, with the DOL rules, the company can't even pay people
| differently, so no bueno even giving the high performers
| keeping things afloat better bonuses - unless they happen to
| meet the diversity criteria and it makes the stats look good.
|
| Which it's already hard enough to do properly when there is
| only one dimension, and impossible when there are 2-3.
|
| so the bigger the company, the faster it has to cut its own
| throat.
| binaryblitz wrote:
| Blaming an outage on DEI... Man that's a new one for me.
|
| Could you let us know where you work? I want to make sure I
| never apply there.
| lazide wrote:
| Bwahaha, just wait until you see the shrapnel flying over
| the next year.
|
| You don't think the steady erosion in system reliability
| and ever increasing outages is unrelated to these pressures
| do you?
|
| I've seen the sausage being made at the middle manager
| level in big corp for a long time. It's never any one
| person/hiring decision, but the pattern and it's impact has
| been obvious (and getting unavoidable) for a long time.
|
| That no one seems to want to talk about the actual issues,
| but doing character assassination and black listing (like
| this comment) is part and parcel of the problem.
| ijwann wrote:
| > You don't think the steady erosion in system
| reliability and ever increasing outages is unrelated to
| these pressures do you?
|
| Outages have steadily decreased at major companies. I
| don't know what you're looking at.
|
| Remember AWS taking out a good chunk of the internet many
| times a year because their east coast data center kept
| going down? Remember the fail whale meme-ing because
| Twitter was so unstable?
|
| Industry site reliability has only gotten better over the
| years.
| lazide wrote:
| Bwaha, so now everything is actually getting better and
| more reliable in big corp land!
|
| I'm sure AT&T, Google, Facebook/Whatsapp/Meta, BofA,
| Apple, MS, and many others who have had prominent massive
| outages and embarrassing product launch failures this
| year will be happy to hear this.
|
| Notably, Amazon is one of the few companies that has
| managed to avoid a lot of the DEI noise somehow. Perhaps
| due to their reputation for having such a brutal work
| culture already?
|
| I can't wait to hear what you're going to say next.
|
| Big Corp Software quality improving AND running faster on
| existing hardware?
| dv_dt wrote:
| Your spitball is flying in a completely different universe
| than my spitball. Imo anytime you have layoffs that cut so
| deep, you cut through informal capability, knowledge and
| relationships that take a long time to form. If anything DEI
| helps create internal resilience because the personal
| networks end up a little different, giving you wider and more
| angles of coverage.
|
| HN talks about people in open source holding up major
| functionality with little to no recognition. That happens
| within corporations too. Indiscriminate layoffs may directly
| fire those people, or signal to them that it's better to move
| elsewhere leaving gaps that only get discovered over time.
| lazide wrote:
| Of course, and not breaking anything with layoffs is
| already hard when your sole criteria as a manager is 'are
| they effective'. Which it's never been that simple,
| especially in big corp, but it's waaay more difficult now.
|
| And so what happens when you're required by the gov't and
| leadership to also comply with coarse grained population
| statistics AND you can't find qualified people that meet
| those statistics enough? On top of having to make layoffs?
|
| My ex was a reasonably qualified software engineer, and
| even 4 years ago was getting no-interview offers because
| she was a woman - as explicitly stated by the recruiters.
|
| It's only gotten worse since then for hiring managers. She
| was offended because they literally didn't seem to care if
| she was qualified or not.
|
| I can provide links to signed and in force legal agreements
| between the DOL and Google for instance which formalize the
| need for this, and can point towards public records of
| evidence submitted to court of emails (internal) between
| recruiters which state the same too, btw.
|
| Actual job qualifications (as in skills) did not enter the
| conversation at all. Just course grained DEI attributes.
|
| So then they end up disproportionately cutting from the
| non-protected tranches (the groups that DO have to be
| qualified to stay) first because your stats still have to
| look good. I'm not saying DEI folks overall have no one
| qualified or hard working in them - rather, that there are
| little to no structural incentives for them to be. In many
| cases, they're also unfireable/unlayoffable.
|
| And eventually, the non-protected folks leave, burn out, or
| give up because f-this. Why do so much extra work when you
| literally can't even get paid more for it, or be recognized
| because it will piss everyone else off?
|
| And even if you're superhuman on that front - everyone
| burns out eventually. Which is also why you tend to see
| what you see in Open Source.
|
| It's never any one decision, but stochastic movement in
| this direction has been relentless and inevitable.
| tapland wrote:
| Shortly before I received password recovery codes to my email,
| from all facebook accounts I have.
|
| My email requires 2FA, as does my facebook account, but when I
| went there and clicked "forgot password" there was an unknown
| email address added to my account. That shouldn't be possible.
| yreg wrote:
| Your mouse click broke it.
| montekristooGDB wrote:
| Discord too ?
| neogodless wrote:
| https://metastatus.com/ seems to be suffering a hug of death
| now...
| perfectritone wrote:
| It really should be a static page.
| slig wrote:
| Built on React, though.
| Capricorn2481 wrote:
| React can be used fine for static pages. That's orthogonal.
| pests wrote:
| React does have server components which are completely
| rendered server side, ala PHP or similar. Even SSG/pre-
| rendering support.
| stefandesu wrote:
| Such irony...
| Galacta7 wrote:
| It could save others from clicking on downed sites, but not
| itself.
| hiccuphippo wrote:
| Tangent to this but I saw a link to metastatus last week and
| thought it would be a status page for other services' status
| pages. This makes it sound like a useful thing now, too bad the
| name is taken.
| jcgrillo wrote:
| Could call it metastasis, apropos when one outage cascades
| into others.
| thomasjudge wrote:
| During the outage it was cheerily reporting no problems
| Tarball10 wrote:
| I see Corporate Memphis has even infected status pages now.
| suyash wrote:
| Official page to check status: https://metastatus.com
| suyash wrote:
| It would have been smarter if they hosted this on a separate
| reliable server altogether.
| Ambroos wrote:
| It's on AWS, but likely just has issues with provisioned
| capacity now that it's actually being hammered.
|
| Meta has a small collection of tools on AWS to deal with
| large SEV0 events like these. Another one of them is a basic
| communication tool that does not use Meta's own servers for
| anything (including auth), a super basic version of the
| internal SEV tool.
| miyuru wrote:
| X-Amz-Apigw-Id:
|
| X-Amzn-Errortype: TooManyRequestsException
|
| X-Amzn-Requestid:
|
| Looks like the endpoints are on aws lambda and its getting
| rate limited.
| pwenzel wrote:
| Might as well rename it from Meta Status to Metastasis.
| Woshiwuja wrote:
| Thats down too. lmao
| coldcode wrote:
| It appears to be more than Meta, lots of things are showing a
| spike in reports, including AWS.
| rightbyte wrote:
| That might be the FB traffic instead going to the content
| producers? I mean half the internet users need to go somewhere
| else if they want procrastination.
| meindnoch wrote:
| Meta stock in freefall. Going short at 20x leverage.
| robjan wrote:
| Risky move. The slight drop (and subsequent recovery) today is
| unlikely related to this outage in any way.
| getlawgdon wrote:
| Can't tell if serious, but I've seen worse trades. Wouldn't
| call this a freefall yet: https://imgur.com/a/0UN3HZa
| greenie_beans wrote:
| OH MY GOD WHAT IS THE WORLD GONNA DO!!!!!?????
| nathanaldensr wrote:
| Touch grass. xD
| albatros1313 wrote:
| I love how no one here is talking about the "coincidental" timing
| that this is happening on Super Tuesday. Carry on, nothing to see
| here.
| cjohnson318 wrote:
| We've seen continuous layoffs for the past year or so, and the
| stability of everything is wonky now. Yesterday I had trouble
| with LinkedIn, last week it was Coinbase, a month or so ago
| Gmail was hanging. I don't think it's because of Super Tuesday.
| nathanaldensr wrote:
| I have yet to see one plausible reason for why this would
| affect primary voting in any way. Voting is done at the local
| level on local hardware; we don't vote on Facebook.
| dawnofdusk wrote:
| Messenger is also down. For some people this is an important
| means of communication (many young people I know in the US).
| People may either be in a panic or otherwise cannot
| coordinate sufficiently the plans they may have had to get to
| the polls.
|
| That being said this year's primaries have got to be
| historically uncontested that it could not matter less.
| angst_ridden wrote:
| It's not direct disruption.
|
| Grassroots organizers use FB and messenger and Insta for "get
| out the vote" communications. People use these services to
| goad their friends to go and vote. People use FB as a search
| engine to identify their polling place.
| Waterluvian wrote:
| Same reason we're also not talking about the ten million non-
| coincidences that didn't seem to happen today. It's a kind of
| survivorship bias that coincidences seem like they mean
| something.
|
| Or you could imagine it as this: your animal brain has evolved
| to notice patterns. Seeing coincidences like this is akin to
| seeing faces in the stars. The challenge of evolved being is to
| override those impulses when they're not logically sound.
| getlawgdon wrote:
| Sure. But if millions of evolved brains are "seeing the same
| pattern", they will behave as such, and so the meaning of
| that otherwise random noise in fact becomes something
| material.
| Waterluvian wrote:
| And that's why conspiracy nuts exist.
| getlawgdon wrote:
| It's also, you know, a foundation of behavioral
| economics. But either one, whatever.
| binaryblitz wrote:
| If this was 2016, I'd think there could be a possibility. Given
| that the candidates for the election are almost 100% a foregone
| conclusion at this point, what purpose would it serve?
| rantee wrote:
| Fueling clickbait conspiracy theories is one probable outcome
| that comes to mind. For example, even a winning candidate may
| claim that a "woke" employee sabotaged the site in an effort
| to subvert the will of the people to have free and open
| discourse, they won despite the media deck stacked against
| them, etc.
| angst_ridden wrote:
| I'm seeing a lot of non-FB services down too. Mostly AWS-based,
| but not all. My original conspiracy thoughts (this being Super
| Tuesday in the US) are giving away to thinking it's some low
| level routing issue.
| albatros1313 wrote:
| It's definitely related to Super Tuesday. Why kill 2 birds with
| one stone when you can kill 10?
| Waterluvian wrote:
| I'm having one heck of a time getting into Discord or getting
| YouTube to show videos. Anyone else experiencing issues outside
| of the Meta realm?
| NKosmatos wrote:
| Obligatory DNS joke :-)
|
| It's not the DNS, It can't be the DNS, It might be the DNS, It's
| the DNS
| Thaxll wrote:
| SRE now have real work to do.
| wly_cdgr wrote:
| I've never seen an outage this bad and I've been on Facebook
| since the oughts.
| Ambroos wrote:
| The 2021 outage was worse, it completely killed the network
| access to FB data centers for hours, and even led to issues
| with FB employees accessing offices (since the badge readers
| were also offline). It was so bad it got it's own Wikipedia
| page: https://en.wikipedia.org/wiki/2021_Facebook_outage
| wly_cdgr wrote:
| I don't remember that one - must have been on a hiking trip.
| This one might turn out to be worse yet! Hopefully not.
| jeromegv wrote:
| Mastodon is up!
| marcosdumay wrote:
| Just checked again to be sure, even lemmy.world is up.
| joemanaco wrote:
| Has a meta developer used Chat GPT again?
| DeusExMachina wrote:
| Looking at the Downdetector home page [1], it looks like many
| more services are having outages, not just the ones owned by
| Meta, including:
|
| - Google
|
| - YouTube
|
| - Google Play
|
| - T-Mobile
|
| - X (Twitter)
|
| - Discord
|
| - TikTok
|
| - Pokemon Go
|
| - Snapchat
|
| It looks like they all have the same failure point.
|
| [1] https://downdetector.com
| andtheboat wrote:
| iirc these all use GCP which would make sense for them all to
| be disrupted at the same time. I wouldn't have thought Meta was
| GCP reliant though?
| Yiin wrote:
| all of them are using oauth, likely auth provider issue?
| rany_ wrote:
| https://www.cloudflarestatus.com is reporting an issue with
| SSO login. So seems like you might be onto something..
| jgrahamc wrote:
| Definitely not us.
| kurthr wrote:
| And I assumed it would be DNS _again_. Would referral
| traffic cause issues like this?
| bburnett44 wrote:
| There's no way fb uses gcp
| ajross wrote:
| Most of those seem OK for me now, and DD agrees. This seems to
| have been a temporary blip for all of them, possibly some kind
| of service switchover/fallback "not entirely unrelated" to the
| Meta outage?
|
| Edit: actually a more attractive theory, given the very short
| timelines and near simultaneity of all those failures, is that
| downdetector _itself_ had a failure, possibly a Meta-
| dependence, that they noticed and corrected quickly.
| peterleiser wrote:
| GCP seems fine, and no issues logging into Google Cloud
| Console.
| peterleiser wrote:
| S**'s gettin' real: https://downdetector.com/status/pokemon-go/
| AndyJames wrote:
| Some piece of core infrastructure went down because everyone
| got spike at the same time. Surprisingly DoorDash and Steam
| was up
| agilob wrote:
| Everything is going down, except Steam. You know where to
| find me.
| dwighttk wrote:
| Scrolling down their list, why is DoorDash the only one that
| didn't have a spike this morning?
| esnard wrote:
| Pokemon GO player here, the app was absolutely fine unless
| you tried to use Facebook as your login provider.
| tuan wrote:
| Fb is down but YT is still up for me.
| jon-wood wrote:
| Or people are using Facebook Auth for them. I don't really
| trust Down Detector, which despite the claims is really People
| Winging on Twitter Detector.
| redserk wrote:
| Yeah, it's wild that it's now treated as an authoritative
| source, especially by some news organizations.
|
| It's as good as asking a neighbor what happened with a loud
| noise down the street. Sometimes you'll get something good,
| sometimes it'll be completely wrong.
| colpabar wrote:
| Downdetector is nice because it answers my question of "is
| anyone else having issues with this?" When it takes AWS an
| hour to even acknowledge "increased error rates", and tells
| me that everything is a-ok in the meantime, I want another
| perspective.
| ceejayoz wrote:
| Twitter's search used to be my go-to for this - a search
| for "AWS down" would typically be very illuminating - but
| it's tough to get it to genuinely spit out the most
| recent tweets with a keyword these days.
| AceyMan wrote:
| The New York Times just posted a news flash ... citing Down
| Detector :-P
| rimunroe wrote:
| > Yeah, it's wild that it's now treated as an authoritative
| source, especially by some news organizations.
|
| > It's as good as asking a neighbor what happened with a
| loud noise down the street. Sometimes you'll get something
| good, sometimes it'll be completely wrong.
|
| Asking my neighbors if they know what some loud noise was
| or about some local disturbance has been extremely reliable
| in my experience. The one time someone gave me an
| explanation about something which wasn't mostly right they
| qualified it with something like "So-and-so said it might
| be such-and-such but I don't know if it's true".
| Arainach wrote:
| You must have an exceptional neighborhood. Everywhere I
| lived, here's a handy map of "actual cause" :: "what the
| neighbors said it was"
|
| Car exhaust :: gunshot Appliance delivery truck liftgate
| :: gunshot Transformer explosion :: gunshot Garbage truck
| :: gunshot 787 at 25000ft :: complete ruining of peace
| and quiet Any police activity :: probably someone robbed
| a bank
|
| For the record, my city has (statistically
| indistinguishable from 0) homicides and bank robberies
| and, by American standards (I know, I know) no particular
| issues with gun crime.
| rimunroe wrote:
| I can imagine it being different in a city. I'm in a
| fairly quiet suburban area.
|
| One time I heard a loud boom. A few hours later I saw a
| neighbor outside and asked if he'd heard it and if he
| knew what it was. He told me a house a few neighborhoods
| over had exploded. I was a bit skeptical of it but he
| turned out to be right.
| sterlind wrote:
| exploded? the house... exploded? like, a gas leak or
| something?
| paulddraper wrote:
| It's as good as asking a thousand neighbors if they heard
| anything down the street.
| rimunroe wrote:
| > I don't really trust Down Detector, which despite the
| claims is really People Winging on Twitter Detector.
|
| I'm confused. Isn't listening for spikes in complaints about
| outages a great way to detect them? I know for a fact some
| service companies monitor social media channels for this
| purpose (among others). I'd be surprised if that wasn't more
| or less standard practice.
|
| I've checked Down Detector for ISP outages in my area many
| times now. It's always confirmed them before my ISP did.
| Cheer2171 wrote:
| The problem is that people use FB login for other sites,
| and if FB login is down, many users report a problem with
| that other site, not with FB.
| schrodinger wrote:
| The point is that the "other site" _is_ down for those
| who use Facebook to login to it.
|
| Maybe they should have a backup password (if they site
| allows it, f-ing Spotify doesn't), but it's still
| effectively down for them!
| noncoml wrote:
| The OP means there is a lot of collateral noise from people
| who are just tech savvy. Eg. "oh no, I can access Facebook,
| my internet must be down. Let me login in Down Detector to
| file a complaint against my ISP"
| ceejayoz wrote:
| > Isn't listening for spikes in complaints about outages a
| great way to detect them?
|
| When there's a major ISP outage, people report problems
| with all the major sites. When Facebook's down, people
| report problems with any site that has "Login with
| Facebook" as an option.
|
| It's almost never actually an outage impacting all of FAANG
| at once.
| ImPostingOnHN wrote:
| _> When Facebook 's down, people report problems with any
| site that has "Login with Facebook" as an option._
|
| If users log into your site with Facebook, then the login
| functionality of your site effectively is down when
| "Login with Facebook" is down.
|
| From the user's perspective, your subcontractors,
| including authentication subcontractors, are a problem
| for you to deal with and never show them. From your
| perspective, you could have architected your site in a
| way that logging in doesn't "go down" when Facebook login
| is down.
|
| If the user chooses "Login with Facebook" over other
| authentication options available, and they don't want to
| use other options, educating them with a good error
| message might help. Or you could remove the Facebook
| login option, if you (totally reasonably) don't want
| Facebook's failures to reflect poorly on you.
| ceejayoz wrote:
| > If users log into your site with Facebook, then the
| login functionality of your site effectively is down when
| "Login with Facebook" is down.
|
| There are plenty of sites where "Login with Facebook" is
| a convenience but hardly the only way to log in. Reddit,
| for example, has "Login with Google" and "Login with
| Apple"; it would be highly misleading to claim "Reddit is
| down" if Google's OAuth flow was having an outage.
|
| > educating them with a good error message might help
|
| Nothing in the API or OAuth flow would make that doable
| in an automatic fashion with this outage. It'd have to be
| something you put up manually as a banner after hearing
| of the outage.
|
| > Or you could remove the Facebook login option, if you
| (totally reasonably) don't want Facebook's failures to
| reflect poorly on you.
|
| I don't particualrly care; we're talking about why
| DownDetector isn't necessarily ideal for assessing. It
| can be a useful signal, in some scenarios, but I've seen
| plenty of spurious signals come from it.
| ImPostingOnHN wrote:
| _> Nothing in the API or OAuth flow would make that
| doable in an automatic fashion with this outage. It 'd
| have to be something you put up manually as a banner
| after hearing of the outage._
|
| That is fair: if I choose to architect my site such that
| a user-critical feature goes down when a 3rd party
| service goes down, it behooves me to monitor the 3rd
| party service and do whatever necessary to properly
| inform users what's going on.
|
| I edited my post unfortunately after you replied, but
| another option is removing the parts of your site that
| rely on 3rd parties, if you don't want the failures of
| those 3rd parties to reflect poorly on you (which they
| reasonably would).
|
| _> we're talking about why DownDetector isn't
| necessarily ideal for assessing. It can be a useful
| signal, in some scenarios, but I've seen plenty of
| spurious signals come from it._
|
| Indeed, and if a bunch of users say that a feature of
| your site is down, even if it's a result of a 3rd party
| failure: chances are, that part of your site is down, and
| it's partially your fault for relying on a 3rd party for
| that feature. The users correctly don't care what the
| root cause is, they expect you to either mitigate it or
| don't have a feature they rely upon be unreliable.
| ceejayoz wrote:
| All that's fine, but totally misses the point.
|
| Take a look at https://downdetector.com/status/aws-
| amazon-web-services/ ; scroll down to the comments.
|
| "SSH and Dbconnect stopped on all of my EC2 instances.
| Anyone else?"
|
| "I can't add a payment method"
|
| The chart shows a big spike this morning, but there was
| no AWS outage, nor does Amazon use Facebook login.
|
| Again, DownDetector can be a useful "is something unusual
| happening right now" signal, but it'd be a mistake to
| take its _attribution_ at face value.
| ImPostingOnHN wrote:
| Ignore the comments on DownDetector for a moment and
| check out that huge spike in reports recently. Clearly
| something wrong happened with AWS's user experience.
| That's something AWS needs to resolve, in the eyes of
| their users.
|
| _> The chart shows a big spike this morning, but there
| was no AWS outage_
|
| Are you sure? If hundreds of users simultaneously
| reported there was some sort of outage, particularly a
| huge spike like we saw, chances are there was an outage.
|
| _> Again, DownDetector can be a useful "is something
| unusual happening right now" signal_
|
| Exactly! Specifically, _" is something unusual happening
| right now with my site, in the eyes of my users?"_ Every
| site owner should know when that condition is true. What
| you think about your site "up-ness" isn't as important as
| what your users think about your site "up-ness". What you
| attribute your downtime to, isn't as important as what
| your users attribute your downtime to (you.)
| ceejayoz wrote:
| > Clearly something is going on with AWS's user
| experience.
|
| But that's not the case. It's a false positive.
|
| Pick a DownDetector service and open the page every day
| for a few days. You'll see it most of the time just
| reflects people waking up in the US timezones.
| ImPostingOnHN wrote:
| Is it a false positive, though? The data shows there was
| an outage. We would need more evidence to conclude
| hundreds of users, at that 1 spike, weren't actually
| having issues.
|
| In other words, we have hundreds of people saying there
| was an outage, and 1 person saying there wasn't.
|
| That's a problem AWS needs to resolve, regardless of what
| they think might be the root cause. If the users weren't
| experiencing any issues with AWS, I doubt they'd be
| reporting it.
|
| Your comment about timing is a good point: if people are
| working with AWS early in the day, and AWS is giving them
| problems, then they will probably report problems with
| AWS early in the day. I wouldn't expect them to report
| problems while they're sleeping.
| ceejayoz wrote:
| > Is it a false positive, though?
|
| Yes. AWS was not down this morning.
|
| > In other words, we have hundreds of people saying there
| was an outage, and 1 person saying there wasn't.
|
| We have hundreds of _millions_ using AWS and AWS-backed
| services successfully this morning.
|
| I'm out.
| ImPostingOnHN wrote:
| Hundreds of users, representing more users who didn't
| bother reporting, say they experienced issues when
| interacting with AWS this morning, so we'll need better
| evidence to the contrary to conclude otherwise.
|
| The fact that _some_ people accessed AWS without
| reporting issues does not mean that _all_ people did. For
| those who had issues, AWS is responsible for dealing with
| those perceptions.
|
| Indeed, it could have been a fault that affected a subset
| of users, for example 1 service in 1 availability zone.
| That's still an outage in the eyes of users, which AWS is
| responsible for managing. It could have been an issue
| with a route from 1 ISP. That's still an outage in the
| eyes of users, which AWS is responsible for managing.
|
| An even better example is the DownDetector page for
| Facebook, with hundreds of thousands of reports. Do we
| really think there's no correlation between what
| DownDetector reports and what users experience?
|
| tl;dr: what users think about your site is more important
| than both what you think about your site and the reality
| of your site, and you should be tracking it.
| baby_souffle wrote:
| > It's almost never actually an outage impacting all of
| FAANG at once.
|
| Exactly. If you click through down detector when things
| are _up_ you'll see people still complaining that $site
| is down. Could be a local power outage or even a flaky
| connection in their own home.
|
| Down Detector is one of many signal sources and should
| have a "credibly" score associated with it that's
| proportional to the number of people complaining that
| something's down.
| whatwhaaaaat wrote:
| do you really think there are masses of people who can't
| tell the difference between a single sign on service
| being down and individual sites being down and reporting
| it to downdetector?
|
| Even if there were doesn't the outage graph give you
| exactly the information your asking be curated?
|
| I very much don't like the idea of downdetector using
| some special sauce to hide or limit outage reporting.
| ceejayoz wrote:
| > do you really think there are masses of people who
| can't tell the difference between a single sign on
| service being down and individual sites being down and
| reporting it to downdetector?
|
| Yes, absolutely. 100%.
|
| > Even if there were doesn't the outage graph give you
| exactly the information your asking be curated?
|
| https://downdetector.com/status/aws-amazon-web-services/
|
| Was there an AWS outage this morning? The graph sure
| looks like it, but there wasn't.
| samtheprogram wrote:
| > do you really think there are masses of people who
| can't tell the difference between a single sign on
| service being down and individual sites being down and
| reporting it to downdetector?
|
| Have you never seen The Website Is Down?
| https://www.youtube.com/watch?v=uRGljemfwUE
|
| The answer is: way more people than a software developer
| might think. Ask anyone in IT, or go to anywhere bugs are
| reported and read a handful.
| whatwhaaaaat wrote:
| yeah that's my point. No one who arranges icons by penis
| is taking time to go file a report on down detector.
| samtheprogram wrote:
| Ahh, I see. In that case, most of DownDetectors data are
| from Twitter and other sources, not first party
| reporting, although even in the case of first party data,
| it is also sourced via "visits to DownDetector" which can
| be from a simple Google search for "is Instagram down?"
|
| If DownDetector relied primarily on direct reporting,
| they'd be the last to know.
| callalex wrote:
| I can guarantee you with 100% confidence from experience
| that the call centers for AT&T, T-Mobile, Comcast, etc.
| are all blowing up right now because of users who assume
| that if the Instagram app isn't loading it means the
| "wifi" is broken. Also keep in mind "wifi" doesn't mean
| 802.11, it means "anything related to the internet" up to
| and including 4g/5g and Ethernet.
| whatwhaaaaat wrote:
| Ok great. How does that equate to having down detector
| filter reports?
| KMnO4 wrote:
| Heh, as soon as I saw Instagram failing to load, I
| immediately assumed it was Roger's fault. They just suck
| when it comes to reliability and Instagram has a much
| better track record.
| lobocinza wrote:
| Opened devconsole, saw it was a server error then went to
| HN for confirmation.
| bitfilped wrote:
| Yes, https://www.nngroup.com/articles/computer-skill-
| levels/
| qeternity wrote:
| > do you really think there are masses of people who
| can't tell the difference between a single sign on
| service being down and individual sites being down and
| reporting it to downdetector?
|
| Absolutely without a doubt.
|
| 99.9% of people don't know what single sign on means or
| how it works.
|
| Sometimes I wonder what world HN lives in.
| jordanthoms wrote:
| That's kinda the point though isn't it? DownDetector is
| showing an early indication of a major outage in both of
| your examples. The issue may not be caused by the
| indicated service, but it's still a useful information
| source especially when we can correlate reports on there
| with what we are seeing in our internal monitoring.
| ceejayoz wrote:
| A big spike on DownDetector is an indication of
| _something_ going on.
|
| Its attribution of _what /who_ is often incorrect. You'll
| see "maybe it's more than Big Site X!" comments come up
| on every HN thread like this citing DownDetector; it's
| almost never the case, and folks on HN should know
| better.
| rimunroe wrote:
| > When there's a major ISP outage, people report problems
| with all the major sites. When Facebook's down, people
| report problems with any site that has "Login with
| Facebook" as an option.
|
| Yes? That's how all top-level reporting is going to work.
| It's not going to tell you which part of your service is
| inaccessible. It's just telling you that people can't
| access it. You obviously have to do additional
| investigation to figure out why people are having
| trouble.
| ceejayoz wrote:
| > It's not going to tell you which part of your service
| is inaccessible.
|
| Scroll up the thread a bit;
| https://news.ycombinator.com/item?id=39605354
|
| Even here on HN, where people should know better, people
| take its incorrect attribution as useful info. TikTok
| isn't down. X isn't down. Google isn't down.
| rimunroe wrote:
| I would completely agree that people are bad at
| _interpreting_ Down Detector-type results, but that doesn
| 't mean it isn't providing a very useful signal.
| jajko wrote:
| Indeed I haven't noticed any blip in functionality, but
| then again I don't ever do FB (or other external service)
| login. Absolutely no reason to do so, long term drawbacks
| are too serious to be lazy about this.
| ipaddr wrote:
| You can't login to snapchat, tiktok, youtube with your
| facebook
| ceejayoz wrote:
| Yes, those are all just Down Detector's bog standard
| false positives.
|
| Pull the page up tomorrow and you'll see the same morning
| spike there as people wake up.
| ldoughty wrote:
| The problem is the source of the reports and display of the
| reporting.
|
| I'd trust Down Detector a lot more if it was filled with
| Hacker News community -- people who are able to understand
| that there's "DNS" and "Routing".. and that your phone can
| have internet access at home while your home PC does not.
|
| I personally hate Down Detector's graphing because it can
| make it 'look' like there's an issue when there isn't
| really... Facebook with 500,000 reports looked as down as
| Google with 1,000 reports... For equally sized / used
| entities, I would not trust that "Google" is down with
| 1,000 reports. I had a coworker ask me what was going on
| with the internet because "everything is down.. Facebook,
| google, gmail, microsoft!" (when seeing the Down Detector
| home page)
|
| DD should normalize the graphs against the service history
| in some way. A service shouldn't spike because it had 30
| reports / hour for a day, then suddenly has 100... when it
| has a history of being out with 100,000+ reports. The 100
| reports are probably mis-reporting, but you can't tell
| until you dig into each service, one by one, with separate
| page loads.
| cdchn wrote:
| In the Twitter operations area there was a big TV that
| streamed searches for #failwhale etc. It was actually very
| useful to detect problems with Twitter by looking for
| people complaining about Twitter on Twitter.
| fmajid wrote:
| Well, Twitter was down for me at least. Google, on the
| other hand, was not unlike what DownDetector claimed.
| EasyMark wrote:
| I think it's a good data point, but it's not a "tell all"
| indicator. So I agree with you.
| johnfn wrote:
| People are using Facebook Auth for YouTube?
| KineticLensman wrote:
| Seems likely. TikTok and YouTube are currently working for
| me, while Meta platforms aren't.
| tflol wrote:
| i trust Down Detector more than the (majority of) companies
| who are silent during outages
|
| hell, i'm surprised Down Detector hasnt been outright sued
| due to the graphs being an actual honest representation of
| availability that shitty companies cannot hide
| consumer451 wrote:
| > Or people are using Facebook Auth for them.
|
| Gmail is also on the list. You can't use FB auth to login to
| Gmail, can you?
| jordanthoms wrote:
| Gmail was also experiencing issues: https://www.google.com/
| appsstatus/dashboard/incidents/shD5Vv...
| tzs wrote:
| YouTube was definitely doing something weird that doesn't
| seem likely connected to Facebook.
|
| A couple hours ago after watching a video I went to my home
| page, which usually shows recommendations based on what I've
| recently watched plus a few videos labeled as sponsored that
| have nothing to do with any of my interests.
|
| Instead everything on the home page was either a sponsored
| video, or a movie that was free to view with ads, or
| something from one of their music products.
|
| I tried from an incognito window to see if it had something
| to do with being logged in. Normally going incognito loses
| the history-based recommendations but at least recommends
| user uploaded content. But now it has just like my logged in
| home page. No user content. Just ads and videos from Google's
| movie and music services.
|
| Refreshing gave an error that said something went wrong. I
| then logged in on that page and again got something went
| wrong. Another refresh got a page with some user content.
| Another refresh was the ads and Google stuff page.
|
| A little later it seemed to clear up and now my home page is
| back to normal.
| EscargotCult wrote:
| Their outage heatmap is also basically a population density
| map too. https://xkcd.com/1138/
| baby wrote:
| It's not that, getting some issues using X
| harryquach wrote:
| Thanks for sharing this, from what I have read it looks like an
| issue beyond just Facebook.
| sbrother wrote:
| HN seems to be struggling too, but that could just be everyone
| here to talk about the outages.
| passwordoops wrote:
| Happens whenever Sama sneezes too
| GaggiX wrote:
| That's the standard HN experience, this site runs on a single
| core I believe.
| HumblyTossed wrote:
| Does it really or is this a joke?
|
| Edit: I found the following, I wonder if it's still the
| case.
|
| https://news.ycombinator.com/item?id=16076041
| neocritter wrote:
| It's real. Single core performance improves all the time.
| People overestimate how much power it takes to handle
| lots of queries per second on a well-tuned system and
| well-written software in 2024.
|
| https://hn.algolia.com/?dateRange=all&page=0&prefix=true&
| que...
| __s wrote:
| Note that's the application server process being single
| threaded, but the server machine is 4 core, so nginx
| cache etc use other cores
| neocritter wrote:
| A worthwhile distinction!
|
| I looked up the CPU mentioned in the link from your other
| comment. It looks like HN handles enormous traffic on
| about 2x the power of the last Celeron chip ever made.
|
| https://www.cpubenchmark.net/compare/2383vs5793/Intel-
| Xeon-E...
| rightbyte wrote:
| Doesn't he mean single socket by single core?
| ndiddy wrote:
| I see the "sorry, we are receiving too many requests, try
| again in a few minutes" error several times a day on
| here. I don't think that HN is reliably able to handle
| the amount of users it currently has.
| 6510 wrote:
| It's a feature
| busymom0 wrote:
| I believe that's by design if you send an action request
| very quickly after a previous one. It's very easy to
| replicate. Open a post. Then click the upvote button and
| very quickly click the favorite button too. That will
| trigger it. I think it's used to rate limit.
| yreg wrote:
| >"Sorry, we're not able to serve your requests this
| quickly" is our little server process saying "help, I
| only have a single core and I'm out of breath here". If
| your account were rate limited it would say something
| like "You're posting too fast, please slow down."
|
| dang, linked in one of the ancestor comments. But I still
| suspect you are correct.
| busymom0 wrote:
| I just tested it by quickly upvoting your comment and
| then favoriting it and the error was:
|
| > Sorry, we're not able to serve your requests this
| quickly. reload
|
| Note that this only seems to happen for actions. Doesn't
| seem to be the case if I am just loading a page quickly.
| newswasboring wrote:
| I have been using HN daily since I was a teenager. I've
| seen that message maybe 10 times outside of serious
| issues in last 15 years. It's strange to me that it
| happens so frequently for you.
| yreg wrote:
| That's difficult to believe to be honest. I get it
| several times a week.
| tomrod wrote:
| I get something like that when I try to comment and then
| upvote too quickly.
| disqard wrote:
| I can confirm -- I've only seen this as a form of
| throttling (i.e. preventing users from sending too
| frequent HTTP requests).
| joecool1029 wrote:
| There's a manual rate limit that can be assigned to your
| account if you post frequently on controversial topics.
| Afaik once it's there it stays until removed by a
| moderator.
| jermaustin1 wrote:
| I've been seeing this multiple times a week for the past
| couple of years. It's gotten worse since 2020. I think
| that they are preparing to upgrade it, or did upgrade it?
| AnimalMuppet wrote:
| Hmm. I get a different message. Something like "We are
| having trouble handling your request. Sorry!"
|
| I saw it just a few minutes ago, but I don't remember the
| exact wording...
| ndiddy wrote:
| Yeah that's the one I meant, I don't remember the exact
| wording either.
| callalex wrote:
| That is a generic rate limiter that is independent of
| system load. As far as I can tell if you make more than
| one request per every 5 seconds, you will always be
| served the rate limit page.
| marcosdumay wrote:
| The message for this is something like "we can't serve
| your requests that fast". The GP is quoting the site
| overloaded message.
| HumblyTossed wrote:
| > People overestimate how much power it takes to handle
| lots of queries per second on a well-tuned system and
| well-written software in 2024.
|
| I wasn't overestimating anything, but with how easy it is
| to write concurrently software today, why limit your site
| to a single core.
| neocritter wrote:
| Maybe it's a lisp thing. Who knows what mysteries lurk
| here
| Zak wrote:
| It's not a lisp thing. Many lisps are capable of
| multithreading, including implementations of Common Lisp
| that have had it far longer than HN has been around, and
| Clojure, which is extremely good at it.
|
| It even looks like Arc, the lisp HN is written in has
| threads now, but Arc is built on top of Racket and uses
| Racket's green threads, so it only takes advantage of one
| CPU core. Racket _does_ have OS threads, but Arc does not
| use them.
| hoosieree wrote:
| > well-tuned ... well-written
|
| These are good for actual business needs, but bad for
| resume-driven development.
| __s wrote:
| See also: https://news.ycombinator.com/item?id=28478379
| Solvency wrote:
| You don't run a massively profitable VC company by just
| throwing money away at a second core.
| paulddraper wrote:
| HN is always struggling.
|
| Couldn't use it two nights ago, IDK why.
| govg wrote:
| There is no consistent scale on that graph, so any local maxima
| of reports received would look similar to any other.
| mankyd wrote:
| Exactly this. FB topped out around 520,000 reports. Google
| topped out around 1,400. That's a massive difference in
| scale.
|
| Both are above their baselines, but I bet some is just mis-
| reports, or increases in awareness due to more people
| checking in.
|
| Meta seems to be the only one really affected from what I can
| tell.
| jordanthoms wrote:
| We saw a big spike in latency and failures on the Google
| OAuth apis starting at the same time (15:21 UTC)
| NelsonMinar wrote:
| I made that same mistake after seeing someone post an
| unlabeled set of graphs to a Slack. The Google peak reported
| outages is about 0.25% of the Facebook peak. It seems
| reasonable some people just made a mistake.
| btbuildem wrote:
| What's your fave conspiracy theory? Massive cyberattack for
| Super Tuesday? Powers-that-be mandated takedown? Mossad sleeper
| agents activated? Covid-brain struck that one engineer
| attending to that one wire that kept everything going?
| N19PEDL2 wrote:
| Houthis destroying underwater cables in the Red Sea.
| op00to wrote:
| i knew my packets took a wrong turn at Albuquerque
| dylan604 wrote:
| I guess the swamp got drained, so there's no more flow
| through the tubes.
| Solvency wrote:
| It's honestly despicable how billion dollar companies at the
| forefront of making AI have such insanely brittle networking.
| kordlessagain wrote:
| Dealing in absolutes is a result of misaligned expectations.
| sandspar wrote:
| "The Force SHOULD have saved my mother's life!" - Anakin
| Skywalker
| packetslave wrote:
| "blah blah blah I don't like the company so all their
| engineers must be really stupid..."
| Solvency wrote:
| "The metaverse is the future of this company! We've hired
| the very best brains in the world to make these 1998 era
| PS1 polygon avatars!"
|
| _whole company goes offline_
| packetslave wrote:
| The Metaverse engineers and AI researchers don't actually
| run the infrastructure, but by all means, keep
| shitposting. It's so clever.
| Solvency wrote:
| You keep defending developers while I'm pointing the
| finger at corporate. But good victim complex.
| oytis wrote:
| Yes, but there's been a trend recently to cut costs on
| all engineering except AI. Might be related
| Turing_Machine wrote:
| Yep. I've also noted that the people making such claims
| never seem to cite their own work as an example of how to
| implement something at Facebook or YouTube scale that is
| less "brittle".
|
| Armchair quarterbacking isn't just a U.S. football
| phenomenon.
| eddof13 wrote:
| Google was acting up for me as well, so that could be
| jedberg wrote:
| Man the postmortem on this is gonna be fun.
|
| "Yeah so it turns out when Facebook and Instagram goes down so
| does Google"
|
| I do not envy the SREs at either company. I'm pretty sure all
| those other ones use Facebook or Google as their OAuth provider
| which is why they are all being reported as down.
| MandieD wrote:
| Additional fun factor: today is Super Tuesday - primary
| elections in a lot of US states.
|
| This outage will result in absolutely no ridiculous conspiracy
| theories.
| kurthr wrote:
| If your election integrity relies on Facebook, YouTube, or
| even DNS to be up... there are bigger issues.
|
| Actually, if all of them including Xitter went down, maybe
| things would get better? All the sunlight photons might get
| sucked in by too many eyeballs though, and there could be
| grass trampling.
| Turing_Machine wrote:
| > If your election integrity relies on Facebook, YouTube,
| or even DNS to be up... there are bigger issues.
|
| I agree. While I don't think it likely that Facebook or
| YouTube would enter into it, I'd pretty much bet that DNS
| being down would cause problems.
|
| And yes, there are bigger issues with that. Much.
| chakintosh wrote:
| Cloudflare or AWS ?
| nickpsecurity wrote:
| A lot of sites have features that say "log in with your X or Y
| account." They connect to each other somehow. I never studied
| that protocol. I wonder if authentication failures across
| services could be tied to it.
|
| For process of elimination, do all of these services do multi-
| platform logins? Or do some not connect to anyone else?
| dylan604 wrote:
| that actually infuriates me more than cookie banners. the one
| from Googs is the worst offender.
| tdudhhu wrote:
| Services are coming back to live.
|
| Interesting to see that all static content was still working
| during the outage (at least for Instagram). It was still
| possible to swipe through all reels (I assume the list was
| cached).
| yosito wrote:
| I think the content was probably cached on your device.
| partiallypro wrote:
| Down Detector is so unreliable. People that can't call an AT&T
| phone via Verizon will think (and report) that Verizon is down,
| when it's really AT&T. People can try logging in using
| Facebook's on click login and not be able to get in, so they
| think Tiktok is down. It's not all that useful. I hate when
| journalists cite it.
| dylan604 wrote:
| It's only slightly better than "my mom claims". My mom would
| ask if I had the internet at my house. Yup. all of it. in a
| rack in my bedroom closet. She'd also report the "internet is
| down" when a single website was having issues. To me, that is
| down detector carrying on the legacy of moms everywhere.
| jordanthoms wrote:
| A single report on there is useless. A sudden flood of
| reports is a good sign that something interesting is
| happening.
| ambichook wrote:
| and something is usually happening. the issue is that a
| lot of end users (so the people that down detector is
| pulling from) don't understand the systems well enough to
| point to _where_ that something is and will often
| misattribute it, which is what both parent and
| grandparent is claiming.
| jordanthoms wrote:
| It has false positives and noise for sure, but it's also very
| sensitive and shows issues very quickly.
|
| I wouldn't trust it as a single source, but in a case like
| this where our internal monitoring shows a spike of issues
| with the Google APIs and we can see a huge spike in reported
| issues for Google on Downdetector starting at the same time,
| it's useful to confirm that the issues have an external
| source.
| pixl97 wrote:
| Eh, mostly it's people misunderstanding what it represents.
|
| If I can't login to tiktok because FB is down, then tiktok is
| effectively down for me. When it comes to technology most
| people don't care about the trip, they care about the
| destination.
|
| So yea, tiktok isn't "down" but for a lot of people it might
| as well be, hence coupling your infrastructure/auth on other
| providers has side effects like this you must take into
| account.
| tonymet wrote:
| Facebook audience is in the billions, so you will see 100k
| false positives when a big site like that goes down.
| callalex wrote:
| Downdetector has successfully detected 150 of the last 20
| outages.
|
| Its mention should honestly be banned from this site.
| dzlobin wrote:
| This is just a knock-on affect from 1B+ users moving their
| timespent elsewhere during the outage
| blastadon wrote:
| Yeah, I had issues accessing GCP's documentation site for
| AlloyDB around 8:00-9:00ish Eastern this morning. The page just
| said 'Service Unavailable'. Had to use Google Cache.
| Vicinity9635 wrote:
| X (Twitter) wasn't affected:
|
| https://twitter.com/elonmusk/status/1765048551023734801
|
| Downdetector is user reports, not automated monitoring. It's...
| semi-trustworthy.
| chimeracoder wrote:
| In general, linking to an Elon Musk tweet is hardly proof
| that a Twitter service degradation _didn 't_ happen.
|
| Even more so when the tweet in question isn't even a direct
| claim about Twitter, but just a meme making fun of a
| competitor.
| TuringNYC wrote:
| All the electronic candy/soda vending machines in our office
| are also not working. Shudder to think of the chain of
| dependencies inside these machines.
| johntiger1 wrote:
| Ironically, Meta employees are also affected by this
| jeltz wrote:
| Just like last time they had an outage.
| lenerdenator wrote:
| Gonna have my Angle Grinder controller ready for the new
| Sysadmin Simulator release
| ototot wrote:
| They layoff too many ppl.
| fmsf wrote:
| I was considering buying the Quest 3 this morning, this outage is
| timely. The fact that I can't use my perfectly working headset to
| play an offline game because facebook is down, makes me wonder if
| I should go for a different provider. Any recommendations?
| Excluding apple vision pro since it is too expensive.
| practice9 wrote:
| Unless it's a recent change, it works perfectly fine offline
| (wifi turned off).
|
| As for alternatives, there is Pico, but Quest 3 may be superior
| in games selection. Or go wired which is of course less
| portable
| fmsf wrote:
| In my case I was doing iRacing with a cable, it requires the
| Oculus app to run for the link to work. Which in turn needs
| the login to be active.
| dartos wrote:
| I think you need to be online for the first setup of the
| headset, otherwise it won't work at all.
| IggleSniggle wrote:
| Yes. You login first, and then it completes device setup.
| At least, that's how it works for the Rift.
| extraduder_ire wrote:
| This is different from it being offline, it's like the device
| is kicked off the associated account. A "something went
| wrong" window pops up with a "generate device code" button,
| with instructions on how to remove and re-pair the headset.
|
| There is no way to even access wifi settings or anything else
| to disconnect the device from the internet. If it's still a
| problem much later in the day, I'll try turning off my router
| to see what that does.
| practice9 wrote:
| Interesting. I had a problem a few months ago with DNS not
| resolving Meta servers on my Starlink internet connection,
| but I was able to use the UI and the apps nonetheless, just
| couldn't open the store or update firmware.
|
| Seems like they really did change something in the latest
| firmwares.
| bloaf wrote:
| I think the current buzz in the VR space is the "Bigscreen
| Beyond" which eschews all of the nice-to-haves in order to make
| the headset as light as possible, and the result is
| surprisingly compelling.
| TulliusCicero wrote:
| It looks compelling for high-end PC gaming VR enthusiasts,
| but if GP is more of a mainstream consumer it probably won't
| make sense for them.
|
| At least from what I've read, there's a bunch of downsides
| for regular consumers: very expensive ($1000) -- SUPER
| expensive if you bundle in controllers and tracking points
| (~$1600), needs external tracking, wired instead of wireless,
| no built-in audio, can effectively only be used by one person
| (because each one is built custom to your face), and of
| course it's not a standalone headset, it has to be hooked up
| to a gaming PC.
| xp84 wrote:
| Great news: Thanks to Apple, $1600 is now a _budget super-
| economy_ VR headset!
|
| Courage.
| basilgohar wrote:
| As a principle I think it is a good idea to avoid any
| technically unneccessary coupling of hardware or software to
| something else that is not strictly user-servicing, else the
| priorities are inverted in favor of the vendor's priorities and
| not your own.
| CobrastanJorji wrote:
| I'm pretty sure the priorities are almost always inverted
| these days, unnecessary coupling or not.
| bdangubic wrote:
| I have been in the industry for the last 25 years and have
| done more "technically unnecessary" things than there are
| grains of sand on Earth :)
| waynesonfire wrote:
| why are you smiling about that?
| docmars wrote:
| Probably sarcasm because it's likely out of their control
| due to product & executive demands.
| auselen wrote:
| Why are you serious about that?
| nine_k wrote:
| People casually doing things that feel slightly evil
| is... disconcerting.
|
| Not going to judge anyone, but also nothing to be happy
| about.
| zer00eyz wrote:
| There is a point where you learn that everything is some
| sort of moral compromise. There is always going to be
| someone ahead of you and someone behind, it means someone
| is always going to get left out.
|
| No system changes that, or makes it better, just
| different.
| nine_k wrote:
| No system indeed, only your own choices.
| lewsid wrote:
| I'm typing this from Immersed in the Quest 3, no issues.
| TulliusCicero wrote:
| The Quest 3 is kind of the obvious mainstream consumer choice
| for a VR headset.
|
| Standalone wireless headset, reasonably powerful chipset, can
| optionally stream from a PC either wired or wireless, good
| optics/resolution, decent controllers/tracking, large game
| library, large suite of features (including hand tracking and
| color passthrough), all for a reasonable price. Not sure any
| other headset really competes on all those things at once.
| citrusybread wrote:
| can we really call it standalone if you need it to phone home
| to Meta? and doesn't it require an account one of their
| services too?
|
| they did a fantastic job with hardware. I just wish they
| didn't couple the software so tightly.
| bonton89 wrote:
| I know they raised the price recently but it seemed pretty
| obvious to me they were selling these at a loss to try and
| get people locked in by the software.
| docmars wrote:
| It makes me wonder how the world would be if monitor
| manufacturers did the same thing, it would be unacceptable.
|
| I view VR headsets and their peripherals as no different
| than a mouse, keyboard, and display.
|
| Companies requiring all this nonsense to _use_ your device,
| put in that light, is ludacris.
| hyuuu wrote:
| comparing it with a monitor is rather unfair, you have to
| bundle the computing along with it, not to mention the
| applications to make it an actually fair comparison. At
| that point, is it still ludacris?
| TulliusCicero wrote:
| A silly comparison. A standalone VR headset is more
| comparable to a smartphone or game console than a monitor
| or keyboard. The latter have little to no compute.
| ifyoubuildit wrote:
| So compute requires vendor lock in? That seems silly to
| me.
|
| Edit: Can we just acknowledge that a lot of the bells and
| whistles are for the companies benefit at the expense of
| the user? Thats their right, but it's also our right to
| want something better.
| TulliusCicero wrote:
| > So compute requires vendor lock in? That seems silly to
| me.
|
| Correct, it's very similar to game consoles, though it
| _is_ somewhat more open than those (sideloading is
| possible, including standard Android apps IIRC, and you
| can run PC VR games from other stores while tethered).
|
| > Can we just acknowledge that a lot of the bells and
| whistles are for the companies benefit at the expense of
| the user?
|
| It's the same model as XBox or Playstation, seems like.
| They sell the hardware at cost or at a loss, and make it
| up via software.
|
| A fully open headset with comparable specs would probably
| cost much more for the hardware. From a business
| standpoint that would be very stupid for a company like
| Meta, but this is hacker news, and many commenters here
| see nothing wrong or silly about asking businesses to
| commit suicide.
| ifyoubuildit wrote:
| > Correct, it's very similar to game consoles
|
| This doesn't explain why its _required_. It just means
| there is precedent.
|
| Your other point is better, although I think you mean it
| would cost the consumer more for the hardware, right? The
| hardware would cost the same to produce, it's just that
| the company would miss out on surveillance based revenue.
|
| It's a reasonable point, fb would make less money if they
| made an open headset, possibly to the point that they
| wouldn't make it all.
|
| But the world where fb doesn't make any headset, and the
| world where they make an unacceptable headset are
| basically equivalent to me - the former might even have
| an edge in that shitty relationships with corporations
| aren't being encouraged (like they are throughout
| everything tech related currently). Granted, them blazing
| the trail has a tiny chance of enabling a reasonable
| alternative to come along in the future.
|
| But I am a bit of a Luddite, and I know that people want
| their toys, and they want them now.
| TulliusCicero wrote:
| > the company would miss out on surveillance based
| revenue.
|
| More than likely most of Meta's revenue from the Quest
| series other than hardware is based off of, y'know,
| selling games. I doubt tracking what games you play to
| target ads in the OS is more valuable than the money they
| make when people actually buy games.
|
| In Facebook or Instagram, you're looking at a space that
| they can shoot lots of ads into, and it's otherwise very
| hard to monetize. But a gaming-focused VR headset is a
| different story. Most of the time you're not looking at
| anything that can have ads in it, but you _can_ actually
| sell stuff very easily.
|
| Maybe this'll change someday if they actually get social
| media shit in there that's popular, I'm sure Meta would
| love that, but so far that hasn't happened.
|
| > But the world where fb doesn't make any headset, and
| the world where they make an unacceptable headset are
| basically equivalent to me
|
| Popularizing the format is useful for pushing the tech
| forward. A big player pushing lots of devices means that
| the supply chains feeding the manufacture of those
| devices bulk up too, not to mention other knock-on
| effects like greater consumer awareness, and "free
| research" for whoever copies what the market leader does
| (at least for things that aren't IP-protected).
|
| > But I am a bit of a Luddite, and I know that people
| want their toys, and they want them now.
|
| I can hear the sneer from over here, yes.
| ifyoubuildit wrote:
| > More than likely most of Meta's revenue from the Quest
| series other than hardware is based off of, y'know,
| selling games. I doubt tracking what games you play to
| target ads in the OS is more valuable than the money they
| make when people actually buy games.
|
| Isn't that a great argument for why they don't need to
| have such a hard requirement for a logged in session?
| Consoles didn't have an internet connection for the
| longest time, though only because it wasnt feasible yet.
| They moved a lot of games.
|
| > I can hear the sneer from over here, yes.
|
| I don't mean it as judgment, I know I'm the weirdo here.
| Sorry if that came off rude.
| TulliusCicero wrote:
| > Consoles didn't have an internet connection for the
| longest time, though only because it wasnt feasible yet.
| They moved a lot of games.
|
| Consoles had physical games. VR headsets don't. Consoles
| treat digital games the same way Meta is doing them here,
| I think; if you get logged out, no more games.
|
| The problem here isn't that Meta servers are merely down
| -- losing connection usually doesn't mean losing access
| to your library of games on consoles, or Steam. The
| problem appears to be that authentication is failing such
| that you're actually being essentially logged out, which
| would definitely lose you access to digital games on
| every console as well as Steam.
|
| Which, I mean yeah, that's a big fuck-up on Meta's part.
| ifyoubuildit wrote:
| > Consoles had physical games. VR headsets don't.
| Consoles treat digital games the same way Meta is doing
| them here, I think; if you get logged out, no more games.
|
| Again, consoles and steam do this because they want to,
| because it benefits them, and consumers don't put any
| meaningful pressure on them for doing so. It's not some
| kind of fundamental requirement. It's helpful for e.g.
| anti piracy stuff, but not necessary. It is 100% feasible
| to sell me a digital copy of a game and then not hang
| around on my system and watch me play it.
|
| People let triple A PC games basically put rootkits on
| their systems. It's not like the games wouldn't work just
| fine (or better even!) without them. It's just that
| approximately nobody cares, and the companies will do
| whatatever you let them do.
| agar wrote:
| > I view VR headsets and their peripherals as no
| different than a mouse, keyboard, and display
|
| That could be valid when VR headsets were tethered to a
| PC via a DisplayPort or HDMI connection and essentially
| mirrored the display.
|
| The Quest is closer to an iPhone or Android phone or an
| all-digital handheld gaming device. With integrated
| compute, display, battery, text input, pointing devices,
| mic, and speakers, it bears little functional resemblance
| to peripherals like a mouse, keyboard, or display with no
| utility unless slaved to another device.
|
| Considering I can use my Quest with no wifi or other
| network to log in (once initial set up is complete), it
| seems that the Meta back-end APIs must have broke in some
| way that confused the headsets into thinking they were
| available when they weren't.
| TulliusCicero wrote:
| It sounds like a server-side bug that forced a log out
| somehow. Which does really suck, Meta deserves criticism
| for that, but acting like this means the headset "isn't
| standalone" is silly, since that's not what "standalone"
| means in the context of VR headsets.
| agar wrote:
| Agree, many posts I read seem like classic "I don't like
| Meta/VR/big companies/social media so let me use this
| specific incident to confirm my biases."
|
| As you say, there's valid criticism to be made but it's
| hard to find the signal through the noise.
| TulliusCicero wrote:
| > can we really call it standalone if you need it to phone
| home to Meta?
|
| Yes.
| ipaddr wrote:
| This doesn't work anywhere without internet service.
| Standalone it is not.
| TulliusCicero wrote:
| Not true. You can turn off the wifi and the headset works
| fine.
|
| The current problems sound like a server-side bug while
| it phones home. But usually it can work fine without
| internet.
|
| Standalone just means the VR compute is happening on the
| headset itself, not on a console or gaming PC the headset
| is tethered to. Of course, most of the people disputing
| "standalone" already know that, they're just playing
| definitional games.
| oven9342 wrote:
| I wonder how many people are secretly boycotting the
| product because of the prison Reputation of the maker
| TulliusCicero wrote:
| Oh there are definitely people who avoid it because it's
| made by Meta. Maybe a bunch. On the other hand, it seems
| to be the most popular VR headset line by a wide margin.
|
| It would be cool if Valve came out with a standalone
| headset, they're one of the few companies I can see that
| would be in a good position to do that: they already have
| a good amount of VR experience with one high-end headset
| + SteamVR APIs + a couple VR games, they have their own
| highly popular store/platform, they generally have a
| positive reputation with gamers, and they have a decent
| amount of hardware experience in general including the
| recent Steam Deck for mobile gaming hardware
| specifically.
|
| And of course, a Valve headset would probably be
| significantly more open than the Quest. The Steam Deck
| has gotten some good reputation among more FOSS/hacker-
| oriented people for being fairly open: you can use it in
| a regular Linux desktop mode, you can install Windows (or
| presumably other OSes) on it, it's fairly repairable,
| etc. The default behavior is very console-like, but it's
| not very locked down if you don't want it to be. Best of
| both worlds, really.
| ben_w wrote:
| > can we really call it standalone if you need it to phone
| home to Meta?
|
| Yes, because etymologically it's "standalone" vs. "wired";
| this is akin to how phones are "mobile" vs. (when I was a
| kid) "landline".
| TulliusCicero wrote:
| Nitpick: strictly speaking it's standalone vs tethered
| (which is _usually_ wired, but could be wireless).
| beardbound wrote:
| For what it's worth it does require a meta account, but not
| a Facebook one. I refused to buy one while it required a
| Facebook account since I deleted a couple of years ago.
| Once they made the change I figured that was an OK
| compromise. I just found out today during the outage that
| my headset won't work if I get signed out of my Meta
| account. That was an unpleasant realization, although I
| suppose it's partially my fault for trusting Meta not to
| hamstring the hardware their selling.
|
| It's the equivalent of finding out that if Microsoft's auth
| servers go down no one with a Windows PC can use it since
| they can't authenticate. I'm fairly displeased.
| Analemma_ wrote:
| The parent's comment is about how it quite literally is not a
| standalone headset, as is currently being demonstrated.
| TulliusCicero wrote:
| It is a standalone headset. You don't need a tethered
| gaming PC or console for the Quest series of headsets.
| schmorptron wrote:
| I thought that too, but it seems like you need a tethered
| cloud PC somewhere at meta to open apps after all.
| TulliusCicero wrote:
| Only because there's a bug, seems like. Normally you can
| turn off your wifi entirely and the headset continues to
| work fine. Tried disconnecting entirely in a non-
| standalone headset and see how that works out for you.
|
| Standalone just means you don't need to tether to a PC or
| console.
| nine_k wrote:
| Standalone, but not autonomous.
| DonHopkins wrote:
| Lucky you didn't already have a Quest 3, and you were in
| CyberSpace when Facebook crashed, because then you would be
| trapped in CyberSpace and die. That's how it works, you know.
| idle_zealot wrote:
| I can confirm that I was unable to use my Quest 3 this morning.
| I left it connected to the internet, it tried to phone home I
| guess, and then locked itself into a "please connect this
| headset to your Meta account" state.
|
| I am so sick of companies "selling" computers that they
| continue to control. In what universe does Meta have the right
| to remotely lock _my_ headset and prevent me from using it to
| run the software I installed on it? If I were to sell my
| current desktop computer, or phone, or whatever, on any
| marketplace, and leave a remote login account on it that I then
| used to continue to operate the computer as though it were mine
| remotely, installing software, playing games, and occasionally
| peeking at what the current owner was doing, that would be
| obviously criminal. How is this any different? Because I signed
| away my rights when I "agreed" to their Terms and Conditions
| box (which I was compelled to do to use the hardware I
| purchased)?
|
| Something is so fundamentally broken in the current
| ownership/property landscape. We somehow ended up in a world
| where people don't own the most critical tools in their lives,
| companies have managed to recreate feudal fiefdoms within the
| bounds of the market.
| rixthefox wrote:
| Ubisoft Director of Subscriptions really opened the
| floodgates of bad behavior when they came out saying "Gamers
| need to get comfortable with not owning their games".
|
| I think these companies need to be reminded they do not own
| our PCs either.
|
| I'm really starting to like that mantra of "If buying isn't
| owning then piracy isn't stealing".
| ljm wrote:
| The full context of that quote, which everyone conveniently
| omits, is _"for subscriptions to take off"_.
|
| As in, gaming subscription services like Xbox's GamePass
| won't succeed if gamers prefer to buy games over paying a
| monthly sub.
| bonton89 wrote:
| I can understand your frustration but were you not aware of
| the software lock in when you bought it? I'm not defending
| the ownership erosion, but I avoided these things
| specifically because of who was selling it and how it was
| locked to them.
| IggleSniggle wrote:
| Not OP but I bought a Rift when it was still just Oculus.
|
| ...then Facebook bought Oculus...
|
| ...and then _required_ you to have a Meta account to
| continue using the Oculus drivers.
|
| It's a real "boil the frog" strategy and this is still
| early days for VR in terms of realized market value. The
| time to push back on this bullshit is yesterday. As we can
| all see, nobody can compete with Meta on price with the
| Quest 3, but the cost to purchase is heavily subsidized by
| the expected futures.
| bonton89 wrote:
| If you bought a Rift before facebook purchased them I
| wouldn't call it boiling the frog, more like being
| stabbed in the back. Not much to do there but sell your
| device but I guess most people probably hoped things
| would turn out differently than they did. This is one of
| the most infuriating parts of America now, if you hate a
| company and never want to interact with them some merger
| comes along and throws you into being their customer
| again against your will.
|
| Of course, OP owns a Quest 3 so its more cut and dry
| there.
| idle_zealot wrote:
| I was aware. They are the only game in town when it comes
| to standalone VR. I want to play BeatSaber, a game I
| purchased when I used an Oculus Quest, and the only way to
| do that now is by subjecting myself to Meta's whims. I
| compromise on my ideals to have nice things, but will
| continue to complain when I feel that I or others have been
| wronged.
| d00nut wrote:
| Steam Index is great if you don't care about cords / needing a
| PC. Easy to do VRchat, IRacing, or even blade & sorcery game
| sessions for 3-4+ hours without any eye strain, headaches,
| motion sickness, or discomfort from the headset.
|
| It also fits over / around glasses
|
| Biggest reason not to IMO is of the rumors around an "index2".
| CoastalCoder wrote:
| Seconded, _I think_. I 've been very happy with my Valve
| Index, but I don't know how it compares to newer headsets on
| the market.
|
| More broadly, I find that Valve represents my interests far
| more than Facebook/Meta does. So I'd much rather send my
| money to Valve.
| munk-a wrote:
| I'll third this - Valve is really unobtrusive about the
| steam related features of the index. It does have some
| requirements with steam for setup but if you want to run a
| local binary and mess around with dev tools it's extremely
| easy to do. It's also extremely well sealed and designed -
| I tend to sweat a lot and a few times I've been beat
| sabering for quite a while without any long term damage to
| the headset.
| cesarb wrote:
| > Biggest reason not to IMO is of the rumors around an
| "index2".
|
| Another big reason not to buy a Steam Index is "not available
| in your country". The only VR headsets I've seen in stores
| here are the Quest 2/3 and the PSVR2; and the Steam store
| page for the Valve Index (and the Steam Deck) says "not
| available in your country".
| darrenf wrote:
| I used my Quest 3 for a couple of gaming breaks during the
| outage with no issues.
| YeBanKo wrote:
| I could not play a steam game on a PC in offline mode when my
| internet was down. This issue is not exclusive to fb.
| samtheprogram wrote:
| Have never had this problem. There must have been something
| specific to your scenario, like the game wasn't downloaded,
| or it's a specific title?
| IggleSniggle wrote:
| That's not a Steam thing though, but rather the specific
| software. Steam explicitly has an exit path for the user if
| Valve disappeared overnight that allows their downloaded
| games to continue working offline.
|
| The difference being that we are discussing platforms, not
| the things that run on those platforms.
| YeBanKo wrote:
| Steam Offline wants you to go online and perform a bunch of
| steps, including launching games and then enabling offline
| mode. Every game launch is is probably a download of at
| least few hundred megabytes of data. And then every game
| requires its own networks where your account linked to
| steam acc, etc. and Rockstar games iirc it you must be
| online when you launch the game. So the fact that Steam
| client has offline mode is irrelevant and misleading.
| IggleSniggle wrote:
| That has not been my experience. Any online game is going
| to be its own thing, dependent on the choices of the game
| company. Inherently, Steam does not require all the steps
| you're describing.
|
| Sidenote, but my experience with the Rockstar launcher
| has been absolutely atrocious, to the point that I just
| avoid rockstar at this point even though I'd otherwise be
| interested because I've been burned so many times. That's
| a Rockstar issue, not a Steam issue.
| paulryanrogers wrote:
| Doesn't Steam still offer an offline mode that works with
| most games?
| callalex wrote:
| It does offer an offline mode. It does NOT work with most
| games, because the publishers literally can't help
| themselves but add more layers of DRM on top of steam DRM
| and most of those these days require always online
| connections.
| cesarb wrote:
| In my experience, the Steam offline mode only works if your
| computer is actually offline (without any network
| connection); if you're connected to the local network but
| the Internet connection from your router is down, it still
| tries to connect to the authentication servers while
| starting up.
| YeBanKo wrote:
| Rockstar games require you to have an account at the launch
| iirc. And this us kind misleading, because while
| technically Steam has offline mode, but not necessarily the
| games you purchased on Steam. But having a unified UX is
| why I want to use the platform like steam to begin with.
| cesarb wrote:
| Were you still connected to your local network? Next time it
| happens, try completely disconnecting from your local network
| before starting Steam, it seems to use the presence of a
| local network connection to decide whether to enable offline
| mode or not.
| ZealousFerret wrote:
| Quest is the best option right now
| makestuff wrote:
| Is there any reason to get the quest pro over the quest 3? Or
| should I wait for a quest pro refresh?
| callalex wrote:
| Whatever you do, get one of the all-in-ones that doesn't
| require setting up tracking beacons. They are universally a
| pain and will cause you to never bother playing.
| m3kw9 wrote:
| You can't have everything. If you want a VR headset you
| described, there may not be any good ecosystem of software yet
| and you'd have to wait. This means you are gonna use your
| headset even less, think about how often fb goes down, it goes
| down for 2-3 hours once a year and it also has to coincide with
| the time you are using it? It doesn't make sense to be this
| risk adverse
| squidsoup wrote:
| The Sony PSVR2 is getting PC support by end of year apparently.
| zer0zzz wrote:
| This is pretty cool. I might get one if I'm convinced it's
| better than my quest3s display link.
| ahupp wrote:
| There may be a bug or change since I left, but I built the app
| library and authorization logic, and it was explicitly designed
| to work offline. Of course, using it day-to-day and initial
| setup are different and I'd imagine if Apple is down it's hard
| to setup an AVP as well.
| EMCymatics wrote:
| Offline gaming should never have been destroyed.
|
| Wonder what data they collect from gamers.
| Jemm wrote:
| I was able to use my Quest 3 throughout the outage for offline
| games.
| xp84 wrote:
| imho this is a stance you certainly have every right to take,
| but good luck. If you want to be part of the world of things
| like VR, smartphones, etc. then refusing (on principle) to
| participate in things like "accounts" and "cloud" is going to
| cost you far more time than the number of hours a massive
| company like Meta may have downtime. Likewise, yes, at some
| undetermined future date a lot of this hardware will become a
| complete doorstop due to their supporting servers being taken
| offline, however again, if you are doing this advanced gadget
| thing, it'll be long after you would have decided to upgrade to
| new hardware anyway because it can't do any of the latest
| stuff.
|
| (and _yes_ , there are ways if you're devoted enough, to roll
| your own everything and run Linux on a Framework laptop, and
| use some kind of custom ROM on your phone without Google
| anything, 3d print yourself a VR headset, etc. But all of this
| will cost you several orders of magnitude more time than Meta
| outages ever would.
| wojciem wrote:
| Llama3 become sentient
| rany_ wrote:
| The status page seems down for me as well. I just get a never-
| ending loading screen.
| dudeinhawaii wrote:
| If you have an adblocker, try disabling that. The page fails to
| load with ublock Origin running.
| XCSme wrote:
| What could affect so many different services? Some DNS server?
| Clould provider like AWS?
| jgehrcke wrote:
| Underwater cables seemingly got attacked; the Red Sea situation
| is likely related: https://apnews.com/article/red-sea-undersea-
| cables-yemen-hou...
| epgui wrote:
| Timeline doesn't seem to line up
| jeltz wrote:
| Probably the authentication serer being down.
| nonrandomstring wrote:
| Lovely bit of cloaked dramatic irony in "Leave The World Behind"
| t=1:11:20 [0]
|
| Ruth (Myha'la): This is for the better.
|
| G.H (Mahershala Ali): For who?
|
| [0] https://en.wikipedia.org/wiki/Leave_the_World_Behind_(film)
| dmk wrote:
| Google login also seems to be having issues, multiple people
| reported to me that the login isn't working and they've been
| logged out of their Google accounts.
| ExoticPearTree wrote:
| Yes, I tried logging in today in two distinct Google accounts
| on separate Chrome profiles and it would sign me out in about ~
| 5 seconds after logging in. And the login process was very
| sluggish.
| slowhadoken wrote:
| YouTube is down too. Only thing working is Twitter.
| bookofjoe wrote:
| YouTube works fine for me in Charlottesville, Virginia, U.S.A.
| slowhadoken wrote:
| It's working again. It's been experiencing outages since 6am
| I guess.
| coffedean wrote:
| YouTube seems okay here. East coast US.
| jgehrcke wrote:
| Might be related to this event?
|
| > The cut lines include Asia-Africa-Europe 1, the Europe India
| Gateway, Seacom and TGN-Gulf, Hong Kong-based HGC Global
| Communications said. It described the cuts as affecting 25% of
| the traffic flowing through the Red Sea.
|
| https://apnews.com/article/red-sea-undersea-cables-yemen-hou...
|
| https://www.bbc.com/news/world-middle-east-68478828.amp
| suyash wrote:
| time to move away from undersea cables to satellites.
| BearOso wrote:
| LEO satellites would be too inconsistent, and further orbits
| have way too much latency.
| pksebben wrote:
| We have satellites. We use cables b/c they lack the speed and
| bandwidth necessary to support the total requirements of the
| modern internet. Satellite-only is only feasible if you're
| fine with going back to waiting minutes for your saucy jpegs
| to load (elder millennials, you know what I'm talkin' about).
| suyash wrote:
| ever heard of Musk's Starlink? From thier website "Starlink
| users typically experience download speeds between 25 and
| 220 Mbps, with a majority of users experiencing speeds over
| 100 Mbps" -
| https://www.starlink.com/legal/documents/DOC-1400-28829-70
| SSLy wrote:
| I was thinking is deploy of DMA "compliant" unbundling the day
| before it takes force.
|
| Could be both.
| bhawks wrote:
| > BY JON GAMBRELL Updated 8:25 PM PST, March 4, 2024
|
| Timelines don't match nicely
| danbala wrote:
| maybe a long tail consequence of further shifting traffic?
| hsuduebc2 wrote:
| This shouldn't affect europe. It just stopped working
| wongarsu wrote:
| Somebody might have fat-fingered a BGP configuration while
| trying to improve traffic routing that was impacted by the
| cut cables.
| hsuduebc2 wrote:
| Yea I thought too that core of this is not at the services
| itself but at thr network somewhere.
| elorant wrote:
| It's not unreachable. I can easily see the FB page on my
| browser. It's just that even after resetting my password it
| doesn't accept it. Probably something's fucked up in the
| credentials database.
| ajross wrote:
| Those lines were cut yesterday, so it seems like a poor
| candidate for explaining the current outages. Likewise the
| geography doesn't match up with the outages.
| toast0 wrote:
| I'm pretty sure that report is just mainstream media reporting
| this week old cut
| https://www.datacenterdynamics.com/en/news/at-least-one-subs...
|
| Pretty neat if a week after a cable is cut, FB falls over.
|
| Especially when most of the source of truth databases are in
| the US and Europe, and that sort of data flow doesn't cross the
| Red Sea. FB has datacenters and points of presence all over,
| but outside the US/EU it's almost all caching.
| jgehrcke wrote:
| Thanks for the reference. Yeah, may also be entirely
| unrelated.
| dylan604 wrote:
| > Pretty neat if a week after a cable is cut, FB falls over.
|
| that'd be one helluva cache!!
| passwordoops wrote:
| I'm 95% sure it'll turn out to be a mundane config error
| somewhere
| barkingcat wrote:
| It will be DNS related.
| lvncelot wrote:
| Handy troubleshooting tool for outages:
| https://isitdns.com/
| dylan604 wrote:
| reminds me of http://iscaliforniaonfire.com/ even if not
| directly related to topic at hand
| Infernal wrote:
| The sysadmin's haiku:
|
| It's not DNS
|
| There's no way it's DNS
|
| It was DNS
| newsclues wrote:
| Nothing of value was lost.
| deadbabe wrote:
| Entire businesses and livelihoods would crumble without Meta's
| platforms.
| newsclues wrote:
| They can migrate to other platforms.
| binaryblitz wrote:
| What platforms have anywhere near the reach and services
| that Meta does for a marketplace/advertising?
|
| I'm not personally a fan of most social media, but saying
| "they could go somewhere else" is a pretty naieve/ignorant
| response.
| tjpnz wrote:
| If your business only exists at the sole discretion of Meta
| then it's not a viable business, at least not in the
| longterm.
| aloe_falsa wrote:
| Technically true, just as if you replace Meta with Amazon,
| Google Cloud, Etsy, eBay, the local city council, etc. If
| you don't have a diversified, multi-market, preferably
| international presence, you're not viable in the long term.
|
| Of course, in the really long term, we're all dead.
| Meanwhile, the local mom & pop candy shops keep advertising
| and selling online and making ends meet.
| bluetidepro wrote:
| On Super Tuesday in the US today, too. Very interesting timing
| for how much political nonsense is on social media.
| stevefan1999 wrote:
| This is so funny because the status page itself actually shows
| 500 Internal Server Error to me in the API calls. So the status
| page clearly isn't isolated from the FB network itself. I highly
| suspect it is either a BGP convergence issue, or their OIDC
| service hits the dirt.
| hsuduebc2 wrote:
| Funny. This page won't work too and is clearly down. So there is
| possiblity of pretty big amateurism from facebook or failure at
| dns servers.
| bigwheeler wrote:
| Tip for Meta engineers: when your service is failing, don't just
| log people out and prevent logins. Display a cute image that
| shows that the service is drastically failing (like a whale or
| something), and then people will know to stop trying to
| repeatedly log in. The public might even come up with a catchy
| name for the whale.
| brk wrote:
| This is a great idea! I know that I was flailing about trying
| to figure out why I couldn't login, so I'd suggest calling it a
| "Flail Whale".
| bandwidth wrote:
| Surely the repeated logins can't be helping the situation. I
| suppose it is entirely auth related across all Meta products.
| The repeated strain could pose a cold start problem for
| example.
| zabzonk wrote:
| > catchy name
|
| Petunia?
| boring_twenties wrote:
| Beyond unbelievable that going on an hour later, they're still
| showing "incorrect password" errors. How many hundreds of
| millions of people have wasted time frantically trying (in
| vain) to reset their passwords and pointlessly freaking out
| that their account might be compromised? What a bunch of
| careless, incompetent excuses for engineers.
| alwyn wrote:
| Blame the managers and product owners that don't think this
| is an issue, not the developers that likely raised it a
| million times already.
| ConorSheehan1 wrote:
| What a poor bunch of overworked human beings, with almost no
| control over the product they work on. Frantically following
| the whims of managers, reduced to labour units in this late
| stage capitalist hellscape.
| jeltz wrote:
| But well paid at least. Working at Meta seems pretty shitty
| except for the pay from the stories I have heard.
| loeg wrote:
| It probably depends what team you're on, but I would not
| describe it as "pretty shitty." Being oncall for a 24/7
| service sucks, yeah, but for my team it is one week a
| quarter and I haven't had any outside-of-biz-hour alarms
| the last few shifts. Other than that -- my work is
| challenging and interesting, my colleagues are friendly
| and smart, and my manager is decent. Not a lot to
| complain about.
| colund wrote:
| That's such a beautiful comment I would almost consider
| printing it and putting it on my wall
| nuz wrote:
| Imagine how many hundreds of millions of users waste their
| time _using_ instagram and facebook on a daily basis. Safe to
| say they don 't mind wasting their customers time
| cratermoon wrote:
| This is your regular reminder that the users are not their
| customers. Advertisers and media outlets are the paying
| customers.
| matsur wrote:
| Major outages are periods of intense stress and extremely
| difficult to operate in. The folks troubleshooting may be
| many things, but careless and incompetent are unlikely to be
| among them.
| Solvency wrote:
| I can almost guarantee you're getting mercilessly downvoted
| because half of the people here are sympathetic Meta
| worshippers who desperately (1) wish they worked there and
| (2) know they'd probably contribute similarly to this same
| horribly engineered system.
| newman8r wrote:
| How many other people here just assumed they had been banned
| for some arbitrary, uncontestable reason? I just use instagram
| to post hiking photos...
| TonyTrapp wrote:
| Having had a look at desperate Twitter posts during a major
| outage of a big German email provider with similar failure
| mode (login failed silently), it seemed like many people
| assumed that their email account was hacked. Close enough.
| sumitgt wrote:
| Could be an outage in the auth service itself
| Pxtl wrote:
| Unless it's an outage in their ability to log into their own
| servers, they should be able to swap out the login page with
| a static HTML page explaining the outage. Maybe a 503 status
| code.
| remram wrote:
| Yes of course it was. The point is, an hour later, they could
| have hit a circuit-breaker to get people to stop trying and
| going crazy over an error that is completely inaccurate.
| Pxtl wrote:
| Right? I know I was like "oh, I haven't typed in my FB password
| in eons... maybe I changed it at some point and forgot? But if
| I change it what happens to all related services, is it going
| to log out my kids' Messenger Kids devices? Those are such a
| pain to log in. Should I change my password or not? What do I
| do?"
|
| Then I saw the news that it was an outage.
| zerr wrote:
| For anything outside like-ing and post-ing, facebooks UI/UX is
| horrendous. Even Internet search does not help to find out how
| something trivial is done... The only way is to watch youtube
| videos.
| dspillett wrote:
| _> For anything outside like-ing and post-ing, facebooks UI
| /UX is horrendous._
|
| It isn't perfect for even that IMO.
|
| _> Even Internet search does not help to find out how
| something trivial is done... The only way is to watch youtube
| videos._
|
| That says more about where the web is heading than about
| facebook. Video is easier to monetise ATM0, and these days
| people don't put helpful stuff out there just to be helpful
| as much as they once did1, so content creators are making
| them instead of simple web pages.
|
| --
|
| [0] Everyone wants to be the next big influenza who doesn't
| need a day job to get by.
|
| [1] That sort of people are still out there, though they are
| somewhat drowned out as the signal-to-noise ratio heads
| inexorably towards "WHAT? WHAT?! I can't hear a single thing
| above the manscaping adverts!".
| bluetomcat wrote:
| > Display a cute image that shows that the service is
| drastically failing (like a whale or something), and then
| people will know to stop trying to repeatedly log in.
|
| Probably not so easy to implement in behemoth apps, consisting
| of 20'000 source files...
| mkl95 wrote:
| What do you mean my login flow sucks? Its time complexity is
| really good!
| kossTKR wrote:
| Cloudflare also having issues as of 1 hour ago:
|
| https://www.cloudflarestatus.com
|
| I wonder if it's connected to this. Seems like most large players
| are having issues now.
| magk wrote:
| Cloudflare reporting that they were implementing a fix for
| something SSO-related at 16:02.
|
| https://www.cloudflarestatus.com/
| annoyingnoob wrote:
| I don't use Meta's services but I have a number of co-workers
| that are all convinced that their Meta accounts have been hacked.
| AtNightWeCode wrote:
| SSO in Cloudflare also seems to have problems.
| hsuduebc2 wrote:
| It looks like AWS is down if we can trust downdetector which is
| reason why nothing works. Almost fifty percent of these services
| runs on this cloud.
| sammy2255 wrote:
| We can't trust downdetector. Its just user reports. And users
| dont know anythig
| cdme wrote:
| Eh, leave them down.
| shoebham wrote:
| hackernews is also v slow for me
| xseparator wrote:
| You know shit's bad when even their status page is down.
| hsuduebc2 wrote:
| Well if everything down maybe isn't problem at services itself
| but at the DNS side?
| vb-8448 wrote:
| Seems like even the status page doesn't work ... i'd say typical
| for this type of situations :D :D :D
| buserror wrote:
| I can confirm that IRC still works! Pfewwwww..... /me takes his
| coat.
| op00to wrote:
| unless your bouncer is offline!
| corobo wrote:
| buserror has quit (*.net *.split)
| alliao wrote:
| /me put on my wizard robe
| guhcampos wrote:
| Yeah so much for graceful feature degradation.
| jawns wrote:
| Just last week, the Meta paper about their Defcon system landed
| on HN front page.
|
| https://www.micahlerner.com/2023/07/23/defcon-preventing-ove...
| bookofjoe wrote:
| I find it remarkable that Typepad, my blog host, not only isn't
| down but also is MUCH FASTER than its usual slothlike,
| approximately 5 seconds response time.
| peterleiser wrote:
| Fixed. I can login to facebook.com now.
| nuz wrote:
| HN also seems unusually slow. Wonder what's going on
| Deukhoofd wrote:
| HN is always slow when everything else goes down, tech people
| swarm here to check for information, and it overloads the site.
| Etheryte wrote:
| Happens every time a major outage happens, everyone pours onto
| HN to discuss it which gives the one-core server HN runs on a
| really hard time.
| belter wrote:
| Probably DNS ...it's always DNS....
| thinkingemote wrote:
| BGP is my guess
| joejoesvk wrote:
| good luck to those on call
| btbuildem wrote:
| Hope it stays broken. Imagine? People would blink, confused, and
| slowly start to wake from a decade-long stupor.
| kome wrote:
| how many people, realistically, use it daily? I think not a lot
| explain wrote:
| Really?
| lenerdenator wrote:
| Boomers probably do.
| thehappypm wrote:
| Like, everyone I know is on TikTok all the time and Reels is
| right up there
| gpvos wrote:
| Lots and lots. Look at how many people are in this thread.
| vincnetas wrote:
| couple billions.
| randomnumber314 wrote:
| >Facebook: global daily active users 2023.
|
| >Statista https://www.statista.com > statistics > facebook-
| global-dau Feb 9, 2024 -- During the fourth quarter of 2023,
| the number of daily active users on Facebook reached 2.1
| billion, a minor increase on the previous quarter.
|
| So roughly 28% of the planet.
| randunel wrote:
| Facebook accounts != individual humans.
| pests wrote:
| Even if half that, still crazy, no?
| AntiqueFig wrote:
| He's quoting the daily active users, not the number of
| accounts.
| cmpxchg8b wrote:
| Enough for Meta to be a $1.2 Trillion company. What a silly
| comment.
| baby wrote:
| Man I would lose contact with a lot of friends if that were to
| happen
| pdabbadabba wrote:
| ...and then install TikTok.
| bookofjoe wrote:
| My Meta Ray-Ban Stories glasses are working just fine FWIW
| bratwurst3000 wrote:
| lol how ?
|
| https://en.m.wikipedia.org/wiki/2024_Facebook_outage
| kome wrote:
| I am surprised by how many people still use Facebook around here,
| as my social circle deleted or stopped using Facebook a long time
| ago.
| belter wrote:
| Internet Outages Map - https://www.thousandeyes.com/outages/
| worldwidelies wrote:
| Is it coincidence that Meta is trying to get portions of NSOs
| source code via the court system. Also today is apparently a big
| voting day. The techno-apocalypse is near.
| didip wrote:
| Laid off too many people? Critical services no longer have
| owners?
| ur-whale wrote:
| Since we're in full-on speculation mode ... Russian cyber-attack
| ?
| tinycoder25 wrote:
| Status page show some services are recovering. I wonder why ad
| services is the first service to be mentioned as recovering
| instead of login.
| hengistbury wrote:
| No reason to allow people on the platform if you can't monetise
| them, especially after monetisation has been paused for a few
| hours
| ricardobayes wrote:
| Oh, that's why I was logged out then.
| solfox wrote:
| Seeing such a large web property go down like this is
| fascinating.
|
| It's like a power plant grid failure, except for _attention_
| instead of energy.
|
| When meta is down, a hoard of internet users desperately seek
| somewhere else to place their attention... But the system is
| designed with the expectation that meta will take all that
| traffic... And boom! everything starts falling over. Wild.
| Vicinity9635 wrote:
| Classic failure casdcade.
| alwaysrunning wrote:
| It's DNS
| nayuki wrote:
| Note that this web page only covers the status of Meta's
| offerings for business users. This doesn't track the downtime of
| Facebook, Instagram, WhatsApp, Messenger, etc. as normal users
| experience them.
| jeanlucas wrote:
| Curiously, WhatsApp did not suffer an outage in Brazil.
| FerretFred wrote:
| > Ads Manager - Recovering from disruptions
|
| LOLLLL .. oh dear
| fuzzfactor wrote:
| At least it's not a mission-critical website . . .
| srvmshr wrote:
| I have an active Instagram account. Today, coincidentally was the
| first time I thought of promoting my best few posts as an ad to
| improve reach to fellow photographers. Bad luck! The app now
| seems to be back online but my ads which were supposed to run for
| 48-72 hours, show <Disabled>. And there is a new "Pay Now" link,
| even though it was paid for and seems ad-spend is already showing
| it used some of that payment.
|
| As an individual, this is pretty confusing. I don't have much to
| lose. I am glad I spent 10-15 bucks on favorite 2-3 posts only. I
| can imagine many others to be more affected. What is normally to
| be expected for SME users? Does Meta resume the ads
| automatically? Do they make good for the lost time since the
| clock seems to be ticking -- although no one saw any impression.
|
| Edit: I submitted a ticket to Instagram Help and they responded
| by asking for a screencast video. The first time I sent, the
| video bounced. I have re-sent this by trimming the video.
|
| Out of curiosity I want to know firsthand how Meta handles the
| small customers.
| Babadas wrote:
| Everyone Beware, The "Black Hat Hackers" that have accomplished
| this massive Global hack represent a Clear and Present Danger,
| immediate and significant threat/danger to National Security, as
| well as all People's personal and financial Security
| Babadas wrote:
| Everyone beware, The "Black Hat Hackers" that have accomplished
| this massive Global hack represent a Clear and Present Danger,
| immediate and significant threat/danger to National Security, as
| well as all People's personal and financial Security
| kosolam wrote:
| Why the outage?
| ragmondo wrote:
| Doesn't this kind of thing happen in every film regarding an AI
| becoming sentient ??
| uselpa wrote:
| For now, some countries preparing a larger war seems more
| likely.
| YeBanKo wrote:
| So I guess this is what that national security warning and
| purchase of Palo Alto Networks stocks was all about in mid
| February.
| WobblyTyre wrote:
| Can you elaborate further?
| Razengan wrote:
| Since my dedicated post about this got no response, I want to use
| this opportunity to ask:
|
| WHERE the HELL does Facebook store tracking data on my iPhone?
|
| It shows my previous account even after I delete the app, clear
| the cache and KeyChain, disable iCloud Drive, AND sign out of
| iCloud??
|
| Why can't I see where this data is stored? Same for TikTok.
|
| WHY does Apple, parading around as a pompous paragon of privacy,
| allow this bullshit?
| ak_111 wrote:
| I have a very tangential question to this situation prompted by
| some conspiracy theories linking the outage to today being Super
| Tuesday election day in the US.
|
| Why does Facebook need sharing news media or political ad
| targeting on its platform from a business point of view? Am I
| being naive or is it really such an important revenue driver to
| the business or the core experience of the app?
|
| I think it is the source of enormous reputational damage and
| risk, that if I was running the company I would even happily
| trade 10% if not more of the market cap to removing any feature
| that enables news sharing on the platform.
|
| Actually if you remove these two things (political ad targeting
| and commenting on news media) I struggle to find any other issue
| that would make facebook a "political" target, they can literally
| shutdown the fake news division that employs 10,000 people...
| cheeseomlit wrote:
| The ability to affect the outcome of elections is very much
| worth the political flak for them
| dtdynasty wrote:
| My guess would be that they would lose a significant amount of
| their user base and engagement without politics and news media.
| stemlord wrote:
| And nothing of value was lost
| perlpimp wrote:
| probably some sort of BGP flood.
| m3drano wrote:
| wild speculation: they were rolling out enforcement for DMA.
| baby wrote:
| this is the message I'm getting on X now:
|
| https://www.threads.net/@cryptodavidw/post/C4JRaX9vZYE
| seb1204 wrote:
| So why is this site metastatus.com and not status.meta.com as
| mentioned in an earlier article here the east amount of domain
| names these big corps have is not helping with making sure this
| is not a scam. So why metastatus.com and not metastatuscheck.com?
| Sure that the buffs at Mets could come up with an independent
| machine that can show the status even if everything else is down.
| pvg wrote:
| Edit: I'm leaving the comment as penance but as
| https://news.ycombinator.com/item?id=39610130 points out, this is
| an old message about an old update, not the current one.
|
| A short update from meta with some initial high level technical
| details:
|
| https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A...
| HenriTEL wrote:
| So it's a bug of the "automated remediation that makes things
| worse" family.
|
| This system that checks for invalid values in the cache looks
| like a very bad idea in the first place as in my understanding
| it checks things beyond "is the cache up to date?".
| gene31337 wrote:
| That's from 2010.
|
| edit:
| https://www.cnn.com/2010/TECH/social.media/09/24/facebook.ou...
| pvg wrote:
| You're absolutely right, thanks, my mistake trusting random
| URLs on HN/the internet!
| pilgrim0 wrote:
| Interesting how such large scale outages can ONLY happen because
| of human errors, in this case by a poorly thought out heuristics.
| Like, a literal explosion could hardly achieve this level of
| disruption when there's physical failover protections everywhere.
___________________________________________________________________
(page generated 2024-03-05 23:01 UTC)