[HN Gopher] Error 404 (Not Found)
___________________________________________________________________
Error 404 (Not Found)
Author : fairytale
Score : 391 points
Date : 2021-11-16 17:41 UTC (5 hours ago)
(HTM) web link (spotify.com)
(TXT) w3m dump (spotify.com)
| addcninblue wrote:
| It looks like everything is back now. That was a short outage by
| recent standards...
| caffeinated_me wrote:
| My company was seeing GCP Airflow environments not responding,
| but they seem to have recovered in the past few minutes.
| jaredeklee13 wrote:
| Global: Experiencing Issue with Cloud networking
|
| Incident began at 2021-11-16 10:10 (all times are US/Pacific).
|
| https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
| [deleted]
| dave_aiello wrote:
| Right now homedepot.com and the APIs that drive their mobile app
| are down too.
| humanistbot wrote:
| The title of the 404 page on all the down sites has an extra "1"
| after the exclamation points: "Error 404 (Not Found)!!1"
| decrypt wrote:
| That has been the case for years!
| jaywalk wrote:
| That's just a cutesy Internet meme thing.
| jaredeklee13 wrote:
| Global: Experiencing Issue with Cloud networking Incident began
| at 2021-11-16 10:10 (all times are US/Pacific).
| https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
| fsflover wrote:
| Time to switch to https://bandcamp.com.
| casi18 wrote:
| or https://ipfs.io/
| dymk wrote:
| Cool, how do I stream the newest Taylor Swift album on that?
| iso1631 wrote:
| My colleague (who loves Taylor Swift) bought the mp3s from
| somewhere (amazon?) and uploaded them onto her plex server
| the hour it came out.
|
| That server continues to work just fine.
| cheeze wrote:
| As a heavy plex user, I can't imagine using it as my
| default music player. CX isn't great for music, IMO.
| [deleted]
| benbristow wrote:
| You write the domain (ipfs.io) on a blackboard with a chalk
| pen then take your nails and scratch the board.
|
| (jk.)
| BraveryOverload wrote:
| DAE THINK POP MUSIC BAD?!
| cronix wrote:
| Seems to be fixed.
| soheil wrote:
| Funny thing is when you google Home Depot or Paramount Plus you
| get ads served by Google as the first result. When click on it
| Google then shows you a 404 page. I wonder if they'll get a
| refund on their Adwords campaign.
| makecheck wrote:
| One of my pet peeves with so many services! Their obnoxious
| pre-ads can play flawlessly (stealing your time/eyeballs and
| giving them benefit), and they can _still fail_ to give you the
| content you exchanged your time /eyeballs to see. Worse, they
| can repeatedly fail and repeatedly drill the same ads into your
| brain.
|
| There ought to be a law that essentially says if ads are
| "paying" for content, there _must_ be a flawless link between
| ads and content such that the system can tell if the content is
| available (or detect after the fact that something was not
| delivered properly). And then, based on that, it either is
| required to ensure the ad _never plays_ (since the content
| cannot be delivered), or that the user _must_ be compensated in
| some way (e.g. we see you were forced to see an ad but got
| nothing so we are crediting $1 to your account).
| progbits wrote:
| Why? That's not part of the ad contract. They will get refunded
| for GCP if it goes out of SLO.
| terramex wrote:
| Etsy is 404 too: https://www.etsy.com
|
| Seems to be a bigger issue.
|
| edit: Nest is down too: http://nest.com
|
| Fitbit.com is 404 too: https://www.fitbit.com
|
| Big GCP issue?
|
| edit2: Downdetector.com shows multiple website and services as
| down, including Pokemon GO or Rocket League.
|
| GCP status page is still green all over the board:
| https://status.cloud.google.com
|
| 19:10 CET update: Some websites are coming back, including
| spotify.com, but their app still does not work for me.
|
| information about outage just added to GCP status page, direct
| link:
| https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
|
| _Description: We are experiencing an issue with Cloud Networking
| beginning at Tuesday, 2021-11-16 09:53 US /Pacific._
|
| _Our engineering team continues to investigate the issue._
|
| _We will provide an update by Tuesday, 2021-11-16 10:40 US
| /Pacific with current details._
|
| _We apologize to all who are affected by the disruption._
|
| 19:20 CET update:
|
| _Description: We believe the issue with Cloud Networking is
| partially resolved._
|
| _Customers will be unable to apply changes to their load
| balancers until the issue is fully resolved._
|
| _We do not have an ETA for full resolution at this point._
|
| _We will provide an update by Tuesday, 2021-11-16 11:28 US
| /Pacific with current details._
|
| Spotify desktop app still not working for me.
|
| 19:45 CET: Spotify app is back online for me.
| whymauri wrote:
| Status board got updated:
| https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
| Nextgrid wrote:
| I'm assuming all these websites point to Google web frontend
| servers and for some reason it's no longer able to map the Host
| header to the proper backend to proxy to.
| profmonocle wrote:
| For what it's worth, not only HTTP(S) load balancers went
| down. We have a couple TCP proxy load balancers that went
| down too.
| notreallyserio wrote:
| I was seeing regular levels of health checks throughout the
| entire outage, so they still had configs in place. That seems
| plausible.
| jeffbee wrote:
| This is a sort of repetitive outage for Google. They've wiped
| out the GSLB configs before. A year ago there was also that
| big outage where they blanked out the whole Gmail delivery
| configs and started rejecting all mails (even for gmail.com).
| Config safety is not their strong suit.
| bobo_legos wrote:
| Homedepot.com appears down as well.
| parthdesai wrote:
| discord seems to be down as well
| [deleted]
| iso1631 wrote:
| Another cloud issue? Wow, nearly a week's uptime.
| https://news.ycombinator.com/item?id=29197832
| hk1337 wrote:
| Egnyte had a pretty major outage today too.
|
| https://status.egnyte.com
| ly wrote:
| When Spotify went down I switched to Youtube Music. Funny how
| that's still running but Spotify isn't, if it is in fact a
| Google issue.
| progbits wrote:
| Internal Google infra and GCP are different beasts. Sure some
| failure modes take both down but more likely than not they
| are independent.
| Jugurtha wrote:
| I have things running on GCP that are OK. I think it's related
| to nameservers.
| profmonocle wrote:
| It is (was?) definitely a load balancer issue. We had
| multiple load balancers on completely different DNS providers
| fail.
| cscheid wrote:
| https://deno.land down too. I guess a lot of CI actions will be
| failing just like ours...
| sholladay wrote:
| deno.land is working for me, while lots of other sites are
| not. Did they failover to another provider?
| profmonocle wrote:
| It seems things are gradually coming back. All of our load
| balancers started working again.
| cscheid wrote:
| Working now, yes. Though deno.land's current IP still
| nslookup's to something inside googleusercontent.com so
| maybe google has fixed their Side. But it was definitely
| offline earlier today (cf. our github CI failures and
| downforeveryoneorjustme)
| XCSme wrote:
| All work for me in the EU apart from Etsy.
| nuccy wrote:
| Parts of google are also down it seems, if one clicks three
| lines on the top right on google.com in mobile browser, then
| the https://www.google.com/mobile/?newwindow=1 opens, which
| also appears 404 not found.
| uubk wrote:
| We found extra rules in our GCLB routing config - removing them
| restored our service.
| pcbro141 wrote:
| https://downdetector.com/
|
| yup
| neelr01 wrote:
| Its back up now :)
| hs86 wrote:
| https://toggl.com/ is also affected.
| aalbertson wrote:
| Lowes search was also down.
| _nickwhite wrote:
| 1:10PM - either it has resolved itself, or a regional issue, but
| I'm not seeing anything being down from the East coast USA.
| mcintyre1994 wrote:
| Looks like it's made it to the Google status page:
| https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...
| humanistbot wrote:
| Sites that are down according to https://downdetector.com include
| Spotify, Discord, Snapchat, Etsy, Pokemon Go, Epic Games, Target,
| Paramount+, Evernote
| scame-miv wrote:
| I was experienced this issue with my spotify app. Initially
| thought, it was my internet issue lol.
| neelr001 wrote:
| Its back up now :)
| neelr001 wrote:
| its back up :)
| NaughtyShiba wrote:
| Seems to be back already
| saranshk wrote:
| Our instances started working again, so seems to be fixed
| algorithm_dk wrote:
| This is clearly the hottest thing on HN right now, and it was
| bumped from #1 to #6, anyone knows why? Is it some kind of bot
| protection?
| floatingatoll wrote:
| User flags, because outages are a fact of everyday life.
| mbesto wrote:
| Which is dumb, linking to status pages shouldn't be on HN. A
| blog that has analysis and explanations of outages or post
| mortems should.
|
| knock-knock dang
| floatingatoll wrote:
| Dang doesn't see messages like that unless you use the
| footer Contact link, but I remember a comment from him a
| while back that I would summarize as "Some site users think
| it's a good use of HN, and other site users disagree and
| flag it, and we downweight/dedupe them sometimes and/or if
| someone emails us with the Contact link". I just didn't
| want you to wait for a reply that'll never come unless you
| Contact them.
| MarcScott wrote:
| Ironically it appears that IsItDownRightNow? is also down,
| although that could because they're experiencing what is
| basically going to be the equivalent of a DDOS.
| judge2020 wrote:
| downdetector.com is the most resistant in my experience.
| Causality1 wrote:
| I wonder if they have advertising methods in place to turn days
| like this into a windfall.
| collinmanderson wrote:
| See also https://news.ycombinator.com/item?id=29243740
| markbnj wrote:
| We were down. Just came back. Things seem to be resolving.
| soheil wrote:
| I also wonder how many companies didn't want to admit they were
| using Google for their infrastructure. Downdetector shows AWS
| being affected, it'd be embarrassing if they were caught using
| Google Cloud Platform.
| grumple wrote:
| Seriously doubt that AWS, Facebook are using Google for infra.
| There's probably some other effect at play, like people using a
| Google service to connect to these things. Also don't see any
| effects on those services personally.
| johanam wrote:
| https://overleaf.com/ is also 404 now
| milesward wrote:
| It's back up
| [deleted]
| thegranderson wrote:
| Seems like another DNS issue - switching to Cloudflare 1.1.1.1
| got me back online...
| deberon wrote:
| GCP outage? Status page shows green but a bunch of sites seem
| down (Rocket League most importantly).
| carlhjerpe wrote:
| Most importantly this game I play for leisure is down!
| NaughtyShiba wrote:
| I mean, that's a fair point... leisure activity, whatever it
| is, is required for rest and to avoid burndown
| mosen wrote:
| I think it was a joke.
| ukd1 wrote:
| We're also affected.
| ukd1 wrote:
| and back up and running!
| Jugurtha wrote:
| My stuff is running OK on GCP, with GKE usage. Maybe it's related
| to nameservers?
| deforciant wrote:
| if you are using regional load balancers or serving traffic
| directly from nodes then you would be fine :) "only" global LB
| failed
| jacobkg wrote:
| We bypassed our Google load balancer and pointed DNS directly at
| the IP addresses of our servers and that seems to have helped
| IceWreck wrote:
| https://www.navidrome.org/
|
| Self hosted Spotify. Compatible with subsonic clients.
| NaughtyShiba wrote:
| On one hand, Spotify is much cheaper, on other, perhaps artistd
| gets paid more (assuming you acquire music legaly)
| dyeje wrote:
| Discord seems to be down.
| contrahax wrote:
| Seeing the same - I have projects in us-east1 that went offline
| first, then us-west1 went offline a few minutes after. Everything
| green on their status page and nothing in the dashboard -
| everything returns a 404 so I'm assuming a really high level LB
| just took a dump.
| Jansin3 wrote:
| Rocket league perhaps epic games even
| ksajadi wrote:
| Half the internet is down because of a Google Cloud global issue
| on their load balancers, including Spotify and Etsy and GCP
| status is all green: https://status.cloud.google.com If you ever
| wondered why GCP is a distant third runner in the enterprise
| cloud space, here is your answer.
| yuliyp wrote:
| Expecting an instant public post is a bit unrealistic. They had
| a post up just over 20 minutes after the incident start, which
| is not that crazy, given that they needed time to triage all of
| the alarms and understand which component was actually breaking
| and confirm some technically correct information around it,
| even if the actual internal incident response can run without
| the public post.
| corobo wrote:
| It's Google though.. they stop automating everything?
|
| At least make the screen not show all green or something
| automatically
| yuliyp wrote:
| Which things should not be green?
|
| If the automation is working the services will be up. When
| an incident is happening it's because _something is
| significantly broken_ , and automation won't properly
| understand what is and is not working.
|
| For instance, lots of follow-on alarms might be firing for
| what are not actually issues with the things being
| monitored: As an example, I would imagine that datacenter
| temperatures and fan speeds dropped due to the incident,
| which might cause automation to suspect a facilities issue,
| but announcing a facilities issue would be misleading.
|
| Or metrics around instances live might be tanking as
| autoscaling groups start downsizing. This would not be an
| issue with the autoscaling service, and automatically
| announcing an autoscaling outage would again be misleading.
|
| In an incident, taking the available data and reaching a
| conclusion about what is broken and what are effects is
| something which requires skilled manual effort and is
| error-prone.
| gowld wrote:
| Monitoring should be on a different system, unaffected by
| an outage in the monitored system.
| iso1631 wrote:
| I'm old enough to remember the old claim that the internet was
| designed to cope with a nuclear attack.
| deathanatos wrote:
| Azure has the same issues with updating their status page.
| Sometimes it never happens.
|
| I might at least hold out some chance that Google Cloud will
| write an interesting PM, which is something Azure would never
| do IME.
| yupper32 wrote:
| The big three cloud platforms all have this issue of delayed
| status updates. Why do you think it's just GCP?
| mostdataisnice wrote:
| ...and there's a reason for that. Automations to update the
| status page are rarely acceptable, since the status page
| statuses have legal and financial implications. Therefore,
| the IM usually has to update it (or tell someone to update
| it). But, realistically, when you get paged, you first need
| to figure out what exactly is wrong and at least a vague idea
| of why. Then, you need to tell someone to update the page.
| Then, it gets updated.
|
| The status page will _always_ lag the outage. It 's not a
| conspiracy.
| kazen44 wrote:
| Also, in most teams, people who do external communication
| are different from those doing triage and troubleshooting.
| cheeze wrote:
| Yeah, but they are still people who are responding to a
| page, working on wording and getting it approved, and
| then updating.
|
| 20 minutes seems pretty reasonable to me.
| gowld wrote:
| Not updating the status page, to avoid the legal and
| financial implications, is fraud -- taking money on false
| pretenses.
| deathanatos wrote:
| Status pages _should_ be driven that way, though. "legal
| and financial" implications and "It's not a conspiracy" is
| a poor excuse.
|
| Now, I'm on Azure, but it seems like from the comments the
| situations are similar. So, instead of an automatically
| updated status page that would help engineers do their
| jobs, we get a status page that isn't accurate, and
| customers have pull teeth to get a service credit
| where/when one is due. And it seems like you can have the
| cake and eat it too here: while IANAL, a footnote in the
| SLA or the status page that "this is a machine estimate and
| not reflective of what goes into the SLA" should do it, no?
| brown9-2 wrote:
| AWS typically has the same issue
| [deleted]
| ishikawa wrote:
| it was back in a few minutes. But that wasw pretty weird. App
| Engine was affected.
| deforciant wrote:
| https://linear.app/ is also down
| lenniez wrote:
| Also experience that with all my GCP related infrastructure
| (Europe)
| mixedbit wrote:
| Looks like perhaps an issue with Google Load Balancer. We have a
| load balancer in front of Google Storage Buckets, and can access
| resources directly from the buckets, but getting 404 when going
| through the load balancer.
| te_chris wrote:
| Yep, it's the GLB. Went down for us at 5:46 GMT. Just
| responding 404 and logs reporting an internal error.
| neom wrote:
| Non-engineer here - Is there an easy way to multi-provider
| redundancy around this? Can you have LBs on multiple clouds and
| use dns to move around or something? Or does your LB have to be
| at the provider the app is at? Sorry if this makes no sense. :o
| sparrc wrote:
| yes it's 100% technically possible, the main issue is it
| would be significantly more expensive.
| cinericius wrote:
| I wonder if a legal discovery will ever find internal status
| dashboards that reflect reality rather than fictitious SLA
| liability-aware status pages.
| hmrr wrote:
| Your cynicism is justified and clearly based on the same
| experience I have :)
| paxys wrote:
| You don't need legal discovery for that. Every "X as a service"
| contract you sign will explicitly say that SLAs aren't
| dependent on dashboards/ping tests but rather a mostly
| subjective measure of "availability".
| melling wrote:
| I noticed Discord went down for me.
| ruined wrote:
| nice
| arjan_sch wrote:
| The 404's changed into 502's.. I guess that's progress. Fingers
| crossed it's back up soon
| dustinmoris wrote:
| Things are back online again!
| lukeschlather wrote:
| As far as I can tell everything is up it's just that our load
| balancers aren't routing traffic and just returning 404s.
| 1cvmask wrote:
| Is it regional? Surprised Spotify is not active-active on other
| platforms like AWS and Azure.
| Thaxll wrote:
| It's back!
| christophclarke wrote:
| Snapchat also having issues refreshing
| benbristow wrote:
| Snapchat runs on the Google Cloud.
| davidkuennen wrote:
| My app was down too. Can confirm it is most likely the GCloud LB
| Borrible wrote:
| Yes, but we're still on DEFCON 5.
| [deleted]
| cglace wrote:
| Everything seems to be working on our end as of 1:08 PM EST.
| colewilson wrote:
| looks like it's back up?
| cronix wrote:
| back up as of now (10:10a PDT)
|
| This sure makes it easy to know who is hosted by google by going
| to downdetector.com.
| jakub_g wrote:
| Github graphql API seems to be throwing error responses for me
| still
| jonnylangefeld wrote:
| Hmm it shows the same spike for instagram and aws as well.
| Would be funny to me if something on their end depends on GCP.
| https://downdetector.com/status/aws-amazon-web-services/
| https://downdetector.com/status/instagram/
| Jenk wrote:
| Google Analytics registering as client fails, maybe?
| judge2020 wrote:
| Not exactly hard to do that in the first place
| whois $(dig +short spotify.com A) NetName:
| GOOGLE-CLOUD
| uvdn7 wrote:
| That only says spotify.com is using GCP's DNS but not
| necessarily for hosting?
|
| EDIT: I was wrong, as pointed out by shizcakes.
| uvdn7 wrote:
| While we are at it, looks like spotify.com is using a mix
| of NS1 and GCP for DNS.
|
| spotify.com. 172800 IN NS dns1.p07.nsone.net.
|
| spotify.com. 172800 IN NS dns2.p07.nsone.net.
|
| spotify.com. 172800 IN NS dns3.p07.nsone.net.
|
| spotify.com. 172800 IN NS dns4.p07.nsone.net.
|
| spotify.com. 172800 IN NS ns-cloud-a1.googledomains.com.
|
| spotify.com. 172800 IN NS ns-cloud-a2.googledomains.com.
|
| spotify.com. 172800 IN NS ns-cloud-a3.googledomains.com.
|
| spotify.com. 172800 IN NS ns-cloud-a4.googledomains.com.
| shizcakes wrote:
| That's not what that says. It's doing a whois on the
| hosting IP address, not the DNS service.
| [deleted]
| uvdn7 wrote:
| Ah you're right! Thanks!
| fairytale wrote:
| Can confirm. https://spotify.com is now working fine.
| corobo wrote:
| Can unconfirm. I can see their website but the app isn't
| playing nicely at all
|
| Any non-offline playlists/songs just sit there not playing or
| telling me I'm offline
|
| edit @ 37min: App also seems to be working again now
| crdrost wrote:
| Yeah, per the status page, there's a temporary mitigation
| rolled out while the team tries to figure out more... so no
| more 404s but load balancers will be locked down for a while
| while investigation continues.
| te_chris wrote:
| Affecting us. Busiest time of the year and now down 20 min. It's
| the Global Load Balancer, so god knows what bit of the global
| edge has been taken out.
| iso1631 wrote:
| At least you don't run your own servers
| oussama-gmd wrote:
| Same here, seems to be an issue with load balancers
| DiFronzo wrote:
| Oh okay, thought something was on my end.
___________________________________________________________________
(page generated 2021-11-16 23:01 UTC)