[HN Gopher] GCP Outage
___________________________________________________________________
GCP Outage
Author : thanhhaimai
Score : 1223 points
Date : 2025-06-12 18:11 UTC (4 hours ago)
(HTM) web link (status.cloud.google.com)
(TXT) w3m dump (status.cloud.google.com)
| thanhhaimai wrote:
| The status page is green, but there are outages reported:
| https://downdetector.com/status/google-cloud/
| ransom1538 wrote:
| Why can't companies be honest with being down. It helps us all
| out so we don't spend an hour internalizing.
|
| We are truly in gods hands.
|
| $ prod
|
| Fetching cluster endpoint and auth data. ERROR:
| (gcloud.container.clusters.get-credentials) ResponseError:
| code=503, message=Visibility check was unavailable. Please
| retry the request and contact support if the problem persists
| jeanlucas wrote:
| Because there are contracts related to uptime :)
| rustc wrote:
| Does any service even say they're "down" anymore? All I see
| is "elevated error rates".
| colechristensen wrote:
| 4 to 6 hours after the flames are visible from orbit and
| management has finally given up on the 37th quick fix you
| do get that red X
|
| But really not until after it's been on CNN a while.
| rixthefox wrote:
| Those contracts will be monitoring their service
| availability on their own. If Google can't be honest you
| can bet your bottom dollar the companies paying for that
| SLA are going to hold them accountable if they report the
| outage properly or not.
| datadrivenangel wrote:
| The real point of SLAs is to give you a reason to break
| contracts. If a vendor doesn't meet their contractual
| promises, that gives you a lot of room to get out
| contracts
| 9rx wrote:
| The program that updates the status page is hosted on Google
| Cloud.
| ashu1461 wrote:
| So even then, it should have been able to correctly report
| the status, it somehow shows that the status page is not
| automated and any change there needs to go through someone
| manual.
| 9rx wrote:
| A program that updates the status page failing does not
| imply that the status page is manually edited. It is not
| like you would generate a status page on every request.
| ashu1461 wrote:
| How do we know that the program is failing ?
|
| How hard is it for the frontend to detect if the last
| update to the status page was made a while ago and that
| itself implies there is an error and should be reported ?
| rapus95 wrote:
| the services ARE healthy, status page is correct. The
| backbone which links YOU to the service isn't healthy.
| Take a look at cloudflare, they are already working on it
| ikiris wrote:
| Not even close. The status page is manual and cloud
| flares outage is because of Google not the other way
| around.
| tfsh wrote:
| It's not. You might be joking, but that comment still isn't
| helpful.
|
| My understanding is this is part of Google's internal PSD
| offering (Public Status Board) which uses SCS (Static
| Content Service) behind GFE (Google Frontend) which is
| hosted on Borg, and deploys other large scale apps such as
| Search, Drive, YouTube, etc.
| oxymoron wrote:
| Because a lot of the time, not everyone is impacted, as the
| systems are designed to contain the "blast radius" of
| failures using techniques such as cellular architecture and
| [shuffle sharding](https://aws.amazon.com/builders-
| library/workload-isolation-u...). So sometimes a service is
| completely down for some customers and fully unaffected for
| other customers.
| Eduard wrote:
| > Because a lot of the time, not everyone is impacted
|
| then such pages should report a partial failure. Indeed the
| GCP outage page lists an orange "One or more regions
| affected" marker, but all services show the green
| "Available" marker, which apparently is not true.
| deepsun wrote:
| There's always a partial outage in large systems, some
| very small percentage. All clouds should report all red
| then.
| hnuser123456 wrote:
| "there is a 5% chance your instance is down" is still a
| partial outage. A green check should only mean everything
| (about that service) is working for everyone (in that
| region) as intended.
|
| Downdetector reports started spiking over an hour ago but
| there still isn't a single status that isn't a green
| checkmark on the status page.
| spwa4 wrote:
| Just say it: they want to lie to 95% of customers.
| deepsun wrote:
| With highly distributed services there's always something
| failing, some small percentage.
| jobs_throwaway wrote:
| That is still 100% an outage and should be displayed as
| such
| johannes1234321 wrote:
| They still could show that so.e.issues exist. Their
| monitoring must know.
|
| The issue is that they don't want to. (For claiming good
| uptime, which may even be true for average user, if most
| outages affect only small groups)
| rozap wrote:
| Please, won't somebody think of the KPIs.
| kingstnap wrote:
| Because they have unrealistic targets so they make up fake
| uptime numbers. 99.999% would mean not even having an hour of
| downtime in 10 years.
|
| I remember reddit being down for like a whole day or so and
| they claimed 99.5% in that month.
| wbl wrote:
| Ma Bell hit that decently often.
| Uehreka wrote:
| Is that even knowable? Like, I know they called it "The
| Astonishing, Unfailing, Bell System" but if they had an
| outage somewhere did they actually have an infrastructure
| of "canary phones" and such to tell in real time? (As in,
| they'd know even if service was restored in an hour)
|
| Not trying to snark, I legit got nerdsniped by this
| comment.
| wbl wrote:
| They absolutely did. Note that the reliability estimates
| exclude the last mine because trees falling and the like
| but they had a lot of self repair, reporting, and
| management facilities.
|
| Engineering and Operations in the Bell System is pretty
| great for this.
| Dylan16807 wrote:
| Running a much simpler system with much more independent
| nodes.
|
| It's a lot easier to keep packets flowing than to keep
| non-self-contained servers serving.
| voytec wrote:
| > Why can't companies be honest with being down
|
| SLA agreements.
| organsnyder wrote:
| Any customer with enough leverage to negotiate meaningful
| SLA agreements will also have the leverage to insist that
| uptime is not derived from the absence of incidents on
| public-facing status pages.
| rapus95 wrote:
| if half the internet is down, which it apparently is, it's
| usually not the service in question, but some backbone
| service like cloudflare. And as internal health monitoring
| doesn't route to the outside through the backbone to get back
| in, it won't pick it up. Which is good in some sense, as it
| means that we can see if it's on the path TO the service or
| the service itself.
| supportengineer wrote:
| Nobody gets a promotion, that's why.
| DrBenCarson wrote:
| Whichever product person is in charge of the status page should
| be ashamed
|
| How could you possibly trust them with your critical workloads?
| They don't even tell you whether or not their services work
| (despite obviously knowing)
| FireBeyond wrote:
| Yeah, my company of hundreds of people working remotely are
| having 90%+ failures connecting to Google Meetings - joining a
| meeting just results in a 504.
| nerdsniper wrote:
| Why even have a status page? Someone reported that their org of
| >100,000 users can't use Google Meet. If corps aren't going to
| update their status page, might as well just not have one.
|
| https://www.google.com/appsstatus/dashboard/
|
| https://status.cloud.google.com/index.html
|
| Edit: The GCP status page got updated <1 minute after I posted
| this, showing affected services are Cloud Data Fusion, Cloud
| Memorystore, Cloud Shell, Cloud Workstations, Google Cloud
| Bigtable, Google Cloud Console, Google Cloud Dataproc, Google
| Cloud Storage, Identity and Access Management, Identity
| Platform, Memorystore for Memcached, Memorystore for Redis,
| Memorystore for Redis Cluster, Vertex AI Search
| supportengineer wrote:
| Who gets a promotion from a working status board?
| SOLAR_FIELDS wrote:
| There's no situation where the corporation controls the
| status page where you can trust the status page to have
| accurate information. None. The incentives will never be
| aligned in this regard. It's just too tempting and easy for
| the corp to control the narrative when they maintain their
| own status page.
|
| The only accurate status pages are provided by third party
| service checkers.
| the8472 wrote:
| > The incentives will never be aligned in this regard.
|
| Well, yes, incentives, do big customers with wads of cash
| have an incentive to demand accurate reporting from their
| suppliers so they can react better rather than trying to
| identify issues? If there's systematic underreporting, then
| apparently not. Though in this case they did update their
| page.
| staplers wrote:
| If there's systematic underreporting, then apparently
| not.
|
| You answered your own question.
| SOLAR_FIELDS wrote:
| In practice how this plays out is that the big wads of
| cash holders will make demand, and Google (or whoever,
| Google is just the standin for the generic Corp here)
| will give them the actual information privately. It will
| still never be trusted to be reflected accurately on the
| public status page.
|
| If you think about it from the corp's perspective, it
| makes perfect sense. They weigh the risk reward. Are they
| going to be rewarded for the radical transparency or
| suffer fall out by acknowledging how bad of a dumpster
| fire the situation actually is? Easier for the corp to
| just lie, obscure and downplay to avoid having to even
| face that conundrum in the first place.
| paulddraper wrote:
| > might as well just not have one
|
| This is my position.
| nikcub wrote:
| I have zero faith in status pages. It's easier and more
| reliable to just check twitter.
|
| Heroku was down for _hours_ the other day before there was
| any mention of an incident - meanwhile there were hundreds of
| comments across twitter, hn, reddit etc.
| fooey wrote:
| anecdotally, the status pages have been taken away from
| engineering and are run by customer support and marketing
| milesward wrote:
| It's updated now, shows the impact to console, dataproc, GCS,
| IAM and Identity Platform:
| https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
| jorts wrote:
| Here's the incident:
| https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
| deathanatos wrote:
| It was nearly an hour into our company's internal incident
| channel on this for GCP to finally declare that yes, in fact,
| things on fire.
|
| ... I get that PR-types probably want to massage the message,
| but going radio dark is not good PR.
| matdehaast wrote:
| Not just GCP, most of Googles services are out of action
| milesward wrote:
| I'm on a meet, in cal, editing a dozen docs, in GCP, pushing
| commits and launching containers; it's not clear yet what
| exactly is going on but it's certainly intermittent and sparse,
| at least so far
| parpfish wrote:
| stop it. you're overloading their system by doing three
| things at once. let the rest of us have a turn.
| sourthyme wrote:
| Maybe cloudflare?
| nickporter wrote:
| having issues with cloudflare as well
| aetherson wrote:
| Cloudflare status page reports an issue:
| https://www.cloudflarestatus.com/
| aetherson wrote:
| We're experiencing intermittent slowness and timeouts on our GCP
| everything.
| dorkitude wrote:
| Same here. Even the page to submit support requests is down.
|
| Cloud console does nothing.
|
| They should host their support services on AWS and vice-versa.
| milesward wrote:
| I just logged into several of my GCP accts, everything popped
| up, multiple home regions.. I wonder what % of folks are
| feeling this right now.
| aetherson wrote:
| Does anyone know if it's region-specific? We're experiencing it
| and are in us-west-1.
| data-ottawa wrote:
| Us-central-1 as well
| vbb wrote:
| europe (netherlands) region as well
| izolate wrote:
| us-east-1 too
| johannes5117 wrote:
| Frankfurt seems to be down as well
| mccoyc wrote:
| Can confirm us-east1 (and possibly us-south1) are having VPC
| host reachability problems.
| NeT8235 wrote:
| south korea as well
| tridao wrote:
| it's due to IAM and global
| ea016 wrote:
| Google Cloud Storage seems to be down or very slow
| digest wrote:
| love how their status page is green with no issues detected!
| pbmango wrote:
| https://www.canva.com/design/DAGqKquGD-c/xtRObgH1r_4RoulPAys...
| hambro wrote:
| @dang could you merge this and
| https://news.ycombinator.com/item?id=44260669?
| toomuchtodo wrote:
| No notifications for mentions, have to email the mods at the
| hn@ email address.
| hambro wrote:
| I think I was a bit optimistic in the response time from
| mods. This thread won the popularity contest quite well...
|
| Thanks for letting me know about emailing the mods,
| refreshingly explicit to send email.
| cwillu wrote:
| Do we know if email is still working? kidding-but-not-really-
| because-gmail...
| niij wrote:
| Experiencing 504s in Google Meet.
|
| Google Cloud Console won't load.
| ransom1538 wrote:
| Yeah their status page is all green nothing to see here (but all
| production systems are down).
| alexcroox wrote:
| Cloudflare KV is also having an outage. I wonder who is reliant
| on who here.
| hackermondev wrote:
| seriously doubt Google Cloud is relying on Cloudflare KV lol
| dlewis1788 wrote:
| Looks like more than KV is having an issue. Just tried to load
| dash.cloudflare.com and no bueno.
| eterm wrote:
| Cloudflare speedtest is down too, I assume because of this?
| voxadam wrote:
| Works for me in Portland on Quantum Fiber.
| clairegraham wrote:
| Our site depends on Workers and KV and it's very broken right
| now. Can't login to the Cloudflare Dashboard either.
| gfs wrote:
| Appears to be a separate incident:
| https://news.ycombinator.com/item?id=44261064
| reassess_blind wrote:
| Two big cloud provider outages at the same time? Has to be
| related surely.
| siliconc0w wrote:
| Gemini API isn't working for me :/
| conroy wrote:
| We're in us-west-1 and seeing issues across Cloud Run, Cloud SQL,
| Cloud Storage and Compute Engine.
| oalessandr wrote:
| Having issues with services in cloud run as well
| zacharynewton wrote:
| Ahhh, explains why some of my apps are going crazy... Couldn't
| read a message from my kids pre-school
|
| Thankfully we use AWS at work for everything critical
| atonse wrote:
| Getting a lot of errors for Claude Sonnet 4 (Cursor) and Gemini
| Pro.
|
| Nooooo I'm going to have to use my brain again and write 100% of
| my code like a caveman from December 2024.
| bicx wrote:
| I was in the middle of testing Cloud Storage file uploads, so I
| guess this is a good time to go for a walk.
| matsemann wrote:
| A good excuse for adding error handling, which otherwise is
| often overlooked, heh.
| orangebread wrote:
| lmao i refuse to write code by hand anymore too. WHAT IS THIS
| sujayakar wrote:
| switch to auto mode and it should still work!
| ashu1461 wrote:
| GPT is working in agent mode, which kind of confirms that
| claude is hosted on google and GPT probably on MSFT servers /
| self hosted.
| kenhwang wrote:
| If you want a stronger confirmation about Claude being
| hosted on GCP, this is about as authoritative as it gets:
| https://www.anthropic.com/news/anthropic-partners-with-
| googl...
| scottmf wrote:
| Claude runs on AWS afaik. And OAI on Azure. Edit: oh okay
| maybe GCP too then. I'm personally having no problem using
| Claude Code though.
| burntalmonds wrote:
| Same here. Getting this in AI Studio: Failed to generate
| content: user has exceeded quota. Please try again later.
| crocowhile wrote:
| openrouter.ai is down for me
| sunir wrote:
| I chose sepuku.
| tough wrote:
| hang in there.
| Xavez wrote:
| Apple's local models looking better each day :')
| nolist_policy wrote:
| Google's local models as well (Gemini Nano/Gemma 3n)
| ilc wrote:
| How do you run Gemma 3n locally?
| n0mer wrote:
| https://github.com/google-ai-
| edge/gallery/releases/tag/1.0.3
| cryptonector wrote:
| Devs before June 12, 2025: "Ai? Pfft, hallucination central.
| They'll never replace me!"
|
| Devs during June 12, 2025 GCP outage: "What, no AI?! Do you
| think I'm a slave?!"
| atonse wrote:
| 100% agree... I even thought "ok maybe I'll clean up the
| backlog while I wait" but I'm so used to even using AI to
| clean up my JIRA backlog (using the Atlassian MCP), so even
| that feels weird to click into each ticket, just the way I
| used to do it TWO MONTHS AGO.
|
| This is a good wake-up call on how easily (and quickly) we
| can all become pretty dependent on these tools.
| tough wrote:
| local llm's would work
| thefourthchime wrote:
| So true
| sva_ wrote:
| It appears like "Devs" is not a homogeneous mass.
| robin-a wrote:
| Cursor throwing some errors for me in Auto Agent mode too.
| gigatexal wrote:
| YouTube is also very flakey.
| Brystephor wrote:
| some core GCP cloud services are down. might be a good time for
| GCP dependent people to go for a walk, do some stretches, and
| check back in a couple hours.
| mlb_hn wrote:
| Our GCP is down
| milesward wrote:
| What region?
| ashu1461 wrote:
| I think multiple regions are down. asia-south, us-east
| atleast are impacted.
| a_void_sky wrote:
| asia-south is working for me
| sergiotapia wrote:
| xAI having problems, Supabase down, Discord can't upload images
| to share in chat. Seems like a major backbone outage.
| saltcod wrote:
| We're investigating right now. Looks like a potential issue
| with Cloudflare.
| atsaloli wrote:
| https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
| faizanrupani wrote:
| You're right, https://www.cloudflarestatus.com/ is showing
| outage, which cause google gcp outage, and claude outage.
| cyberflame wrote:
| Cloudflare uses GCP as a provider - it's something more
| upstream
| theflyinghorse wrote:
| identitytoolkit.googleapis.com is 503-ing on us, my whole
| customer success team is locked out from our platform
| madjam002 wrote:
| Google Maps not loading, thought it was my 4g, go to see if my
| connection works by loading Hacker News, GCP Outage XD
| evtothedev wrote:
| Yarn package registry also appears to be down.
| tom1337 wrote:
| npm is, registry.yarnpkg.com is only a CNAME to npm
| kfarr wrote:
| Yes Firebase auth is down and affecting many apps, on Discord and
| Slack groups tons of others are corroborating. A bit
| disappointing that there is no post on the status page for nearly
| 30 mins: https://status.firebase.google.com/
| kentlyons wrote:
| It just updated. Maybe affected by their own outage!
| ashu1461 wrote:
| Just proves how shady the status page and sla stuff is
| dgellow wrote:
| or how difficult it actually is to do that type of thing at
| scale
| rco8786 wrote:
| Google is 10 minutes late updating their status page.
|
| "So shady"
|
| It's really, really hard to make a status page realtime.
| ashu1461 wrote:
| What makes you think it's hard? We have AI generating
| songs and writing code, but setting up basic health
| checks is too much?
| jug wrote:
| An AI generated status page would be the epitome of 2025.
| urbandw311er wrote:
| What makes you think it's easy?
| rco8786 wrote:
| Yes. "Basic health checks" is not a real thing. I mean
| that genuinely.
|
| > What makes you think it's hard?
|
| Being responsible (or rather, on a team of people
| responsible) for a status page of a big tech co made me
| think it's hard.
|
| "Is it down?" Is _not_ a binary question.
| _kush wrote:
| Supabase is also down
| vpuna wrote:
| Yes my project on Supabase is down as well.
| andrewmcwatters wrote:
| Ah darn it. My Spotify DJ just stopped working.
| capital_guy wrote:
| BigQuery is completely dead
| b0a04gl wrote:
| console not loading, storage slow, support forms dead, status
| page green. no fallback, no real-time alert, was just wondering
| when it'll start working. whole stack feels brittle when basic
| visibility tools fail too. everyone's pointing fingers but nobody
| has root access to truth.
| ddtaylor wrote:
| Smells like BGP since there are services people claim have
| nothing to do with GCP being affected. OpenRouter is down,
| Lovable is down, etc.
| DrBenCarson wrote:
| npm as well
| koito17 wrote:
| Initially attributed the unresponsiveness of `npm install` to
| npm (the CLI tool) in general. Tried using bun to install
| dependencies, saw the same result -- but with actual logs
| instead of a vague spinner -- and decided to check Hacker
| News.
|
| Getting 504 errors on anything from registry.npmjs.org that
| isn't cached on my machine.
| yard2010 wrote:
| I just want to say that bun is a gift. It's just like npm,
| but backwards. So you imagine how perfect it is. I'm
| kidding, but really - bun is awesome. If you're using npm
| you can make the switch as it's mostly compatible.
| brown9-2 wrote:
| perhaps Lovable uses GCP somewhere in their stack?
| thallium205 wrote:
| AWS seems fine though. My bet is Cloudflare.
| TN1ck wrote:
| Cloudbuild completely down for us. Getting "Visibility check was
| unavailable" errors.
| quectophoton wrote:
| Twitch was broken too:
| https://status.twitch.com/incidents/b79nyp1yhxql
|
| EDIT: Updated link to point to the specific incident.
| rplnt wrote:
| Is Amazon running Twitch on Google Cloud (at least partially)?
| quectophoton wrote:
| I don't know, at this point I don't know who uses what. This
| is maybe unrelated but even BunnyCDN has an incident from a
| few hours ago
| (https://status.bunny.net/incidents/6g27lbtp67m4).
|
| Seeing how everything seems to be broken everywhere, I'm very
| much looking forward to the post-mortem.
| rcfox wrote:
| I'm able to login to the GCP dashboard, but it isn't able to find
| any of my projects.
| keizo wrote:
| Yup, intermittent db connection issues and cloud storage
| problems.
| admissionsguy wrote:
| Wish there existed a decentralized network connecting computers
| around the world
| redman25 wrote:
| Crazy, they could call it the "internet" or something like
| that... kind of rolls off the tongue.
| tsouth wrote:
| Everyone is down. Cloudflare has problems too. All auth providers
| broken.
| traeregan wrote:
| For us Cloud SQL instances are toast but App Engine Standard
| instances are still serving requests. Google Cloud console is
| borked too, mostly just erroring out.
| ekojs wrote:
| Super duper frustrating having the status page being green. Why
| can't Google do this properly?
| supportengineer wrote:
| Those responsible have been sacked.
| 18172828286177 wrote:
| Those responsible for sacking the people who have just been
| sacked, have been sacked.
| imzadi wrote:
| Can't reach my nest thermometer, but their status page says it's
| fine lol
| charliemeyer wrote:
| the real concerns in life
| andrelaszlo wrote:
| This is pretty crazy :D How did it affect you?
| imzadi wrote:
| I almost died
| andrelaszlo wrote:
| Shock, overheating, hypothermia, or a combination of all
| three?
| tmiku wrote:
| Looks like I'm about to start learning which of my time-killing
| websites are hosted on GCP - The Ringer is down, and since
| Spotify owns them and is a major GCP customer, it looks like
| they've been hit by this. CRAZY that the GCP status page is still
| green.
| ashu1461 wrote:
| When you deploy code generated by Gemini :D
| whitedurna wrote:
| i think it'll be disaster.
| whalesalad wrote:
| Meet is also down for me right now. Cannot attend any video
| calls.
| braunshedd wrote:
| Our GCP workloads are unavailable across several US regions. The
| GCP console is intermittently unavailable for most pages.
|
| Crossing my fingers for a quick resolution.
| morgandoane wrote:
| Storage, CloudRun, Firebase...... All down....
| dana321 wrote:
| Auth, GCP, Windsurf,Augment Code,Udio, the list is endless.
|
| Facebook, Reddit and Hacker News is still up, but thats about
| it
| vpuna wrote:
| is supabase on GCP ? My Supabase projects are down.
| duckarmada wrote:
| Supabase is on AWS, but this is looking like an upstream
| Cloudflare issue. https://status.supabase.com/
| devmor wrote:
| Not just GCP. AWS and Cloudflare too.
|
| Did someone screw up BGP again?
| akash8400 wrote:
| GKE workloads are also affected.
| meltyness wrote:
| Can't upload discord attachments from mobile.
| Axsuul wrote:
| Does anyone know if instance-to-instance networking has been
| affected? My Redis instance has been throwing a lot of connection
| errors.
| markbnj wrote:
| We're not seeing any connectivity issues between pods and vms
| in our vpc, but your mileage may vary.
| Axsuul wrote:
| Thanks
| 18172828286177 wrote:
| YouTube was down for me for some time
| dbacar wrote:
| kaggle not responding correctly, is it related?
| faizanrupani wrote:
| https://www.cloudflarestatus.com/ is showing outage, which cause
| google gcp outage, claude outage, firbase outage
| https://status.firebase.google.com/
| andrelaszlo wrote:
| How would Cloudflare's outage cause a GCP outage?
|
| I'm sure it's not entirely impossible, but sounds backwards to
| me. Sure - a lot of the internet relies on Cloudflare, but I'd
| be very surprised if GCP had a direct dependency on Cloudflare,
| for a lot of reasons. Maybe I misunderstood your comment?
| paulddraper wrote:
| "No major incidents" as of 11:37 PDT.
|
| https://status.cloud.google.com/
|
| File that in the status pages worth ~0 category.
| jschroeder wrote:
| Status page is showing green because GCP admins can't login to
| change it ;)
| tiagod wrote:
| Getting Gateway timeouts on docker hub. Maybe related? I can pull
| images.
|
| Example: https://hub.docker.com/layers/library/eclipse-
| mosquitto/late...
| ZiyadFarhan wrote:
| this aint looking good yall
| sleepybrett wrote:
| npm registry happen to be hosted on gcp, because that seems to be
| down as well.
| supportengineer wrote:
| Interesting how I landed here. I was having trouble with Nest.
| Then I went to Down Detector. I noticed many sites having a
| simultaneous uptick. Then I came to HN, and found this link at
| the top of the front page.
| geekamongus wrote:
| I usually just go here first.
| ryanscio wrote:
| Same here with npm
|
| https://status.npmjs.org/incidents/dn5mcp85737y
| waythenewsgoes wrote:
| Status pages at cloud providers aren't usually based in reality
| -- usually requires VP level political games to actually get them
| changed especially for serious outages.
| rvnx wrote:
| It looks like that it is a central service @ Google called
| Chemist that is down.
|
| "Chemist checks the project status, activation status, abuse
| status, billing status, service status, location restrictions,
| VPC Service Controls, SuperQuota, and other policies."
|
| -> This would totally explain the error messages "visibility
| check (of the API) failed" and "cannot load policy" and the wide
| amount of services affected.
|
| cf. https://cloud.google.com/service-
| infrastructure/docs/service...
|
| EDIT: Google says "(Google Cloud) is down due to Identity and
| Access Management Service Issue"
| VWWHFSfQ wrote:
| There are multiple internet services down, not just GCP. It's
| just possible that this "Chemist" service is especially
| externally affected which is why the failures are propagating
| to the their internal GCP network services.
| rvnx wrote:
| Absolutely possible. Though there is something curious:
|
| https://www.cloudflarestatus.com/
|
| At Cloudflare it started with: "Investigating - Cloudflare
| engineering is investigating an issue causing Access
| authentication to fail.".
|
| So this would somehow validate the theory of auth/quotas
| started failing right after Google, but what happened after
| ?! Pure snowballing ? That sounds a bit crazy.
| whatevertrevor wrote:
| Doesn't cloudflare have its own infrastructure, it's wild
| to me that both these things are down presumably together
| with this size of a blast radius.
| cyberpunk wrote:
| You'd think so wouldn't you?
|
| DownDetector also reports azure and oracle cloud, I can't
| see then also being dependant on GCP...
|
| I guess down detector isn't a full source of truth
| though.
|
| https://ocistatus.oraclecloud.com/#/
| https://azure.status.microsoft/en-gb/status
|
| Both green
| basfo wrote:
| Using Azure here, no issues reported so far.
| iFred wrote:
| Down Detector can have a poor signal to noise ratio given
| from what I am assuming is users submitting "this is
| broken" for any particular app. Probably compounded by
| many hearing of a GCP issue, checking their own cloud
| service, and reporting the problem at the same time.
| mandevil wrote:
| Down detector has a problem when whole clouds go down:
| unexpected dependencies. You see an app on a non-
| problematic cloud is having trouble, and report it to
| Down Detector but that cloud is actually fine- their
| actual stuff is running fine. What is really happening is
| that the app you are using has a dependency on a
| different SaaS provider who runs on the problematic
| cloud, and that is killing them.
|
| It's often things like "we got backpressure like we're
| supposed to, so we gave the end user an error because the
| processing queue had built up above threshold, but it was
| because waiting for the timeout from SaaS X slowed down
| the processing so much that the queue built up." (Have
| the scars from this more than once.)
| spwa4 wrote:
| Surely if you build a status detector you realize that
| colo or dedicated are your only options, no? Obviously
| you cannot host such a service in the cloud.
| mandevil wrote:
| I'm not even talking about Down Detector's own infra
| being down, I'm talking about actual legitimate
| complaints from real users (which is the data that Down
| Detector collates and displays) because the app they are
| trying to use on an unaffected cloud is legitimately
| sending them an error- it's just because of SaaS
| dependencies and the nature of distributed systems one
| cloud going down can have a blast radius such that even
| apps on unaffected clouds will have elevated error rates,
| and that can end up confusing displays on Down Detector
| when large enough things go down.
|
| My apps run on AWS, but we use third parties for logging,
| for auth support, billing, things like that. Some of
| those could well be on GCP though we didn't see any
| elevated error rates. Our system is resilient against
| those being down- after a couple of failed tries to
| connect it will dump what it was trying to send into a
| dump file for later re-sending. Most engineers will do
| that. But I've learned after many bad experiences that
| after a certain threshold of failures to connect to one
| of these outside system, my system should just skip
| calling out except for once every retryCycleTime, because
| all it will do is add two connectionTimeout's to every
| processing loop, building up messages in the processing
| queue, which eventually create backpressure up to the
| user. If you don't have that level of circuit breaker
| built, you can cause your own systems to give out higher
| error rates even if you are on an unaffected cloud.
|
| So today a whole lot of systems that are not on GCP
| discovered the importance of the circuit breaker design
| pattern.
| smoe wrote:
| Latest Cloudflare status update basically confirms that
| there is a dependency to GCP in their systems:
|
| "Cloudflare's critical Workers KV service went offline
| due to an outage of a 3rd party service that is a key
| dependency. As a result, certain Cloudflare products that
| rely on KV service to store and disseminate information
| are unavailable"
| whatevertrevor wrote:
| Yeah I saw that now too. Interesting, I'm definitely a
| little surprised that they have this big of an external
| dependency surface.
| smoe wrote:
| Definitely very surprised to see, that so much of the CF
| products that are there to compete with the big cloud
| providers have such a dependance on GCP.
| derefr wrote:
| Cloudflare isn't a cloud in the traditional sense; it's a
| CDN with extra smarts in the CDN nodes. CF's comparative
| advantage is in doing clever things with just-big-enough
| shared-nothing clusters deployed at every edge POP
| imaginable; not in building f-off huge clusters out in
| the middle of nowhere that can host half the Internet,
| including all their own services.
|
| As such, I wouldn't be _overly_ surprised if all of CF 's
| _non_ -edge compute (including, for example, their
| control plane) is just tossed onto a "competitor" cloud
| like GCP. To CF, that infra is neither a revenue center,
| nor a huge cost center worth OpEx-optimizing through
| vertical integration.
| whatevertrevor wrote:
| But then you do expose yourself to huge issues like this
| if your control plane is dependent on a single cloud
| provider, especially for a company that wants to be THE
| reverse proxy and CDN for the internet no?
| snowwrestler wrote:
| Cloudflare does not actually want to reverse proxy and
| CDN the whole internet. Their business model is B2B; they
| make most of their revenue from a set of companies who
| buy at high price points and represent a tiny percentage
| of the total sites behind CF.
|
| Scale is just a way to keep costs low. In addition to
| economies of scale, routing tons of traffic puts them in
| position to negotiate no-cost peering agreements with
| other bandwidth providers. Freemium scale is good
| marketing too.
|
| So there is no strategic reason to avoid dependencies on
| Google or other clouds. If they can save costs that way,
| they will.
| whatevertrevor wrote:
| Well I mean most of the internet in terms of traffic, not
| in terms of the corpus of sites. I agree the long-tail of
| websites is probably not profitable for them.
| mbreese wrote:
| True, but how often do outages like this happen? And when
| outages do happen, does Cloudflare have any more exposure
| than Google? I mean, if Google can't handle it, why
| should Cloudflare be expected to? It also looks like the
| Cloudflare services have been somewhat restored, so
| whatever dependency there is looks like it's able to be
| somewhat decoupled.
|
| So long as the outages are rare, I don't think there is
| much downside for Cloudflare to be tied to Google cloud.
| And if they can avoid the cost of a full cloud buildout
| (with multiple data centers and zones, etc...), even
| better.
| terom wrote:
| From the Cloudflare incident:
|
| > Cloudflare's critical Workers KV service went offline due
| to an outage of a 3rd party service that is a key
| dependency. As a result, certain Cloudflare products that
| rely on KV service to store and disseminate information are
| unavailable [...]
|
| Surprising, but not entirely unplausible for a GCP outage
| to spread to CF.
| voytec wrote:
| > outage of a 3rd party service that is a key dependency.
|
| Good to know that Cloudflare has services seemingly based
| on GCP with no redundancy.
| bravetraveler wrote:
| Content Delivery Thread
| londons_explore wrote:
| Probably unintentional. "We just read this config from
| this URL at startup" can easily snowball into "if that
| URL is unavailable, this service will go down globally,
| and all running instances will fail to restart when the
| devops team try to do a pre-emptive rollback"
| mrGomesDev wrote:
| I use Expo intermediation for notifications, but with this
| Google context, I imagine that FCM is also suffering, is that
| possible?
| rvnx wrote:
| Very likely. Firebase Auth is down for sure (though
| unreported yet), so most likely FCM too
| chief_jeef wrote:
| Firebase status page has acknowledged it as a "global issue".
| https://status.firebase.google.com/
|
| A contact in google mentioned to me that some bad update to
| Google Cloud Storage service has caused some cascading issues
| affecting multiple GCP services.
| 0xffany wrote:
| _Everything_ appears to be down as of 18:43 UTC...
| https://downdetector.com/
| patapong wrote:
| Perhaps their detection logic is running on Google cloud /s
| throitallaway wrote:
| I believe Downdetector displays user reports.
| brentm wrote:
| Yea I am pretty sure that if you're checking if a service
| is down your essentially casting a vote that indicates that
| service is down.
| lysace wrote:
| Kind of a missed opportunity for Ookla - who's running
| both downdetector.com and speedtest.net.
|
| The have software running in most ISPs around the world:
|
| https://help.speedtest.net/hc/en-
| us/articles/360039164793-Ho...
|
| (OTOH, it's not always trivial to define/detect an
| outage.)
| sillypuddy wrote:
| Well that's interesting. I wouldn't expect AWS or Microsoft 365
| to be affected by a Google outage.
| paxys wrote:
| Who said it's a Google outage?
| AdamJacobMuller wrote:
| Google. https://status.cloud.google.com/regional/americas
| paxys wrote:
| It's more likely to be a broader issue that is affecting
| AWS, Microsoft, Cloudflare, GCP. They aren't all
| dependent on Google infra.
| ikiris wrote:
| Oh look, they were.
|
| Cloud flare was really the gcp problem. Most of the
| others are going to be dependencies on cf or random
| Google stuff.
|
| Discord for example was gcs for updates, etc
| AlienRobot wrote:
| Wait, it's _all_ Google?
| plateng000 wrote:
| "always has been"
| deepsun wrote:
| Google was the first to report probably.
| bananapub wrote:
| all cloud
| voytec wrote:
| Yeah. This service was presenting charts likely probed from
| inside GCP. I was on a call with a Google rep, someone pointed
| out that "AWS is also down" and I foolishly said something
| about "possible BGP attack" out of spite, before checking AWS
| availability myself. Shame on me.
| toast0 wrote:
| Didn't have the feeling of a BGP issue, most services I was
| working with were reasonably quickly returning failures, as
| opposed to lingering death.
| yard2010 wrote:
| I love this kind of fake news. It's like that scene from
| Scary Movie (can't remember which one) in which someone says
| "I heard the japs took out one in Kikoman" :')
| peanut-walrus wrote:
| Downdetector in incidents like this is 100% misinformation.
| johanyc wrote:
| Why
| peanut-walrus wrote:
| Downdetector does not actually monitor the services. It
| aggregates user reports from socials etc. For large-scale
| incidents, the reports get really noisy and it will show
| that basically everything is down.
| baobun wrote:
| Who watches the watchmen?
|
| (downdetector infra also likely affected)
| deadbabe wrote:
| If LLMs are down work grinds to a halt until they return. Just
| the new era now.
| kachapopopow wrote:
| I just realized that the reason the status isn't updated is cause
| they can't access it lol.
| paulddraper wrote:
| Don't host status pages (or their dependencies) on your own
| infra lol.
|
| Seems obvious.
| deathanatos wrote:
| It should be obvious because both AWS and Azure have done
| this in the past and shown what a bad idea it is...
| Axsuul wrote:
| How do you know that?
| fidotron wrote:
| It's completely nuts that Firebase has this:
| https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e...
|
| "Firebase Data Connect unavailable due to a known Google Cloud
| global outage"
|
| While the Google Cloud status page
| https://status.cloud.google.com/ says "No major incidents" and
| everything is green. So Google Cloud know there is an outage but
| just deem it not major enough to show it.
|
| Edit to add: within 10 minutes of this post Google updated their
| status page. More curiously the Firebase page I linked to has
| been edited to remove mention of Google Cloud in the status and
| now says "Firebase Data Connect is currently experiencing a
| service disruption. Please check back for status. ".
| kjuulh wrote:
| Something must be preventing them updating the status page at
| this point. Of course they could still deem it not enough, but
| just from my limited tests, docker, buf, etc (it may not be GCP
| that is down, but it is quite the coincidence). are outright
| down. I'd wager that this is much more widespread.
| sss111 wrote:
| I'm actually on a bridge call with Google Cloud, we're a
| large customer -- I just learned today that their status page
| is not automated, instead someone actually manually updates
| it!
| paxys wrote:
| That's the case with every status page. These pages are
| managed by business people not engineers, because their
| primary purpose is to show customers that the company is
| meeting contractually defied SLAs.
| belter wrote:
| Surelly no SLA will be based on the display of the status
| page...
| Tostino wrote:
| should* be
| phatskat wrote:
| Maybe or maybe not, but someone with nothing better to do
| than monitor that page out of boredom might "get on the
| horn" with lots of people to complain if a green check
| mark turns to a red X.
| paxys wrote:
| They aren't automatically based on that page, but seeing
| a red status makes it too easy for customers to point to
| it and go "see you were down, give us a refund".
| redeux wrote:
| This is actually the norm for status pages. If you look at
| the various status page offerings you'll see that they're
| designed around manual updates.
| quectophoton wrote:
| The best way to consistently having good "time to
| response" metrics, is to be the one deciding when an
| incident "actually" started happening, if at all :)
| kjuulh wrote:
| This feels very much like when facebook, locked themselves
| out of their datacenters. ;)
|
| * https://www.datacenterdynamics.com/en/news/facebook-
| blames-m...
| dinvlad wrote:
| Except that AWS, CloudFlare and a bunch others are also
| down :-O
| cyberpunk wrote:
| AWS looks ok to me?
|
| https://health.aws.amazon.com/health/status
|
| Perhaps CF is dependant on some GCP services?
| diggan wrote:
| > AWS looks ok to me?
|
| > https://health.aws.amazon.com/health/status
|
| Historically, the worst place to figure out if AWS is
| up/down is Amazons own status page.
| peterjliu wrote:
| seems like misinformation for AWS. CloudFlare probably
| depends on GCP.
| kjuulh wrote:
| Downdetector shows they've got issues as well, but it can
| be fairly unreliable, as people don't know which service
| is behind their apps.
|
| I at least have no issues on their services across a few
| regions, and their console works fine.
| spenczar5 wrote:
| That's fairly typical. You want a human in the loop for
| decisions like that.
| paulddraper wrote:
| Most status pages are manual.
|
| At least some of the information has to be.
|
| The weird part is that it took them almost an full hour to
| update it.
| mpalmer wrote:
| The bigger you are, the more you want a human involved in
| the decision to publicly declare an incident.
| codergautam wrote:
| Maybe the outage is preventing them from updating that specific
| page? Hmm
|
| EDIT: Looks like it has been updated now (6:49 PM UTC)
| samdung wrote:
| :))))))
| devMem wrote:
| I hope this is the case, or google is super unreliable for
| production grade work.
| alexcroox wrote:
| Almost an hour to update the page...
| artooro wrote:
| Anytime there is an outage that affects App Engine, Google
| can't seem to get their status page updated for an extended
| period of time.
| samdung wrote:
| GCP just updated their status
| aetherson wrote:
| More likely they are unable to update their own status page,
| but in either case not covering themselves in glory over at GCP
| right now.
| alexcroox wrote:
| CF too: https://www.cloudflarestatus.com/
| cherioo wrote:
| This extra funny that GCP status page even includes a "last
| updated" time, which is exactly built to convey possible
| failure to update in cases like this
|
| No major incident as of " Last updated time: 12 Jun 2025, 11:48
| PDT"
| ike2792 wrote:
| Maybe their dashboard is hosted on GCP and they are displaying
| a cached version. :-)
| blibble wrote:
| lies, from big tech?
|
| say it's not so!
| Workaccount2 wrote:
| I asked testing to see if it was up, and it pointed out that
| Google shows nothing but Nest is showing an outage right now,
| lol
|
| https://status.nest.com/posts/dashboard
| octo888 wrote:
| Status pages are PR. It gets the same PR treatment as anything
| else
| shmatt wrote:
| IIRC status pages drive customer compensation for downtime.
| Updating it is basically signing the check for their biggest
| customers, in most similar companies you need a very senior
| executive to approve the update
|
| On the other side of this, Firebase probably doesn't have money
| at stake making the update
| aiauthoritydev wrote:
| It is not the status page that drives customer compensation.
| It is downtime.
| camdenreslink wrote:
| The status page is essentially an admission of guilt. It
| can require approval from the legal department and a high
| level official from the company to approve updating it and
| the verbiage used on the status page.
| dijit wrote:
| then it's fucking useless. Let's crowd source our own
| Tijdreiziger wrote:
| That's what Downdetector is.
|
| https://downdetector.com/
| hugs wrote:
| working on it! (valet network)
| refulgentis wrote:
| "It can", this is just free-associating, don't let it get
| to ya. (disclaimer: xoogler)
| shuntress wrote:
| We tried to do that. It didn't work. Too much spam,
| scams, and abuse.
| baggy_trough wrote:
| You're in the crowdsourced version right now.
| dpkirchner wrote:
| You are likely right, but it's still gross dishonesty.
| I'm not ready to let Google and their engineers off the
| hook for that.
| refulgentis wrote:
| Inter alia, "is essentially", "it can", tell us this is
| just free-associating.
|
| We should probably avoid punishing them based on free-
| associating made by a random not-anonymous not-Googler
| not-Xoogler account on HN. (disclaimer: xoogler)
| hodgesrm wrote:
| > It can require approval from the legal department and a
| high level official from the company to approve updating
| it and the verbiage used on the status page.
|
| Is that true in this case or are you speculating? My
| company runs a cloud platform. Our strategy is to have
| outages happen as rarely as possible and to proactively
| offer rebates based on customer-measured downtime. I
| don't know why people would trust vendors that do
| otherwise.
| refulgentis wrote:
| Nah, its just some client side caching / JS stuff. Clicking
| the big refresh button fixed it for me, 15 minutes before OP
| noted it.
|
| (n.b. as much as Google in aggregate is evil, they're smart
| evil. You can't avoid execs approving every outage because
| checks without _some_ paper trail, and execs don 't want to
| approve every outage, you'd have to rely on too many
| engineers and sales people, even as ex-employees, to keep it
| a secret. disclaimer: xoogler)
|
| (EDIT: for posterity, we're discussing a "overall status"
| thing with a huge refresh button, right above a huge table
| chockful of orange triangles that indicate "One or more
| regions affected" - even when the "overall status" was green,
| the table was still full of orange and visible immediately
| underneath. My point being, you gotta suppose a wholeeee
| bunch of stuff to get to the point there was ever info
| suppressed, much less suppressed intentionally to avoid
| cutting checks)
| cyberflame wrote:
| Services are recovering in some locations it seems - Discord is
| healing
| nailer wrote:
| AWS has this all the time. If you need to know if a service is
| down in a region, check for other engineers talking about it on
| X.
| paxys wrote:
| Kinda funny that the top post on HN titled "GCP Outage" links to
| the Google Cloud status page which shows...no outage.
| kgwxd wrote:
| Sorry, after decades of being hard wired, I just installed a PCIe
| Wifi6 card on my desktop. Internet took a dive the second I got
| it connected. Must have done something wrong.
| happy-camper wrote:
| Text messaging on Android is broken
| happy-camper wrote:
| Text messaging for android is broken as well
| ipsum2 wrote:
| Cloudflare is down too. From https://www.cloudflarestatus.com:
|
| Update - We are seeing a number of services suffer intermittent
| failures. We are continuing to investigate this and we will
| update this list as we assess the impact on a per-service level.
|
| Impacted services: Access WARP Durable Objects (SQLite backed
| Durable Objects only) Workers KV Realtime Workers AI Stream Parts
| of the Cloudflare dashboard Jun 12, 2025 - 18:48 UTC
|
| Edit: https://news.ycombinator.com/item?id=44261064
| paulddraper wrote:
| Broken link? EDIT: Weird, definitely was just empty
| ipsum2 wrote:
| Should work, but its also on the front page.
| 0xy wrote:
| Seems like a major wtf if Cloudflare is using GCP as a key
| dependency.
| a2128 wrote:
| Some day Cloudflare will depend on GCP and GCP will depend on
| Cloudflare and AWS will rely on one of the two being online
| and Cloudflare will also depend on AWS and the internet will
| go down and no one will know how to restart it
| 0xffany wrote:
| Does anyone know of a good dashboard to check for such BGP
| routing anomalies as (apparently) this one? I am currently
| digging around https://radar.cloudflare.com/routing but it
| doesn't show which routes were actually leaked.
|
| I would love if anyone has any good tool recommendations!
| SparkyMcUnicorn wrote:
| I don't know if I've seen CF Radar before. That's pretty cool!
|
| Here are some others, although some seem to be experiencing
| issues due to the current outage I can only presume.
|
| - https://atlas.ripe.net/probes/public
|
| - https://www.ihr.live/en/global-report
|
| - https://www.ihr.live/en/network
|
| - https://bgp.he.net/
|
| - https://ioda.inetintel.cc.gatech.edu/dashboard/asn
| ddtaylor wrote:
| I am a newb at this too, but is it "normal" for the "Announced
| IP Address Space" section to have that large jump from
| addresses like that?
| estees_ecstacy wrote:
| go https://status.gcp.databricks.com/
| cluckindan wrote:
| BGP attack?
| mrngm wrote:
| My default go-to: https://bgp.tools/
|
| Why would you think this outage is (internet) BGP related?
| whalesalad wrote:
| Cloudflare runs all their own bare metal servers. Seems odd
| that they would be impacted by Google cloud. Same can be said
| for all the other issues on downdetector. This points to a
| broad issue at the core internet which could certainly be
| related to BGP.
| ilkkao wrote:
| Cloudflare is now saying:
|
| "Cloudflare's critical Workers KV service went offline due
| to an outage of a 3rd party service that is a key
| dependency."
|
| I really hope CF explains this apparent Google dependecy in
| detail in their post mortem.
| whalesalad wrote:
| Imagine it's just a google spanner wrapper lmao
| stepri wrote:
| They have updated the status page finally
| https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
| acureau wrote:
| Well this explains the issues I've been having with Spotify
| through the last hour.
| artooro wrote:
| What's crazy is that RCS messaging is down as a result of this
| outage. It shows how poorly the technology or infrastructure was
| designed.
| foota wrote:
| Isn't RCS basically just instant messaging? I don't know why
| it's surprising that it would be down.
| roywiggins wrote:
| I'm not sure any single company could have an outage that
| would take out SMS globally, but RCS is presumably more
| centralized.
| watusername wrote:
| It used to be kind of distributed, but Google has been
| strong arming carriers to use their hosted Jibe service
| through a combination of proprietary extensions (e.g., E2E
| which is finally standard) and bypassing carrier control
| (if the carrier didn't provision RCS, Google Messages would
| use their own service iMessage-style).
|
| From the end user's perspective, if the carrier didn't use
| Jibe RCS, it simply wouldn't work well.
| whynotminot wrote:
| People liked to be utterly pissed at Apple for not
| supporting RCS. But there were reasons
| toast0 wrote:
| SMS is pretty much decentralized, although there's a few
| companies with a lot of reach. I don't remember any Global
| SMS outages, but it wasn't uncommon for a whole carrier to
| have an SMS outage and especially for inter-carrier SMS to
| be broken from time to time (sometimes for days). I've
| certainly seen some stuff with SMS aggregators: almost all
| of them claim a majority of direct links, but when you have
| accounts with 4 large aggregators and one of them has an
| outage, you find out which of your other account use that
| aggregator for which links (because their deliverability
| will go to zero to those destinations).
|
| RCS was designed and specced, by GSMA, as a telco run
| decentralized system that would replace SMS as like for
| like; but there were only a handful of rollouts. It's
| really only gotten use as Google pushed it onto Android,
| using their RCS server; recently iOS started using it
| although I don't know what server they attach to.
|
| Since RCS is basically the 5th wave Google IM, it's no
| surprise when they have a major outage, RCS is pretty much
| broken.
| lieuwex wrote:
| > recently iOS started using it although I don't know
| what server they attach to.
|
| According to Wikipedia, only the carrier's RCS server is
| used [1]
|
| [1]: https://en.wikipedia.org/wiki/Rich_Communication_Ser
| vices#So...
| treesknees wrote:
| Thanks for mentioning this, I'm seeing this too and couldn't
| figure out why.
| whalesalad wrote:
| should have used Erlang
| wbl wrote:
| That explains why I couldn't get the photo of my parents dog
| today.
| compscidr wrote:
| well this explains so much lol
| ilovebabyyoda wrote:
| It looks like more than GCP: outages reported across the board
| including aws
|
| https://downdetector.com/
| koliber wrote:
| About the only thing not down is down detector.
| tonyhart7 wrote:
| god send omg, imagine down detector is down lmao
|
| anyone know what tech stack they use and where they host
| ekojs wrote:
| https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
|
| > Multiple GCP products are experiencing impact due to Identity
| and Access Management Service Issue
|
| IAM issue huh. The post-mortem should be interesting at least.
| yard2010 wrote:
| Ha. With all this soviet style euphemism I rather read the
| onion instead.
| bananapub wrote:
| It's not a euphemism - every outage, including the 99.9% that
| don't end up on HN gets a postmortem document written about
| it, which is almost always a fascinating discussion of the
| technical, cultural and organisational situation that led to
| an unexpected bad thing happening.
|
| Even a few years ago senior management knew to stay the fuck
| out except for asking for more info.
| 0xCAP wrote:
| > No major incidents
|
| ... Proceeds to show worldwide degraded service level alerts.
| jimt1234 wrote:
| Yep. Self-reporting status pages are pretty near worthless. At
| my former large company (not FAANG), we weren't allowed to
| update the status page until we got VP approval, which also
| required approval from both PR and Legal. It would take _a lot_
| more time and effort to get those approvals than to just fix
| the problem and move on.
| iFred wrote:
| SLA contracts, clawbacks, and performance obligations make
| these pages a bit of a minefield for CSPs. When I was at a
| top-tier CSP, we had the status page that was public, one
| that was for a trusted tier of customers, one built for a
| customer-by-customer basis, and one for internal engineering.
| genewitch wrote:
| When i worked at a top tier speakeasy, we had a book up
| front for the man, a book in the back for the boss, a book
| for the trusted accountants...
| xan_ps007 wrote:
| Where are the AI agents?
| dionys wrote:
| Poor agents, finally taking a break
| kyleee wrote:
| The AI is over employed
| Jayakumark wrote:
| Someone must have checked in AI Generated code :-)
| cyrux004 wrote:
| GPay which is a widely used payment service in India is down as
| well
| edm0nd wrote:
| India is having a really bad day today
| unsupp0rted wrote:
| The last few times this happened I wouldn't have thought "So this
| is the day AI takes over".
|
| But this time...
| agawish wrote:
| GCP status page now reflect the issues, looks like Google Cloud
| Dataproc, Google Cloud Storage and Identity & Access Management
| SenpaiHurricane wrote:
| Now my api can not connect to PostreSQL...
|
| sslv3 alert bad
| certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3
| dineshsingh1 wrote:
| https://downdetector.in/
| DonHopkins wrote:
| Damn you Bart Simpson!
|
| https://en.wikipedia.org/wiki/Bart_Gets_Famous
| dinvlad wrote:
| #HugOps
| devmor wrote:
| Incident report published:
| https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
| james_m1231 wrote:
| My firebase hosting and firestore db are back online, but GCP
| console and Google SQL instances are still having serious issues
| as of 7:00pm UTC.
| phoenix98 wrote:
| when its going to be fixed i am seeing now more and more services
| are down?
| chupamela wrote:
| internal systems at google are currently broken.
| phoenix__1998 wrote:
| when its going to be fixed i am seeing now more and more services
| are down?
| dinvlad wrote:
| Surprised no one else mentioned "it's always DNS" yet :-)
| phoenix__1998 wrote:
| when its going to be fixed , i am seeing now more and more
| services getting outage started with IAM ?
| kjuulh wrote:
| I wonder what the damage ($) for having a good portion of the
| internet down for an hour or two ;)
| throwcarsales wrote:
| My friends and I are even having trouble getting Rcs text
| messages to send.
| coryvirok wrote:
| Shameless plug for https://rollbar.com
|
| Good luck out there!
| CyrMeta wrote:
| if everything down at the same time - No one is mentioning an
| attack on us cloud services ? ( China or Russia ) Maybe ?
| cyberflame wrote:
| They've now added this as a major incident - before it just was
| listed under overview
| CyrMeta wrote:
| if all services at down at once, no one is thinking or mentioning
| a potential attack on US cloud providers ? (China or Russia)
| Maybe ?
| dpedu wrote:
| reCAPTCHA affected? I couldn't log into my local utilities
| website due to a reCAPTCHA error. Downdetector agrees, but I
| interpret that site as dubious.
| marifjeren wrote:
| Yeah recaptcha is down intermittently
| ashwinsundar wrote:
| Interesting that all Digital Ocean services are fine...
| baq wrote:
| One of these days in which the young engineers learn the concept
| of 'counterparty risk'.
| DonHopkins wrote:
| What is this Touchable Grass stuff I keep hearing of?
| bsamson05 wrote:
| So frustrating, but here's a link to track status of this outage:
| https://status.anthropic.com/incidents/kn7mvrgb0c8m
| CrimsonCape wrote:
| I'm having trouble getting any Street View imagery. Can anyone
| else confirm?
| NameError wrote:
| Yep, street view is not working at all for me
| estee_ecstacy wrote:
| Seems recovering now
| estees_ecstacy wrote:
| Seems recovering now
| estees_ecstacy wrote:
| seems recovering
| jamesrwhite wrote:
| Seems like a wider issue at Google than just GCP, the Sheets and
| Chat APIs are also returning similar "Visibility check was
| unavailable" errors.
| yunwal wrote:
| Presumably many Google products run on GCP
| throwaway7783 wrote:
| Even though BigQuery is not listed in affected services, we see
| errors connecting to it
| tecleandor wrote:
| It's listed by regions :(
| leoh wrote:
| If Google Chat is down per
| https://www.google.com/appsstatus/dashboard/, the ability for
| Google engineers to communicate among themselves impaired,
| despite SREs having IRC as a backup.
| iamdelirium wrote:
| Google Chat wasn't down for me throughout the entire incident.
| miohtama wrote:
| Someone actually uses Google Chat...?
| asadm wrote:
| it's the best
| ZiiS wrote:
| Well given how many they have decommissioned...
| clhodapp wrote:
| Oh no, that's how you know it's nearing the point of being
| reaped and thrown in the graveyard!
| leoh wrote:
| Almost everyone inside Google
| 00deadbeef wrote:
| Google has a chat product?
| sebzim4500 wrote:
| TIL Google chat hasn't been killed yet
| bananapub wrote:
| it at least used to be standard and fairly well known practice
| for non-sres to use the irc bridge.
|
| the much more disastrous situation would have been the irm
| fallback.
| donalhunt wrote:
| They have irc services internally (or at least did when I was
| there 10-ish years ago).
| jim180 wrote:
| Claude Code is down :( too lazy to do manual conversion from
| Cocoapods dependency to SwiftPM
| rectang wrote:
| Haha, I don't ordinarily spend a lot of time in the Google Cloud
| Console but just now I was debugging a squirrely OAuth issue with
| reCAPTCHA failing to refresh several days running. I'm getting
| this weird page error, and I think, "Is this an issue with my
| organization? [futz futz futz] Hey wait is GCP actually down?"
| And it turns out to be the top discussion on HN. XD
| dgellow wrote:
| Well, good luck to all googlers dealing with this, that's not fun
| :(
| enahs-sf wrote:
| Would be comedy if one of the progenitors of this took Sundar's
| buyout offer yesterday and let the world burn today.
| AIorNot wrote:
| sheesh so many side-affected issues accross all systems, maybe
| big tech companies like google shouldn't have laid off all those
| engineers..
| https://www.google.com/appsstatus/dashboard/incidents/Eab7zG...
|
| but no tech bros, just keep following your ketamine addled
| edgelord when he did this with twitter..
| dang wrote:
| Related ongoing thread:
|
| _Ask HN: Is Firebase Down?_ -
| https://news.ycombinator.com/item?id=44260669
| cyberflame wrote:
| Everything except us-central1 is back up - it's recovering now
| though
| pancomplex wrote:
| thank god hn is hosted on a single bare metal server, free of all
| this bloat.
| admissionsguy wrote:
| Google denies the outage.
| https://x.com/Google/status/1933246051512644069
| baggy_trough wrote:
| "clearing cache and cookies"? what is this, 1997?
| bosmanos wrote:
| lol
| andrelaszlo wrote:
| First, check that nobody else in your family is making a call
| on the phone line that your modem is connected to, then make
| sure to disable your Internet Explorer add-ons before trying
| again.
| gred wrote:
| https://x.com/ProductHunt/status/1626586036402003970
| ElijahLynn wrote:
| for those who boycott X:
|
| https://nitter.net/Google/status/1933246051512644069
| desktopninja wrote:
| Borg and K8s were fighting for resources, so Gemini decided to
| take out DNS. Now a sysAdmin has to step in.
|
| * just trying to add a little humour. pretty stressfull outage.
| grarr!!
| lawrenceyan wrote:
| Solana is up -\\_(tsu)_/-
| pier25 wrote:
| "All locations except us-central1 have fully recovered. us-
| central1 is mostly recovered. We do not have an ETA for full
| recovery in us-central1."
| kubectl_h wrote:
| An hour later and everything is a mess in central-1. They
| seemed to jump the gun on that one. Doesn't matter if some
| dinky service like "AutoML Vision" is working, if GCS isn't,
| then they shouldn't post an optimistic message.
| creddit wrote:
| This is at least why Claude is dead:
| https://status.anthropic.com/incidents/kn7mvrgb0c8m
|
| Also spotify isn't working for me so I assume that's also
| related.
|
| These are my most important productivity resources! Sad!
| estee_ecstacy wrote:
| sentry is down https://status.sentry.io/
| cyberflame wrote:
| Root cause has been identified and it's being resolved/monitored
| now
| johnnyApplePRNG wrote:
| This appears to be continuing to cascade over an hour later...
| wow... more and more services mentioned as completely down on the
| outage page.
|
| Kind of nice to not be glued to AI chat prompts for a while to be
| honest.
| sigmaball wrote:
| Guess they used Jules to code their services :)
| alexcroox wrote:
| 2 hour outage at this point
| plerpler wrote:
| GCP Artifact registry still down... Not accepting image push and
| showing 500 status code
| augbog wrote:
| Cloudflare Outage also just updated
|
| > Cloudflare's critical Workers KV service went offline due to an
| outage of a 3rd party service that is a key dependency. As a
| result, certain Cloudflare products that rely on KV service to
| store and disseminate information
| itdependsnet wrote:
| Any chance this is the root being that so many different services
| are effected? https://github.com/kubernetes/kops/issues/17433
| yunwal wrote:
| I doubt gcloud would be affected by an aws-specific cni. Unless
| maybe enough AWS users have a GCP backup environment that they
| flipped on all at once, but it seems unlikely
| itdependsnet wrote:
| good point. I took that as simply the example that they had
| in front of them but a generic issue.
| jamie0 wrote:
| https://cloud.google.com/kubernetes-engine/docs/release-note...
| google did release an update to gcp k8s today, seemingly
| shortly before the outage
| pikdum wrote:
| Was just about to do a demo, but Google Meet was down. Tried to
| use Jitsi as a fallback, but couldn't log in because Firebase was
| down too. Ended up using a Slack Huddle, lol.
| zeke wrote:
| mapbox maps seemed to be down for a few minutes about an hour
| ago. I wonder if it is related.
| ferchysmn wrote:
| Hola
| jwatte wrote:
| Let's say a typical base service (network attached RAM or
| whatever) has 99.99% reliability. If you have a dependency on 100
| of those, you're suddenly closer to 99% reliability. So you
| switch to higher-level dependencies, and only have 10
| dependencies, for a 99.9% reliability. But! It turns out, those
| dependencies each have dependencies, so they're really already
| more like 99.9% at best, and you're back at 99% reliability.
|
| "good enough" is, indeed, just good enough to make it not
| worthwhile to rip out all the upstreams and roll your own
| everything from scratch, because the cost of the occasional
| outages is much lower than the cost of reinventing every single
| wheel, nut, bolt, axle, bearing, and grease formulation.
| geocrasher wrote:
| https://soundcloud.com/ryan-flowers-916961339/the-internet-i...
| max4c wrote:
| If you need gpus rn: check out runpod.io
| quyleanh wrote:
| Look like affect to Cloudflare as well [1] Update
| - Cloudflare's critical Workers KV service went offline due to an
| outage of a 3rd party service that is a key dependency.
| Jun 12, 2025 - 19:57 UTC
|
| 1: https://www.cloudflarestatus.com/
| asim wrote:
| Just our bi-yearly reminder of our over reliance on cloud
| providers for literally everything. Can't say there's an answer
| beyond trying to build more independent tech but we know how that
| goes.
| ocdtrekkie wrote:
| Hilariously, I did not know about any outages today during the
| workday because we discourage cloud service usage and nobody
| complained about anything breaking. :)
| LZ_Khan wrote:
| Is this the new Y2k?
| riknos314 wrote:
| Actual incident link posted:
| https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
| makk wrote:
| And THAT, Smithers, is why we wear hardhats on the job.
| kodisha wrote:
| > Waiting for downdetector.com to respond...
___________________________________________________________________
(page generated 2025-06-12 23:00 UTC)