[HN Gopher] Google Kubernetes Engine incident spanning 9 days
___________________________________________________________________
Google Kubernetes Engine incident spanning 9 days
Author : talonx
Score : 155 points
Date : 2023-10-10 13:59 UTC (9 hours ago)
(HTM) web link (status.cloud.google.com)
(TXT) w3m dump (status.cloud.google.com)
| uniformlyrandom wrote:
| The incident impact (nodepool upgrade issue) seems to be matching
| the speed of mitigation rollout. One does not want the cure to be
| worse than the disease; roll forwards should be slow unless the
| impact is high (and even then, it should be a rollback/freeze
| rather than fast roll forward).
| tuananh wrote:
| sorry what mitigation is that?
| vinni2 wrote:
| I have been pulling my hair to fix this all week.
| mrweasel wrote:
| In Google terminology is a "mitigation" the same as a solution? I
| read it as "Yeah, we still have no idea how to fix this
| correctly, but we have applied a temporary work-around".
| VirusNewbie wrote:
| It more or less means actions have been taken to reduce or
| eliminate the problem, but it is not the same as the long term
| fix. Mitigation might be 'give more CPU' to the system, despite
| the performance regression still being in the code etc.
| yegle wrote:
| Disclaimer: Google SRE working in Cloud but not related to the
| GKE product.
|
| Mitigation can vary: an additional firewall rule to stop
| certain traffic, a rollback to known good version, temporarily
| redirect traffic away from impacted data center. A mitigation
| is part of an incident response mainly focused on stopping the
| pain to whoever are impacted, it may not always be a long term
| solution.
| peddling-brink wrote:
| I can't speak for google, but in common parlance, mitigation
| simply means the problem isn't affecting you any more.
| Doubtless they will perform a full post-mortem, identify the
| contributing factors, and fix them over the coming days, weeks,
| quarters.
|
| Edit: "we are assessing its effectiveness" lol, yeah that
| sounds more like they they are throwing something at the wall
| to see if it sticks.
| denysvitali wrote:
| Which is arguably better than just waiting for something to
| stick to the wall by itself
| dilippkumar wrote:
| Noob here with some meta-questions about developer and operations
| complexity.
|
| From an outsider's perspective, it looks like in a 2x2 matrix of
| developer simplicity/complexity and operational
| simplicity/complexity, the current patterns all seem to be
| heavily biased for developer simplicity/operational complexity.
|
| 1. Is this assumption correct?
|
| 2. Does optimizing for another quadrant: developer complexity /
| operational simplicity make sense?
|
| My intuition is that complexity in code can be managed far better
| than complexity in operations. Developers have abstractions,
| reusable libraries, unit tests/integration tests, etc. There may
| also be weird efficiencies that arise from having developers deal
| with some of these problems right from the design stage.
|
| It seems kubernetes takes a problem and pushes it to fully to
| operations.
|
| Is there a solution that takes this problem and turns it into a
| developer problem?
| diarrhea wrote:
| Infrastructure is, or at least should be, code as well. And as
| it is, you can write tests for it all the same!
|
| However, writing those tests is incredibly hard. It doesn't
| matter if you approach it from a dev or ops angle. The system
| under test doesn't only have side effects, it _is_ side
| effects. You also cannot mock most things (in my opinion...),
| as that is either also very hard to instrument or straight up
| removes the test usefulness altogether. Imagine mocking the AWS
| management API for your integration tests. Not possible.
|
| So what Dev calls integration or e2e tests, ops calls the dev
| environment. Works, but differently to how devs would do it. I
| don't see an alternative.
|
| Next, as much as knowledge siloes are being heralded as evil,
| they exist. Undoing siloes altogether isn't possible. You'll
| end up reerecting them elsewhere. Devs have their skill sets,
| and ops isn't part of that. The opposite is also true. The
| intersection can be substantial, but never enough to have dev
| to it all alone. I don't think that's a bad thing either.
| alectroem wrote:
| Wow, I literally did a full cluster version upgrade last night
| without knowing about this. I would have delayed the upgrade if I
| had known GKE was failing for "a small number of customers"
|
| I wish cloud providers would just communicate outages to services
| I use like this to me!
| danielvaughn wrote:
| looks like it's only affecting clusters on 1.24. If you
| upgraded it was likely to 1.27
| Racing0461 wrote:
| The status page became political.
| jamietanna wrote:
| Sorry what do you mean by this?
| ucosty wrote:
| I think they mean that companies rarely update their status
| pages to reflect reality (for instance, AWS outages are
| rarely shown on their own status pages). This is often by
| design, company policy, or a desire to save face.
| deathanatos wrote:
| And it's so incredibly dumb. Companies need to get it
| through their thick heads that this is so incredibly
| short-sighted.
|
| Not once has a status page that's devoid of information
| or slow to update ever saved face. I am far more annoyed
| to have to continue to verify "no, it is indeed that
| _your_ service is down, not mine " and then file a
| support ticket. I am triply annoyed if the response from
| support is "ah yes that's a known problem and we're
| working on it" -- known, and you just didn't bother to
| communicate.
|
| I miss the days when Github had graphs. Even if they
| simply hadn't had the time to put a message on the page,
| you could tell from the graphs that it was Github. But
| even with "more information" that some PM might not like
| being put out publicly, Github felt more reliable &
| stable in those days.
|
| At the end of the day, no amount of political
| gamesmanship will save you from having to actually run a
| reliable service, and gamesmanship just makes it more
| likely I'll ascribe false positives to your service,
| further lowering my perception of its reliability.
|
| It's so watered down that the "AWS" emoji in our Slack
| instance is literally a meme of the status page.
| networkchad wrote:
| [dead]
| the-dude wrote:
| I bet as long as the status page is not updated, it is
| not taken into account when calculating quarterly or
| yearly uptime statistics.
|
| I am sure that counts. Probably tied into someone's bonus
| as well.
| Racing0461 wrote:
| Any metric that becomes a target ceases to be a good
| metric. https://health.aws.amazon.com/health/status
|
| See all those green there? Once it started becoming
| "monitored" by VPs instead of the software engineers on
| call, they started to become political. I bet there are
| several sev2s happening for several of those services even
| as we speak but it still shows green to the outside
| observer. If one has access to the actual metrics for those
| services, i bet we would see a different story than what is
| shown on the "status" page.
| arzig wrote:
| There was something probably here a few days ago to the
| effect of 'Their 9s are not your 9s'. Like yes, their
| status is showing an error rate of less than .00001%.
| However, all of those errors are being generated by your
| 5 instances that are completely down.
| ljm wrote:
| I haven't had that comms issue with Google but I have to say,
| even though I prefer GCP to AWS in terms of user friendliness,
| it is far too often the case that you find exactly the solution
| you need only to learn it's deprecated in favour of a less
| useful alternative.
| andrelaszlo wrote:
| You can also use the (pre-GA) Service Health API to get alerts
| specifically for the regions and services you use. It's pretty
| nice!
|
| https://cloud.google.com/service-health/docs/overview#how-pe...
| heyoni wrote:
| I will say CircleCI's dashboard makes it impossible to NOT know
| there is an outage going on by putting it in the sidebar.
| Unless it's collapsed you'll be aware of everything breaking
| (that they report) to the point where it feels like everything
| is breaking all the time
| senderista wrote:
| If you've ever worked at a cloud provider, then you know
| everything _is_ breaking all the time. The good ones are just
| able to hide it most of the time /for most customers.
| secondcoming wrote:
| There is a 'Google Cloud Service Health Updates' Slack app that
| has notifications about this incident. Here's what it looks
| like on our channel: 8:18 PM APP
| UPDATE: Global: Google Kubernetes Engine Nodepool Upgrade
| Failures Incident began at 2023-10-02 11:29 (all times
| are US/Pacific).Summary: Global: Google Kubernetes Engine
| Nodepool Upgrade Failures Description: A mitigation has
| been rolling out and we are assessing its effectiveness. We
| will provide an update byTuesday 2023-10-10 12:00 US/Pacific
| with current details. Diagnosis: A small number of
| customers are experiencing failed nodepool upgrades. Customers
| experiencing this, may see "Internal error" in Google Cloud
| Console. Retrying is suggested but may...
|
| There are quite a lot of alerts about various issues.
| talonx wrote:
| They do publish an RSS feed for the status page, but there is
| no direct way to get notified AFAIK. I used to create a Slack
| notifier using IFTTT.
| captn3m0 wrote:
| For infra folks, I always suggest having a Slack channel with
| RSS feeds from vendor incident sites.
|
| Slack let's you subscribe directly to an RSS feed using
| /feed.
| talonx wrote:
| Yes, I used to do that for my teams. Most infra vendors
| have RSS feeds for their public status pages.
| sleepybrett wrote:
| Don't put this in a special channel just for vendor
| incidents. Hopefully you have a channel for each vendor
| tool where you have a vendor representative present. You
| put that bot in there. It's much more likely to be noticed
| and much less likely to be a channel that everyone ignores
| because it's SNR is to low.
| danielovichdk wrote:
| "vendor representative" ? Internal or external contact ?
|
| Sounds like a good approach nonetheless
| alectroem wrote:
| Thats a really good idea!
| nailer wrote:
| I wish I had a better answer but:
|
| When you suspect things are broken, check X (FKA Twitter).
|
| Other devs will be talking about it before there's an official
| status page.
| deathanatos wrote:
| ... if only the site formerly known as Twitter wasn't so
| hostile to being checked these days.
|
| If we as an industry can't think of something better (
| _cough_ honest status pages _cough_ ) ... can we at least
| transition these tweets to Mastodon.
| pixl97 wrote:
| Honest status page are a really really hard problem.
|
| If your connector between your status monitor and the
| service breaks you'll have some subset of users panicking
| and causing problems (or asking for refunds for outages)
| when the service was up the entire time.
|
| 3rd party services are the only ones that you'll get a
| "more honest" but not always correct view of what the
| actual status is.
| solardev wrote:
| Even if they just updated it manually for major
| incidents, it would still be useful.
| pizzafeelsright wrote:
| Define major incident. Defines update.
| piperswe wrote:
| When an incident is declared, have someone tasked with
| determining customer impact. If the impact radius is
| greater than a handful of customers, declare a public
| incident. If customer communication is made a priority,
| then you can actually have a helpful status page.
|
| Where I work, just about any non-false-alarm incident
| ends up on the status page in a timely manner. There's
| nothing stopping the likes of AWS from doing the same
| except for culture.
| solardev wrote:
| Exactly. It's the kind of thing where one person managing
| it and deciding whether to post updates is probably gonna
| work better than anything automated.
|
| Well, maybe not at AWS scales though, if they have
| thousands of everything =/
| solardev wrote:
| Why? It's just a judgment call
| nailer wrote:
| > ... if only the site formerly known as Twitter wasn't so
| hostile to being checked these days.
|
| You can easily check it manually. Search for "EBS" and see
| a bunch of people talking about EBS timeouts or whatever.
| That was what I was getting at.
|
| But yeah scraping is harder now.
| deathanatos wrote:
| I'm not talking about scraping, I'm talking about manual
| usage.
|
| > _You can easily check it manually._
|
| You cannot. Twitter's site is plagued by redirect loops.
| If you work around those, these days /search just redirs
| to the login page. You can view single tweets, but there
| won't be any replies. (I have no idea if the site
| formerly known as Twitter is still rate-limiting views,
| or if they canned that.)
|
| It is unusable if you're not actively logged in, and some
| of us have no desire to give away a phone number just to
| see AWS's true status.
| londons_explore wrote:
| > communicate outages to services I use
|
| In general, they seem bad at communicating relevant
| information. Just looking over at emails from Google, _every
| single one_ of the last 10 emails they sent me was not relevant
| to me specifically.
|
| > "[Important notice] Tax changes in Nepal" (yet I have never
| made a sale in Nepal)
|
| > "Secure your Google Admin account with these best practices"
| (yet I already do all those things)
|
| > "Preparing for the upcoming Google Ads change on October 31,
| 2023" (it's all about mediation waterfalls, and I'm 90% sure I
| don't use them, and 100% sure I don't know what they are)
| Spivak wrote:
| AWS got this very right with the per-account incident
| dashboard.
| FireBeyond wrote:
| Except when their systems don't correctly discern what
| services you may be using...
| shizcakes wrote:
| We've been stuck in this state for all 9 days. We've filed
| tickets, etc, but no resolution has come about yet. Just re-tried
| yesterday, still not able to update nodepool.
| danjc wrote:
| Is it a big deal to just build a new cluster and cut traffic
| over to it via lb or dns?
| talonx wrote:
| The incident dashboard does mention creating a new node pool
| with the new version instead of upgrading - as a workaround.
| Curious to know why this was not an option for you.
| edude03 wrote:
| Creating a new node group works which is super easy on GKE so
| it's pretty much a non issue. Definitely frustrating but not as
| bad as it sounds at first brush
| endisneigh wrote:
| Sucks that there isn't anything simpler than k8s that's
| production grade.
|
| Maybe it's time to yolo with a regular container that just
| restarts on failures, ha...
| ctvo wrote:
| I use AWS ECS (sometimes with ECS Anywhere to use my own
| servers + ECS's control plane) for orchestration where I can.
|
| It's a little surprising how many folks are unaware of the non-
| Kubernetes orchestration options. Going full Kubernetes is
| rarely what organizations need. Getting CI/CD -> containers on
| servers and being able to configure the resources containers
| need, having the placement handled, etc. gets most of these
| folks 90% of what they want with much much less complexity.
|
| - AWS ECS:
| https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
|
| - ECS Anywhere:
| https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
| ldjkfkdsjnv wrote:
| ECS Fargate is a bit priecy, but is also a no brainer. It
| just works
| tebbers wrote:
| We moved from DigitalOcean to ECS 2-3 months ago and it has
| been rock solid. Highly recommended. There is some initial
| setup required with container registries and wrangling AWS's
| vocab but once set up, push to master, GitHub Actions builds
| and deploys.
| whoknowsidont wrote:
| _Erlang /Elixir screaming in the background_
|
| It's a wonderful, observable, manageable orchestration layer
| that's easily and safely customizable if you're wise enough to
| treat it as such.
| acedTrex wrote:
| HA k3s?
| sofixa wrote:
| Disclaimer: I work at HashiCorp, but have been of this opinion
| since before joining (source:
| https://atodorov.me/2021/02/27/why-you-should-take-a-look-
| at...)
|
| HashiCorp Nomad is a perfectly easy and flexible orchestrator
| (not only containers, also VMs, random binaries, Firecracker,
| WASM, etc.) that is production grade. It has a number of
| advantages over Kubernetes (being drastically easier to deploy,
| use and maintain; having integrated native templating with
| logic; being much nimbler but also being able to scale much
| more, etc. etc.) and it's a very good choice in many scenarios.
| Of course there's also disadvantages, most notably ecosystem
| size, but often that doesn't matter. Kubernetes is great and
| extremely powerful, but also very complex and straight up
| overkill for many organisations.
| nh2 wrote:
| We're using Nomad, and appreciate its comparative simplicity
| (we launch plain binaries created with Nix), but there are
| still basic features missing that can create struggles. The
| ones I hit are:
|
| * It can't work down batch jobs in priority order [1] -- a
| feature even the ultra-old university cluster schedulers had
|
| * Sometimes job starts can be in arbitrary order, in the API
| and the UI, making it difficult to find the most recent ones
| [2]
|
| * Nomad seems to store pretty much everything in its Raft
| state, making it extremely slow to startup and operate
| (including GUI) if some rogue process created 100k
| allocations. Cleaning that up seems to require nuking the
| Raft state, thus also losing all info about all jobs.
|
| I don't know how well Kubernetes does in these areas.
|
| [1]: https://github.com/hashicorp/nomad/issues/12792 [2]:
| https://github.com/hashicorp/nomad/issues/17742
| k8svet wrote:
| Someone I know once worked for the company maintaining etcd
| at the time and... they were appalled at how it was
| underpinning millions of dollars of VC dreams on it,
| implicitly via kubernetes, with the flaws and downsides,
| and operational complexities it has. And now to hear you
| say this of Nomad and Raft.
|
| I guess, my take away at this second is to really be
| skeptical of hype that I don't have years of operational
| knowledge and experience with. Which, to be fair, regarding
| my current tech du jour (well, 9+ years), I have a list of
| complaints that could span a short novela.
| stuff4ben wrote:
| yeah, but ya'lls licensing tho...
| [deleted]
| networkchad wrote:
| [dead]
| orangepurple wrote:
| https://www.hashicorp.com/products/nomad/pricing
|
| There is an unsupported community edition
| dharmab wrote:
| In case you haven't heard, the next version of Nomad is
| moving to a non Open Source license.
| mati365 wrote:
| Nomad solution moves complexity from tech to license ;)
| bsdnoob wrote:
| Hashicorp nomad comes to mind.
| jarym wrote:
| It does and most people that used Nomad were very positive
| about it. But with the turn that Hashicorp have taken I'm not
| sure that it now stands much of a chance.
| jdoss wrote:
| Seconded. Hashicorp Nomad has been a breath of fresh air for
| doing HA deployments for my workloads. Getting a small
| cluster setup to self host Nomad is so easier than Kubernetes
| and defining workloads is much easier to understand too IMO.
|
| The only negatives about Nomad is the Hashicorp license drama
| that has happened recently and persistent storage can be a
| pain in the ass.
| hughw wrote:
| I hope somebody can elaborate on the licensing problem? I
| see a community edition...
| gabeio wrote:
| https://news.ycombinator.com/item?id=37081306
|
| They pulled the rug out on a lot of people and people
| tend to hold grudges. Community edition is peanuts
| compared to using an open source tool you _could have_
| fixed your own bugs with by creating PRs or adding
| features with PRs. I used to be a huge hashicorp
| fanboy... _used to_.
| sofixa wrote:
| Just to clarify, you can still access the code, submit
| PRs to add features, etc. The _only_ difference is that
| during a fixed time period (4 years) you cannot use Nomad
| to compete with HashiCorp. If you want to do that, you
| can ask for a license.
| jdoss wrote:
| They changed to the BDL from the MPL
| https://www.hashicorp.com/license-faq which in FOSS
| culture is considered a dick move.
| dwroberts wrote:
| I love Nomad but having used it in two different roles and
| now invested time to understand k8s properly, I would
| absolutely not recommend it.
|
| Nomad is simple on the surface and could have been a great
| tool - but it is basically unusable on its own without
| tighter integration with Hashi's own tools (eg Vault).
| Configuring and maintaining all those things is nontrivial
| and ends up being more annoying (for a lesser end result)
| than just using a fully managed kubernetes cluster like GKE
| robertlagrant wrote:
| You'd imagine that any scheduler that's well integrated
| with Vault would have a huge advantage over other ones.
| Surprising that it's not like that for another product from
| the same company.
| throitallaway wrote:
| Vault is highly integrated with K8s via the vault-agent.
| nailer wrote:
| AWS ECS
| Spivak wrote:
| There is, you create a load balancer, an autoscaling group, put
| an ubuntu image in it, set the userdata to install docker and
| docker-compose up your desired containers.
|
| * Instance refresh to deploy new images.
|
| * Every so often update your AMI to patch.
|
| * Create an RDS database and hook the instances up to that if
| you need such things.
|
| It's really hard to get more boring but works than this. It can
| be set up in a few hundred lines of terraform. I promise the
| code for your entire AWS account will be less than just the k8s
| YAML.
| turtlebits wrote:
| What happens when all your services won't fit on a single
| box, or you need X more of a specific service?
| Draiken wrote:
| You create a new instance and run the same setup.
|
| If your workload is extremely variable, something like k8s
| with auto-scaling makes sense. But a lot of applications
| (maybe the majority?) have very predictable loads and you
| don't always need new services.
|
| When you're at the size of large corps with
| hundreds/thousands of services, then k8s is amazing. If
| you're not even close to that number, it's very likely
| premature optimization.
| secondcoming wrote:
| VM images aren't cool enough!
|
| We do this but create custom images that have all their
| requirements installed via ansible.
|
| Unfortunately there's now a push to move to GKE instead of
| GCE, for no real reason.
| dewey wrote:
| Maybe https://kamal-deploy.org could be worth a try there, as
| 37Signals almost moved all their services from Kubernetes I'd
| say it can be called production grade.
| speedgoose wrote:
| "Don't throw the baby out with the bathwater".
|
| It's just a bug in something most people don't use.
| input_sh wrote:
| They also had an issue with creating and deleting persistent
| volumes on 1.25. It lasted for 15 days, or half a month(!) last
| month:
| https://status.cloud.google.com/incidents/EBxyHQgEPnbM3Syag5...
|
| I'm also incredibly annoyed at them displaying time in PDT. I
| genuinely don't understand why they decided on that instead of
| doing something normal like UTC or detecting my timezone.
| Especially annoying every six months because Europe and the US
| don't do Daylight Savings Time changes at the same time, so for a
| week or two there's an additional hour I have to account for.
| wg0 wrote:
| May be this a hated take but just wondering - The place that
| pretty much invented cluster orchestration and reinvented it as
| k8s is having problems upgrading it.
|
| What chance a bunch of poor sys admins stand running bunch of k8s
| clusters for a mid size company I wonder.
|
| Every time I think of deploying (self managed, have done full
| stack) it for something mission critical, this upgrade scenario
| simply makes me rethink it altogether.
|
| And even managed k8s has no guarantees and if managed is to be
| the option, nothing beats ECS in simplicity and smooth operation
| at certain scales.
|
| PS: Full stack K8s means ingress controllers, DNS auto
| registration, GitOps , logging, monitoring, CI/CD and all the
| bells and wistles including a management UI behind oauth etc.
| 0xbadcafebee wrote:
| The design of K8s is ridiculous. The fact that you're forced to
| upgrade every 9 months is even more absurd. It's "opinionated",
| but the "opinions" are those of one of the largest companies in
| the world, who hire the most engineers in the world, and
| everyone else is supposed to operate like that now.
|
| K8s is a great example of a "tech jobs program": technology
| that is absurdly complicated to the point that you have to pay
| someone lots of money to make it keep working, but you can't
| afford _not_ to pay for this, because everyone has decided to
| be stuck on it and now there 's no alternative (and probably
| never will be)
| sangnoir wrote:
| > K8s is a great example of a "tech jobs program"
|
| No, it is not. K8s is an example of how it's impossible to be
| a jack of all trades and retain simplicity. If a tool seeks
| to be all things to everyone, then there's no sweeping the
| inherent complexity under the carpet.
|
| Full-coverage solutions to complex domains are themselves
| complex by necessity - see SAP, Salesforce, Oracle, Windows
| backwards compatibility, or try writing a parser/stemmer for
| a spoken language that evolved naturally :)
| eitally wrote:
| It used to be straightforward when Google Cloud just offered
| everyone the same version of GKE essentially everywhere. It was
| one of not-too-many services that are vanilla and identical in
| pretty much every region. But, newer offerings have fractured
| things a bit (sovereign cloud, Google Distributed Cloud,
| FedRAMP, Tencent, etc) and I can see why it would become
| problematic to adequately test and keep everything in sync
| while also not breaking interfaces and/or downstream services.
| Also, Google used to allow customers essentially 0 say in when
| upgrades to services were applied. Now, as part of many vendor
| security audit processes, it's a hard stop requirement that
| customers are able to test new revs before applying to prod ...
| and GKE, like GCE, is one of those critical services.
| moondev wrote:
| Check out cluster-api. Complete zero downtime cluster upgrades
| is as simple as bumping the version and machine image name in
| the cluster manifest - or ClusterClass if you run classy
| clusters
| markstos wrote:
| And why is that expected to be more reliable than Google's
| service?
| harikb wrote:
| An equally hated contrary take - may be it is our (as users of
| k8s) desire to do in-place upgrades without downtime..
|
| Sometimes a clean install of fresh cluster and switch over is
| so much simpler if we can just take a few minutes downtime to
| cut over.
|
| Complex requirements (sometimes) end up with complex solutions.
| llama052 wrote:
| I think it's definitely overlooked. You should always have
| the ability to spin off a new cluster and run your workloads
| on that. At a minimum as a disaster recovery scenario.
| xeromal wrote:
| I've never used K8s in my life but if you have to deal with
| downtime, I'd question the use of K8s when I can do it the
| old and busted way.
| harikb wrote:
| I am not sure what "old and busted way" you are referring
| to, but k8s does have value, particularly when we are not
| _upgrading k8s itself_
|
| It would be good to think of the occasional downtime during
| a significant upgrade (once in 6 months?) separate from the
| ability to automatically replace failed nodes, scale up and
| down, having all the k8s magic.
| PH95VuimJjqBqy wrote:
| I would imagine using a load balancer in front of
| multiple servers across several datacenters coupled with
| a stateless design that allows traffic to flow freely
| across either of them.
|
| there are other variations of this, but this seems the
| most obvious. blue/green deploys and the like.
| Freedom2 wrote:
| While I see your point and agree, there's also the view that
| the people who invented the tech are simply different than the
| kind of people who Google hires nowadays - pure algo focused
| LeetCoders with zero-to-little operational experience and
| little interest in even learning how to run and maintain
| software.
| akokanka wrote:
| This shows the astronomical complexity of k8 systems even gods of
| k8 fail.
| Closi wrote:
| To be fair, running a Kubernetes Cluster is simpler than
| running a Fully managed Kubernetes As-A-Service Offering for
| thousands of clients.
| DishyDev wrote:
| Not a great week for managed Kubernetes services as Digital Ocean
| have been having an ongoing issue since yesterday morning on
| their service
| https://status.digitalocean.com/incidents/fsfsv9fj43w7
| k8svet wrote:
| To be honest, I'm pretty disappointed that this thread doesn't
| have some attempt at a root cause now that two major managed
| providers are having extended outtages linked to upgrades.
|
| To think, I once was so in love with it. This kind of crap, and
| imagining the live-site chaos makes me so glad to have noped-
| out when I did. With my modest lifestyle, I probably passed on
| life changing money but I also have 1/10th the stress and
| anxiety I used to.
| truthwhisperer wrote:
| [dead]
___________________________________________________________________
(page generated 2023-10-10 23:02 UTC)