[HN Gopher] Kubernetes: The Documentary
___________________________________________________________________
Kubernetes: The Documentary
Author : wallflower
Score : 121 points
Date : 2022-02-13 17:22 UTC (5 hours ago)
(HTM) web link (www.youtube.com)
(TXT) w3m dump (www.youtube.com)
| tombh wrote:
| I love Kubernetes, but this documentary makes me sad. The story
| of PaaS is absolutely worthy of a documentary, but to imply that
| Google is the protagonist is just pandering to the "rich kid" who
| brought little more to the table than money and influence.
|
| Here's a better sketch for the story. It starts with Ruby, itself
| a game-changing language, that inspired a new culture of
| developer-centric best practices like Rspec's TDD and Bundler's
| lockfiles.
|
| Ruby made the first PaaS possible, Heroku (a YC startup), which
| itself inspired a whole new way of thinking about software. I
| don't think enough people realise just how influential Heroku's
| 12 Factor App manifesto[1] was, in fact I'd argue it's the real
| hero in this whole story. The 12 Factors are just text, it's not
| even software or hard and fast requirements, they're merely best
| practices for the entire experience of writing, collaborating,
| testing, deploying, scaling and maintaining software. The 12
| Factors aren't just about pushing your app on Kubernetes or
| whatever, they're just as much, if not more so, ideas that make
| _you_ a better engineer. Heroku made money for sure, but they did
| something beyond money, they upped the level of our craft as a
| whole.
|
| The central concept in Heroku is the dyno. If you follow the 12
| Factors, then you magically get almost infinitely scalable dynos.
| It plays right into the heart of the new gold rush world of
| startups, where anybody can have an app idea and leverage the
| global marketplace with the abstractions of datacentres and the
| cloud.
|
| Docker is the open source dyno.
|
| So many PaaS projects apart from Kubernetes have risen and fallen
| around Docker. Deis, Flynn, Dokku, Swarm, to name but a few. So
| many of the core concepts of Kubernetes were innovated and
| explored, and arguably mastered, before Kubernetes came on the
| scene. Kubernetes is a success not because of its innovation, but
| because it's backed by The Rich Kid. The world was already
| heading in this direction, Google just threw big numbers at it to
| make sure they took the biggest piece of the pie.
|
| 1. https://12factor.net
| democracy wrote:
| I think GAE (Google App Engine) was in production before Heroku
| got anything out there...
| tombh wrote:
| Yes they were. But like I say, I think the 12 Factors were
| more important. And Heroku in general just understood hacker
| culture better.
| throwaway894345 wrote:
| I think you're very deeply immersed in the Ruby/Heroku world
| and projecting that onto Kubernetes and everything else.
| Kubernetes' derives from Google's lengthy experience with Borg,
| which almost certainly predates Heroku. I doubt very much that
| Google was steeped in Heroku/Ruby culture. Yes, Heroku dynos
| are similar to linux containers, but containerization (the idea
| of binning multiple applications onto a single operating system
| instance) predates heroku by at least 6 years and the idea of
| binning multiple applications onto a single _hardware_ instance
| goes back much further.
|
| Moreover, Kubernetes and Heroku aren't even in the same class
| of tools. Kubernetes is an orchestrator for containers, which
| obviously can be used as the foundation for a PaaS, but
| Kubernetes isn't itself a PaaS even when you consider the
| various managed Kubernetes offerings which include some higher
| level abstractions built atop Kubernetes primitives.
|
| Further still, it's not like Google marketed Kubernetes to any
| significant degree prior to its own adoption of Kubernetes as
| its official container orchestration offering for its cloud
| (which was after the writing had been on the wall for a while
| that Kubernetes would become the de facto container
| orchestrator).
|
| TL;DR: Just because other tools and technologies predated
| Kubernetes didn't imply that it was significantly impacted by
| them.
| p_l wrote:
| And arguably borg and global work queue are predated by
| _generic batch queue_ of VMScluster (and possibly TOPS-20
| Galaxy, but I haven 't played with those) and those are
| predated by _IBM Job Entry Subsystem 3_ , which even operated
| in similar manner to k8s with "master" mainframe, usually a
| lower power model, scheduling jobs on multiple "worker"
| mainframes.
| tombh wrote:
| I haven't written Ruby nor used Heroku in many years. I think
| it's more relevant to say I'm very deeply immersed in Open
| Source. I contributed to OpenRuko[1], Deis, Flynn and even
| wrote my own PaaS, Peas[2] that got on the front page of HN.
| I've seen first hand the journey that PaaS has taken. Maybe
| we're getting hung up on the strict definition of PaaS here.
| Let's just say the thing that lets us easily scale a startup
| idea, like Uber, Airbnb etc. Sure the dyno/docker is the core
| component, but we inevitably have to deal with state, Redis,
| Postgres etc as well. In that sense Kubernetes is very much
| the Open Source version of Heroku.
|
| I think it's fairer to say that Google _repurposed_ Borg, in
| order not to be left out of the changing landscape. Borg may
| be the technical predecessor to Kubernetes, but my point is
| that Borg is very much not the spiritual ancestor of the new
| containerised paradigm.
|
| 1. https://github.com/openruko
|
| 2. https://github.com/tombh/peas
| throwaway894345 wrote:
| I would be very interested to hear why Heroku and not Borg
| is the spiritual ancestor of Kubernetes because that claim
| is contrary to everything I've read and listened to on the
| matter.
| xcambar wrote:
| I've been involved in multiple migrations to public cloud
| infrastructure (some were unnecessary, but such are the
| trends...), and without failure, the 12 factors manifesto shows
| up in the very first days of the discussions as the first
| target to reach before anything else is doable.
|
| I very much enjoy it and I'm glad whether I see someone mention
| it or I can spread and teach its wisdom.
| cbb330 wrote:
| This is an interesting story but it's nothing more than fan
| fiction. Please provide some reason for your assertions.
|
| While the entire comment could be contradicted, a quick example
| is that google's SRE book details much more developer
| strategies, including from a massive scale and enterprise
| perspective, than 12 factor web app. This book was assembled
| but industry experts with experience from Borg, with borg
| emerging from the global work queue -- decades before heroku
| existed.
| tombh wrote:
| > google's SRE book details much more developer strategies
| [...] than 12 factor web app
|
| I would argue that's precisely the problem. Google are
| clearly capable of doing complex things, but inspiring and
| nurturing an entire cultural shift is a different category of
| "complexity", which in the story of Paas, Google did not
| achieve, they merely exploited it.
| throwawaymanbot wrote:
| tester756 wrote:
| Why Google despite having Borg and know-how so early wasn't 1st
| to the Cloud? or bigger threat to AWS/Azure?
| singron wrote:
| Also, Google's infrastructure is so vertically integrated that
| it's not obvious how to offer it others. E.g. Borg assigns 1 IP
| to each machine, which means every pod-equivalent has to listen
| on a dynamically allocated port, so Google eschewed DNS in
| favor of a system that more easily assigned dynamic IP:port
| pairs without relying on TTLs (BNS/Chubby). Google also
| decoupled storage from compute and had a bunch of custom
| databases to run in that paradigm (GFS/CFS/BigTable).
| ra7 wrote:
| I've heard Google considers (considered?) their
| infrastructure prowess as a secret sauce. Could it be that
| they didn't want anyone else to benefit from it?
|
| They had years of headstart on most infrastructure technology
| that's ubiquitous now, but didn't bother to offer it to
| others. Perhaps they thought only Google should benefit from
| it, while Amazon saw a business opportunity with AWS.
| oblio wrote:
| I'm also curious about this.
|
| The empirical evidence seems to point that way, a lot of
| people were noticing that many of the Google stuff became
| open source or got open source equivalents from Google
| (Borg/Kubernetes, Blaze/Bazel, etc.) only after a critical
| mass of former Google employees left Google and since they
| liked Google tooling started replicating it as open source
| software outside of Google. In some cases that open source
| software got traction and Google seemed to be falling
| behind in some areas.
|
| Which basically tells me that in the software world, you
| can't really have trade secrets. I could be wrong, though.
| buck4roo wrote:
| Don't forget Prometheus (clone of borgmon by ex-SRE)
| jeffbee wrote:
| There is no way to perceive Prometheus as Google falling
| behind, though. It's more of a lesson about the perverse
| willingness of the public to subject themselves to
| terrible technologies that Google began deprecating years
| ago.
| oblio wrote:
| Why is Prometheus terrible? It's pretty cool.
| jeffbee wrote:
| At small scale it's fine. Its architecture totally
| forecloses any possibility of distributed evaluation, so
| there is a brick wall of scalability. Its human-readable-
| text-over-HTTP scraping scheme is also hilariously
| inefficient. It is not, in short, "planet scale".
|
| https://www.vldb.org/pvldb/vol13/p3181-adams.pdf
| oblio wrote:
| Well, I get the point of Google not falling behind with
| their tech, but even most Google competitors aren't
| working at Google scale.
|
| For Prometheus they could have just chosen to aim for
| smaller businesses, since that's 99% of businesses out
| here.
|
| So it could just be a tradeoff.
| disgruntledphd2 wrote:
| If you look at development of lots of "Big Data" stuff
| (Hadoop is a good example), Google shipped papers, while
| FB shipped code (not entirely true for Hadoop, but much
| more true for some of Google's later papers).
| samstave wrote:
| Google and FB (and apple, but less so?) -- have been making
| their own servers and hardware for years...
|
| They protect this aggressively, and the tech is impressive.
|
| I remember the first time I accidentally saw a custom
| google node when I went there to pick a friend up from
| there, and I had gone to his desk and I saw a really nice
| looking PCB on his desk he forgot about, and I exclaimed
| "WHATs that' -- I was pretty familiar with a wide range of
| server boards, and I knew that was custom immediately...
|
| He panicked covered it and we didnt really talk about it
| much, but it was known really early that they were building
| a lot of custom kit.
|
| ~2005 maybe?
| deanCommie wrote:
| The mistake also reveals the constrained imagination: Google
| could only conceive of offering what they had internally to
| others, struggled, and continues to struggle.
|
| Amazon didn't try - they just knew that they were good at
| building high scale distributed systems, and thought they
| could build services for others. That's what S3 and SQS and
| EC2 were - built from scratch, not trying to figure out how
| to externalize the internals.
| ec109685 wrote:
| GCP was built from scratch as well.
| servytor wrote:
| It is because Amazon dog-fooded their own technology stack.
| This forced them to have a better interface to their
| infrastructure, which in turn made it easy to open to the
| public. Google on the other hand, hid behind their own
| complexity and let engineers feel stupid for not knowing Borg
| like the back of their hand.
| jthrowsitaway wrote:
| Google's advertising business is really profitable so anything
| else is secondary. Amazon came at it from a business angle.
| They had a ton of spare resources that they only needed once in
| a while and figured out a way to make use of them.
| pid-1 wrote:
| amazon.com also was really profitable when AWS was launched.
| IDK, at this point I think we can just agree Google is bad at
| diversifying. The day their ad business dies, they're done.
| kortilla wrote:
| It was profitable, but nothing like Google's margins. It
| was still just a glorified Walmart with delivery (i.e. not
| much of a moat).
| ec109685 wrote:
| Amazon.com wasn't really profitable. They ran at low
| margins for years. In 2006, their profit was like 500m
| against 10B in revenue.
| oblio wrote:
| That could take a ton of time to die off and they could
| still pivot.
|
| But they're the least diversified, after Facebook, agreed.
| emodendroket wrote:
| The fear that they were plateauing was a major motivator.
| Google doesn't really face that.
| randomsilence wrote:
| >They had a ton of spare resources
|
| Is this true? There was the story about the resources they
| need for Christmas shopping that went unused for the rest of
| the year.
|
| But recently somebody mentioned here (I don't remember where)
| that AWS was a separate business and that Amazon itself
| initially didn't use any resources from AWS.
| dastbe wrote:
| building a public cloud is more or less a rebuild of any
| infrastructure you currently have. it would be just unwise
| for a company to build their private cloud with the same
| trade offs as public cloud.
|
| so while there's the ideal state of running amazon on a
| large multi tenant cloud, there was absolutely no safe and
| reasonable path to making that happen by extending the
| existing infra. it would've also hampered aws by forcing
| them to focus way too large way too early.
| FranGro78 wrote:
| That's correct, the multi year project was known internally
| as MAWS - Move to AWS.
| jeffbee wrote:
| Bingo. There are two facts you need to understand Google's
| relationship with GCP. Cloud margins are the worst margins of
| anything at Google, so if there has to be a choice on where
| to spend money, it can only go to Cloud begrudgingly, or as
| part of some long game. Second thing you need to know is
| Google's internal workloads are way, way bigger than you are
| imagining. So if you see a headline that Google invested $100
| billion in datacenters, don't imagine that all, most, or even
| a lot of that went toward GCP capacity. They spend that money
| on search, advertising, and letting an AI play ten million
| years of StarCraft.
| nevir wrote:
| > They had a ton of spare resources that they only needed
| once in a while and figured out a way to make use of them.
|
| Not actually the case. AWS was a completely independent stack
| (from hardware on up) - we couldn't use it internally for
| quite some time (years); and even then, only in a limited
| fashion (e.g. only S3 for some time)
| deanCommie wrote:
| Hubris.
|
| Steve Yegge talks about it on his podcast:
| https://youtu.be/9v4z46Ea35Q
|
| Amazon actually works backwards like they say - they ask
| customers what they want, and build it.
|
| Google thinks they're smarter than everyone else so they can't
| comprehend that developers out there in the world want
| different things than the developers inside Google.
|
| You can see this permeate even in this documentary, which is
| ostensibly about a success - when Google actually did launch
| something that developers really liked - K8S.
|
| They didn't start from developer needs or wants. They started
| by noticing AWS was eating their lunch, and wanting to compete
| with them, but realizing they couldn't win by doing it
| directly.
| ec109685 wrote:
| To be fair, K8s has caught on even outside of GCP.
|
| Also, while Amazon does work backwards from customers, part
| of them working backwards is how to translate customer
| requirements into a successful business. Amazon is doing this
| as a for profit company, not spending money on altruistic
| initiatives.
|
| E.g. purely for the benefit of customers, having AWS services
| be multi-cloud and open source or reducing egress cost would
| be best.
| mohanmcgeek wrote:
| They didn't realise it was a high margin business until AWS
| numbers were reported separately.
|
| A lot of people used to think that Amazon.com was subsidising
| AWS. Financials from other VPS providers suggested this to be
| the case.
| ec109685 wrote:
| GCP has been in the cloud business since 2008, two years
| after aws launched. They clearly were in a position to
| predict AWS's profit margins before AWS publicized them given
| they were competing for same customers.
| jakupovic wrote:
| I remember when GCP launched, we had an all hands meeting
| for all of AWS, which all fit in a big conference room,
| less than 100 people. People thought now that Google got in
| the business that was it. The leadership, Andy Jassy,
| assured everyone that our jobs were safe and we could get
| back to work so to say. Seems funny now, but at the time
| AWS was S3 and EC2 and google was ahead it seemed.
| sorry_outta_gas wrote:
| Google is poor at support, web hosting/cloud is as much about
| support as it is technology
| hackandtrip wrote:
| In my little experience, although GCP docs leaves a lot to be
| desired, the support was always very thorough and fast to
| answer, even for not critical/emergency situations.
| sorry_outta_gas wrote:
| They have improved a lot recently
| paulluuk wrote:
| To be honest, AWS cloud support isn't that fantastic either.
| Responses to tickets usually come in 48 hours, if not more.
|
| I don't know about you, but in 48 hours I've usually already
| built a workaround and moved on to other projects.
|
| Not saying that AWS support is worse or even equal to GCP
| support, but just commenting that cloud support in general
| feels pretty useless and an afterthought at best.
|
| And let's not even get started on the horrible, awful UI in
| AWS console.
| grogenaut wrote:
| pay for enterprise support, response is within minutes
| ec109685 wrote:
| Borg was good enough that google engineers didn't have any
| desire to use GCP and thus most of the company was not
| dogfooding its own product.
|
| Additionally, the "two pizza teams" style of Amazon's
| development was more amenable to the AWS model of development
| where each piece of Amazon could run effectively in AWS [1].
| Google is structured differently, and it's only recently that
| both AWS and GCP could even approach running the largest
| systems from a company like Google.
|
| [1] Yes I know that AWS wasn't built specifically for Amazon
| engineers, but the fact that it could accommodate them over
| time has been a boon for their cloud, and it gives enterprises
| confidence that of AWS's strategic importance and desire to
| make it world class. There isn't a large hidden layer of
| services that Amazon has access to that the general public
| doesn't.
| emodendroket wrote:
| One theory I've heard (I think from Yegge?) is that Google's
| internal tooling is so good at dealing with breaking changes
| that they're too cavalier about introducing them.
| throwaway984393 wrote:
| Even if that were the case, they clearly don't want to invest
| any extra time or money in refactor. So much of Kubernetes'
| design has been problematic, yet they just limp on monkey-
| patching and bolting on features that take years to go from
| idea to implementation. If you want any major changes or
| simplification in K8s, wait 3 years.
|
| A traditional FOSS project is not limited in this way. The
| developers often just start a new major version and overhaul
| everything. But they don't have to answer to "the business"
| or huge institutional users. Commercially driven open source
| will always be a product beholden to stakeholders, rather
| than technology designed to operate as well as it can.
| emodendroket wrote:
| The last task I want to be doing is having to migrate my
| Kubernetes definition files to accommodate someone's bright
| new idea.
| oblio wrote:
| > A traditional FOSS project is not limited in this way.
| The developers often just start a new major version and
| overhaul everything.
|
| And regular users hate this.
|
| Jamie Zawinski describes this funnily, eloquently and also
| offensively:
|
| https://www.jwz.org/doc/cadt.html
|
| > This is, I think, the most common way for my bug reports
| to open source software projects to ever become closed. I
| report bugs; they go unread for a year, sometimes two; and
| then (surprise!) that module is rewritten from scratch --
| and the new maintainer can't be bothered to check whether
| his new version has actually solved any of the known
| problems that existed in the previous version.
|
| But he's right, and that's why:
|
| * regular application developers who don't care about
| ideology
|
| * and mass market users
|
| hate desktop Linux.
| dastbe wrote:
| if you have the right company setup, headcount, and culture
| then you can absolutely build a world where teams can be
| allowed to force "perfection" over time, even with amazing
| levels of automation.
|
| i think the interesting thing is that while you'd think that
| is the ideal everyone should strive for, the reality is
| that's inapplicable or detrimental to run most businesses
| that way.
| oblio wrote:
| > build a world where teams can be allowed to force
| "perfection" over time, even with amazing levels of
| automation.
|
| I'm confused, did you mean "without" instead of "with"?
|
| Also:
|
| > i think the interesting thing is that while you'd think
| that is the ideal everyone should strive for, the reality
| is that's inapplicable or detrimental to run most
| businesses that way.
|
| Why would you say that is?
| dastbe wrote:
| i meant with. google has built some really great tools
| for minimizing toil when pushing major changes across a
| codebase. even then, there's a nontrivial overhead that
| is borne across the company.
|
| > Why would you say that is?
|
| many/most businesses don't have the margins/scale/time to
| focus on it. in a lot of businesses (amazons included)
| the goal for the business is to try as many things as
| fast as possible and scale that which works and abandon
| what fails. the goal of infrastructure there is to get
| out of the way and enable the layers above to build
| things as naively and simply as possible with little to
| no churn.
| emodendroket wrote:
| I think we see a good reason here: normal people don't
| think "this API is so beautiful," they think "these
| clowns are making me do a migration yet again."
| baalimago wrote:
| Who is this documentary for? Anyone who's interested surely
| doesn't need an explanation on what a docker container, or open
| source software, is
| hackandtrip wrote:
| I am no expert, but I know what a container is and I use it in
| my daily job; same for Kube, but hearing the actual people
| going through it feels a bit like an history lesson.
|
| (The documentary also touches various interesting point, such
| as open-source in the eyes of upper management, and so on...
| surely there is some better source/book/article, but I found
| the format pretty chill and easy to absorb)
| epgui wrote:
| I think this is high-level enough to appeal to non-technical
| people. I've shared this with some friends who have money
| invested in cloud companies they don't really understand, and
| with my family for whom even my simplified job description is
| too abstract to really comprehend.
|
| I think this is good, but it makes me wish for an even broader-
| scoped documentary on the history of cloud technologies.
___________________________________________________________________
(page generated 2022-02-13 23:00 UTC)