[HN Gopher] Comparing k3s with default Kubernetes
___________________________________________________________________
Comparing k3s with default Kubernetes
Author : jhoelzel
Score : 46 points
Date : 2023-04-03 14:48 UTC (8 hours ago)
(HTM) web link (www.hoelzel.it)
(TXT) w3m dump (www.hoelzel.it)
| nailer wrote:
| Kinda off topic but what's the actual word for k3s? We have
| Kubernetes k8s, addreesen horowitz a16z, internationalisation
| i18n, founders f6s. What is k3s?
| cwayne wrote:
| I like to tell people its "kates"
| CGamesPlay wrote:
| "kubes" -> "k3s"
| [deleted]
| stonemetal12 wrote:
| It isn't a word. From their documentation:
|
| We wanted an installation of Kubernetes that was half the size
| in terms of memory footprint. Kubernetes is a 10-letter word
| stylized as K8s. So something half as big as Kubernetes would
| be a 5-letter word stylized as K3s. There is no long form of
| K3s and no official pronunciation.
| jasoneckert wrote:
| In addition to K3s, I've used managed K8s, custom-rolled K8s, as
| well as various other K8s distributions. K3s has - by far -
| provided the least friction for most of my use cases, and is what
| I incorporate in any initial cloud design.
|
| Of course, other stakeholders and constraints may eventually mean
| that we adopt something else before it gets implemented, but K3s
| is what I start with for many of the same reasons outlined in
| this article.
| paddw wrote:
| K3s seems like a terrible name, given how people will be confused
| with the numeronym for Kubernetes.
| apetresc wrote:
| That's the joke, though? It's k8s, but "smaller".
| mulmen wrote:
| What are the three letters? It only makes sense for being
| confusing, which is the base state of Kubernetes.
|
| "Our name is literally a complex joke about complexity"
| doesn't inspire confidence in a project that supposedly
| simplifies complexity.
| infogulch wrote:
| kubernetes -> k8s
|
| k8s -> k3s
|
| Double numeronym'd
| joshmanders wrote:
| Wouldn't k8s -> k1s then?
| [deleted]
| legerdemain wrote:
| Kubernetes is often shortened to k8s, which is often
| pronounced "kates," which is further shortened to k3s.
| mulmen wrote:
| Isn't it actually keigths which means it should be k5s?
| mhink wrote:
| They took a chunk out of the left side of the "8", leaving
| "3". At least, that's how I've always interpreted it.
| sakisv wrote:
| Well, for me the whole replace-letters-with-numbers thing
| is idiotic and requires some inside knowledge to understand
| which letters are replaced by all the numbers in words like
| a11y i19n k8s, etc.
|
| On the bright side you can put whatever you want there.
| Personally I read k3s as "kerts", because why not. Similar
| to "thirds" but with the "t" from kubernetes.
| mulmen wrote:
| I choose to read k3s as "soft boiled egg" because that
| makes as much sense as anything.
| cwayne wrote:
| as the manager of the team that maintains it, this is now
| canon. We will use this exclusively in internal
| communications. thank you.
| mcmcmc wrote:
| The three letters are k, 3, and s
| mulmen wrote:
| The 3 is supposed to be short for three other letters.
| What are they?
| tetraodonpuffer wrote:
| It is a pretty decent name since you can easily search for
| information about it, unlike say "kind" (which I typically use
| for development) which is absolutely un-googleable
| kobalsky wrote:
| the title doesn't make sense, it reads like "comparing Ubuntu
| with Linux, How Ubuntu is often the better choice".
|
| k3s is a kubernetes distribution
| jhoelzel wrote:
| while true, writing "vanialla kubernetes" did not have the
| right feel for me.
|
| What would have been a better title for you?
| user61129028 wrote:
| "vanilla kubernetes" isn't a distribution either.
|
| > Kubernetes and k3s are both container orchestration
| platforms
|
| Your article makes it seem like k3s and k8s are different
| platforms which is simply incorrect.
|
| A more useful comparison would have been to compare k3s with
| another distribution like EKS, GKE, OKD etc.
| jhoelzel wrote:
| i kindly ask again, what is the distribution called that
| you setup with kubeadm?
| glotzerhotze wrote:
| you might be fooled by the meaning of the word
| ,,distribution" in the kubernetes vs. linux context.
|
| in the linux context, a distribution would be an
| opinionated build of user-land tools around the linux
| kernel which might also bring patches done to the kernel
| by the distributor.
|
| in the (cloud) kubernetes context, a distribution would
| be an opinionated (cloud) implementation of core
| kubernetes binaries (EKS/AKS/etc.pp.) on top of a linux-
| distribution of your liking.
|
| to my humble understanding k3s is a stripped down and
| optimized (IMHO opinionated) build of kubernetes binaries
| derived from the official source.
|
| kubeadm is the official projects (opinionated?) idea of
| arriving at a vanilla cluster, leaving you with the
| freedom to make choices for lots of the needed components
| involved.
|
| in the linux context, ,,linux from scratch" would be an
| analogy I suppose.
|
| more knowledgeable people should please correct me if I'm
| wrong in my understanding here.
| ttymck wrote:
| What is the official kubernetes distribution called?
| iameli wrote:
| There isn't one -- the official docs give instructions for
| kubeadm, kOps, and kubespray. k3s would be an alternative to
| those.
| ithkuil wrote:
| Most of the other K8s distributions use the same K8s
| binaries. K3s is slightly different in that it's a trimmed
| down build of the same codebase
| MuffinFlavored wrote:
| i think you mean kubeadm and not kubeam
| iameli wrote:
| ty, edited
| wg0 wrote:
| I would lean towards k0s over k3s. The article has one piece of
| information wrong. K8s no more comes with docker. It is
| containerd I think.
| awinter-py wrote:
| hmmmmm no mention of pvcs eh
| jhoelzel wrote:
| Longhorn ( https://longhorn.io/ ) has been stable for a long
| time now and is also a could native foundation project =)
|
| You can start with a single storage node without replication
| and easily go from there to triple replicated storage
| ph4te wrote:
| Been using k3s(kube-vip + MetalLB) and Longhorn for both my
| personal projects and work projects for a while without any
| issues. It's a pleasure to use instead of the alternatives.
| zzyzxd wrote:
| I tried k3s several times in the past few years but I still can't
| use it in my homelab:
|
| 1. the project claims to be production ready and support HA
| control plane setup, but there's no solution for API load
| balancing out of box. How do you bring up a new node(either
| control plane or worker node)? You write down the join token
| produced by the first control plane node, and hardcode the token
| and the existing control plane's IP in the new node's systemd
| unit file. Btw, if you use the official installation script, that
| file is going to have permission 755 and everyone on the server
| can just read that token.
|
| 2. And how do you bring up the first control plane anyways? The
| official instruction is to `curl` a bash script and pipe into a
| shell. You can probably translate that script into some ansible
| playbook, but the whole running-a-bootstrap-script-and-passing-
| along-secrets approach make the whole process difficult to be
| converted into some something that's supposed to be idempotent.
|
| All the problems can be worked around, in fact I was half way
| there, but then I suddenly started thinking: "didn't I choose k3s
| because I thought it was easy?"
| jhoelzel wrote:
| Truth be told yes that tricky but can be managed with ansible
| easily for instance.
|
| 1) Your main problem though would probably be the need for a
| haproxy or bgp which does load balancing for you. There are
| other solutions like kube-vip but they are more a "failover"
| solution that HA. Which would be fine for a homelab and is for
| instance how Rancher Harvester (kubernetes for virtual
| machines) does it.
|
| 2) you have to pass a parameter called --cluster-init for the
| first node and then join the other nodes. once the cluster is
| running you dont need any node with that parameter anymore and
| its common practise to create the first node wiuth --cluster-
| init then join 3 other ones and take down the first node
|
| And on a personal node, you sound like you would be happy with
| rancher harvester. check it out its bascially turnkey
| zzyzxd wrote:
| Yup, when I said that I was half way there, I was trying to
| use ansible to deploy a nginx systemd unit on every node to
| load-balancing requests to API servers. Many other
| distributions can do this with a static pod managed by local
| kubelet, but that's not possible in k3s.
|
| I know there are solutions to my problem, but I can either
| implement them by myself, or like you mentioned, I can just
| use a different distribution and not having these problems.
| cwayne wrote:
| Could check out https://github.com/k3s-io/k3s-ansible althought
| admittedly it's a bit in need of some lovin'
| zzyzxd wrote:
| Yes, I did. Unfortunately, that playbook only supports single
| master.
| fellowniusmonk wrote:
| Thank you for sharing this and saving me time in exploring k3,
| it's shocking how common it is that an evangelized tool is
| impractical to setup and use in even a simple homelab
| configuration.
|
| Knowing what NOT to investigate because it isn't "ready" can be
| one of the biggest time sucks.
| ph4te wrote:
| I've gone through this a few times recently and have it in my
| homelab and at the office. What works good for me using kube-
| vip to a VIP on the control plane, and then MetalLB to dish out
| private addresses in the respective networks, or even
| statically assigned addresses. I have been turning them all up
| with k3sup which works like a charm.
|
| turn up the first node, install kube-vip, switch config to
| point to the vip, turn up all my other master nodes, then turn
| up my workers, install metallb, setup my subnet, install
| rancher, expose it with a LB, install longhorn. then start
| deploying things. here is an example of what i use to turn up
| the first one with k3sup. all of the servers are turned up and
| configured with ansible doing minimal updates, users, sudo
| access, etc..
|
| k3sup install \ --ip=192.168.1.11 \ --user=k3s-user \ --sudo \
| --tls-san=192.168.1.10 \ --cluster \ --k3s-channel=stable \
| --k3s-version=v1.24.12+k3s1 \ --no-extras \ --k3s-extra-args "
| --flannel-iface=ens160 --node-ip=192.168.1.11" \ --merge \
| --local-path $HOME/.kube/config \ --context=k3s-lab
| MuffinFlavored wrote:
| > but there's no solution for API load balancing out of box
|
| Why do you need this in a homelab?
| alexellisuk wrote:
| I'm a big fan of K3s, however managed Kubernetes from a large
| cloud vendor with a track record has a lot to offer when it comes
| to reducing management and the need for an SRE for K8s itself.
|
| Folks might also be interested in two free resources:
|
| 1 - K3sup https://github.com/alexellis/k3sup - the author
| mentions HA K3s - K3sup is an easy way to get that using SSH.
| It's also a good pairing for K3s with Raspberry Pi 2 - Kubernetes
| at the Edge with K3s (CNCF / LF course) - I was commissioned to
| write this and I talk a lot about the differences and also the
| origin story of K3s and what Darren was aiming for.
|
| Have fun with Kubernetes - whichever flavour you go for.
| gerty wrote:
| Is there any advantage of running k3s if you want to keep etcd? I
| understand that most k3s performance gains come from etcd being
| replaced by sqlite but if you still want a HA control plane,
| sqlite won't cut it.
| jhoelzel wrote:
| you can easily still use external etcd if you really need to.
|
| But in general k3s can be HA without issue and scaled just as
| well as vanilla k8s. The main advantage of it is that
| everything comes neatly packed into a single binary whereas the
| alternative would mean to have a multitude of services running
| for cluster provisioning.
|
| Kubernetes in the end is basically an API server with multiple
| componets and k3s puts a nice bow around all of them.
| iameli wrote:
| We've been using k3s' embedded etcd for as long as its existed,
| and it's great. Setting up the etcd cluster is dramatically
| simplified; let the first node generate a token and feed it to
| all the other nodes. Tons of other advantages to k3s; the
| single-binary deploy process, the built-in networking stack
| (which you can secure with Wireguard out-of-the-box), built-in
| ingress controller if you want one.
| kamikazechaser wrote:
| Title is misleading. k3s is a deployment stack/distribution that
| builds off various Kubernetes modules. It must pass a certain
| test suite to conform to Kubernetes standards.
|
| What you might be trying to compare is kubeadm which is the
| official deployment stack provided by Kubernetes.
| jhoelzel wrote:
| while somewhat yes but no.
|
| im not trying to compare it with kubeadm (which is a more a
| setup script https://kubernetes.io/docs/reference/setup-
| tools/kubeadm/ ) but with the fact that vanilla kubernetes
| comes with moving parts that have to be configured and
| maintained and also updated separately.
|
| you can actually setup "kubernetes" which is often referred to
| as vanilla kubernetes without it too. See "Kubernetes the hard
| way" by kelsey hightower.
| dang wrote:
| Ok, I've stuffed "default" in the title above. If someone wants
| to suggest a better (i.e. more accurate and neutral) title, we
| can change it again.
| iameli wrote:
| Been using k3s in production for nearly four years now and would
| recommend it to anyone. Super lightweight and easy to deploy.
| Opinionated about stuff I don't care about while allowing for
| customization of network stack, backing database, and ingress
| controller if you want to do it yourselves. Their embedded etcd
| is way, way easier to set up than a custom etcd distribution.
| jhoelzel wrote:
| I have built clusters for clients with k3s too and it has
| always been a charm. More recently i was able to bring up a
| Dualstack cluster for a VOIP company that now can basically
| scale endlessly with gitops on top.
|
| The fact that you can integrate existing Sysadmin Teams because
| they will understand that a program that runs a service with a
| binary and a config is all it takes, is worth its weight in
| gold.
|
| They know their Loadbalancers and haproxies as well as how to
| provision true raid systems that are not software based which
| almost makes disk failure go away and maintenance really
| sheduleable.
| wouldbecouldbe wrote:
| It was 01:30. About to go to bed. Next day would fly with my
| daughter to holidays.
|
| I checked the apps of a bunch of clients. None of them loaded. I
| was like what...
|
| I checked the server. Everything down.
|
| I'd been running Kubernetes on Digital Ocean. And Digital Ocean
| forced a Kubernetes update that was incompatitable with mine at
| night.
|
| Took me 8 hours to fix it. No sleep. Ended up moving it back to a
| good old VPS. And throw away K8s.
|
| Now to be fair, I had been getting warnings with deployments. But
| I was used to that, Kubernetes has 10 updates per week. I dont
| have time to update K8S or my helm files every week.
|
| So yeah it was my fault, but I was used to good old VPS hosting.
| There is an old php application I've build 5 years ago with
| Laravel, that never needs anything. I did some updates and
| patches, but it always just works.
|
| Im used to running node on apache or nginx, even though a bit
| less stable, still almost never crashes.
|
| Kubernetes there is always something. I guess there are reasons
| to choose it, but it's not stability.
|
| I ended up taking the plane, and my daughter was super kind and
| patient. But no more kubernetes for me.
| jhoelzel wrote:
| I feel your pain and i would lie if a k8s update never caught
| me off guard, but there are still policies you can set for the
| autoupdate of k3s or trigger them manually at will.
|
| have a look at k3s and maybe you will like kubernetes more
| again. There is no magic to it. Have a faulty node? spin up a
| new one.
|
| And if your hoster is kind enough, there will even be apis with
| cloudinit for you to do that.
|
| Im not trying to imply that you need a managed k3s with this
| post, but rather trying to show how easy kubernetes can be if
| you leave the big clouds and try not to overcomplicate things.
| b33j0r wrote:
| Yep. My problem with a managed cloud is that kubernetes drivers
| are vendor specific. I hit this landmine too when the ingress I
| was using got a breaking update (as far as I could tell).
|
| Still worked on my test cluster. It was a throw away site, so I
| didn't even bother fixing it. And, I honestly can't tell you
| what went wrong. Just deploying docker compose, often with
| ansible, is the most reliable for me at most scales.
| ilyt wrote:
| Yeah, that's the annoying part. You get around 80% of parts
| that just work anywhere and rest is entirely dependant on
| what's underneath k8s or on what cloud it is running.
|
| When going in I assumed there are some kind of standard way
| to implement ingresses/loadbalancers but no, it's just
| different plugins each with different syntax and features.
| amq wrote:
| In my experience, helm is the worst offender. Somehow, every
| second chart update has a breaking change preventing the
| update, defaults that don't mind wiping the persistent volume
| and a discontinued somewhat consistent 'stable' central repo
| make me seriously regret using helm charts for anything that is
| not ephemeral.
| bithavoc wrote:
| DigitalOcean also cordons Nodes for no reason at random times.
| Currently moving to EKS exactly for this reason
| taspeotis wrote:
| In late 2021 we had EKS nodes shit the bed for no apparent
| reason. Max out CPU and couldn't SSH in. They'd get recreated
| eventually by EC2 (multiple hours) despite being tainted in
| k8s pretty quickly. We ran with N+2 nodes to deal with it.
|
| After dutifully updating EKS releases and AMIs it seems to
| have fixed itself last year.
|
| EKS seems to be fine generally, but it would be nice if they
| didn't release new AMIs with missing commits?? [1] and
| especially [2]
|
| [1] https://github.com/aws/containers-roadmap/issues/319
|
| [2] https://github.com/awslabs/amazon-eks-
| ami/issues/278#issueco...
| atmosx wrote:
| Can you share more details please? How did you find out it
| was DO, did you open a support ticket?
| alyandon wrote:
| My experience so far is that k8s itself is relatively stable.
| It's when you start using vendor specific addons/plugins to
| actually do stuff like provision PVs, modify Citrix LB
| settings, etc outside of k8s everything quickly becomes a
| burning tire fire. :-/
| davidkuennen wrote:
| Sounds like Digital Ocean was the problem, not Kubernetes.
|
| Been running managed Kubernetes in GCP for years without any
| issues.
|
| They are bugging me to update for some time now, but I don't
| think they would force an update on me.
| tinco wrote:
| I hope you're right, I haven't had time to look at the same
| warning for 6 months now. There was a hard deadline on it in
| February and it came and went. The backwards incompatible
| Kubernetes api updates are a PITA. I know they said "beta",
| but if you literally have a version field in your manifest
| format, why ever deprecate a way of configuring something?
| They could just have operators translate the configs
| automatically on the fly.
| Rantenki wrote:
| >They could just have operators translate the configs
| automatically on the fly
|
| How do they test that across the infinite number of
| permutations of configurations and deployments of K8S in
| the field though? It'll work for some people on the happy
| path, but it's really hard to maintain over time. Worse,
| it'll break randomly at some point in the future that is
| hard to predict, instead of at some publicly announced
| point in time where the breaking change is deployed (how it
| happened this time).
| wouldbecouldbe wrote:
| I don't know any software library that introduces
| breaking changes as often as kubernetes does. Devops used
| to be leaning more to stability then devs, it's weird.
| Kubernetes devops are very expensive so I guess they love
| being able to spend more time on their tool
| kobalsky wrote:
| gke is pretty good at that, if there is a forced upgrade
| coming they will let you know months in advance.
| beebmam wrote:
| There isn't a single managed vendor of Kubernetes that
| won't force an upgrade (or shut down your entire cluster if
| you don't upgrade). It is too costly for these companies to
| maintain so many old versions of Kubernetes that are not
| supported.
|
| If you want a Kubernetes cluster that never gets forcibly
| updated or shut down, you should not be using a managed
| service, period.
| kobalsky wrote:
| I may have not expressed it well but what I wanted to say
| was that GKE communicates very clearly when you will be
| hit with a forced non-patch upgrade, not that upgrades
| are undesirable. Running a cluster without updates is a
| deathwish.
| neuronexmachina wrote:
| For anyone curious about the actual schedule:
| https://cloud.google.com/kubernetes-engine/docs/release-
| sche...
___________________________________________________________________
(page generated 2023-04-03 23:01 UTC)