On my last straw with using k8s as homelab reddit / selfhosted ---------------------------------------------------------------------- asosnovsky · 2023-12-31 18:06 So I started this journey initially as a way to learn k8s better and to actually get some use of it. The services I’m hosting are 1. The arr suite 2. Jellyfin & Plex 3. Nextcloud 4. Frigate 5. Some self made web apps 6. Cert-manager 7. Traefik ingress My setup is as such I got 1 pc that I installed truenas on. It handles all my drives and 2 vms, one of which is running Postgres, and another running a Debian server as a k3s master node. Then I got 4 minipcs, 2 of which are k3s master nodes (each of these have 8 cpus) and the other are slaves (with 4 cpus). Each machine has around 16gb to 32gb each. These machines each run nixos. Feels like I have a stupid amount of juice, yet I keep having pod failures and “lack of resources” issues. I’ve made a post prior about optimizing the resource limits/requests. But all the strategies I’ve been shown didn’t work in way or another (even tried a mix of them at this point). Seems to me like using kubernetes just over complicates things for homelabs and I may as well just spin up containers on dedicated machines. And don’t even get me started on getting HomeKit discovery to work with go2rtc or Scrypted … that was such a pain. Should I just ditch k3s/k8s in favor of something like podman or rancher with basics compose files? ---------------------------------------------------------------------- MotiveMe · 2023-12-31 18:11 16 to 32gb is just low enough that you might be having some requests vs limits chaos in an environment that is a little constrained. You’d have to share more though to say for certain. Generally I will only set requests, not limits, and do so using the lowest conservative request I can justify. You should keep at it, though! Learning is supposed to be a little challenging at times. 🙂 ---------------------------------------------------------------------- wrath_of_reaper · 2023-12-31 18:24 Been there. Switched after some months back to plain Proxmox containers/VMs. K8S is great for learning, but you don‘t want to have it in home- production if you cannot spend at least a few hours a month in configuration/troubleshooting/messing around and want some sort of stable base infrastructure at home. ---------------------------------------------------------------------- icantreedgood · 2023-12-31 18:26 By master nodes I assume you mean control plane? Why do you have so many. I have a single control plane node and 3 workers in my cluster running probably 50+ pods all on top of a r710 running proxmox. My biggest resource constant is disk speed. ---------------------------------------------------------------------- asosnovsky · 2023-12-31 18:39 So my issue never seems like it’s ram. Based on the queries I see in Prometheus it’s cpu. Which is ridiculous since I used to run the arr suite on a raspberry pi and now I find myself dedicating almost a full node for it with 4cpu and 32gi. Tried setting only requests. In the context of the arrs, who normally sit at <100m when a search happens for a number of things they end up pushing all the other pods out. I basically ended up dedicating certain nodes for certain application. Which seem to keep stuff stable-ish. But I’m out of juice for anything else. Which again is silly, my home-assistant yellow hosts double the containers and it’s been running stress-free for over a year. ---------------------------------------------------------------------- lvlint67 · 2023-12-31 18:39 > Seems to me like using kubernetes just over complicates things Kubernetes is great for things built ground up to run in it. It is wildly complicated.. but it makes SOME things easy. Those things aren't always the same things as the things you want to do. ---------------------------------------------------------------------- asosnovsky · 2023-12-31 18:40 This is where I’m at as well. I’m debating proxmox vs rancher. Any reasons you chose proxmox? ---------------------------------------------------------------------- vanchaxy · 2023-12-31 18:43 You should always set RAM limits equal to RAM requests. You should NEVER set CPU limits and always set CPU requests. ---------------------------------------------------------------------- wrath_of_reaper · 2023-12-31 18:44 Rancher Harvester? Good in theory, unfortunately no practical experience (still on my to-do list). Proxmox because of simplicity (clustering, backup+retore, LXC, VM cloning, and off course: free). My 3-node-NUC-cluster runs stable for over a year now. Basically it‘s set and forget. ---------------------------------------------------------------------- shoesli_ · 2023-12-31 18:55 K8S is very abstract, even more so than Docker. So it can seem pointless when setting up at home with a couple of workers. But the advantage is that if your application runs on a whole datacenter full of servers you can deploy a full stack of new software, with ingress controllers, networking, load balancing etc to a thousand physical servers using a single configuration file and one command. You don't need to know what worker is running what pods or keep track of IP addresses. Kubernetes does everything for you. But in order for such automation, the configuration is much more complex than traditional software ---------------------------------------------------------------------- clintkev251 · 2023-12-31 19:22 Well if you want a highly available control plane, you need at least 3, so one is fine for no HA, 3 is minimum for HA, no clue why you would run 2, no HA there anyway so you may as well just run 1 ---------------------------------------------------------------------- clintkev251 · 2023-12-31 19:23 What are you setting for CPU requests? ---------------------------------------------------------------------- clintkev251 · 2023-12-31 19:24 Well you need 3 if you want a highly available control plane ---------------------------------------------------------------------- dargx001 · 2023-12-31 19:39 That’s a good idea for a corporate production environment, but a bit overkill for a home production environment. Especially considering they are running all theirs on the same machine. ---------------------------------------------------------------------- neyfrota · 2023-12-31 19:39 Same here. Keep simple and stupid. Proxmox at home with 2 desktop vms (ubuntu for work. Windows for some extras) and one vm for self-hosted things. Self hosted vm is debian server, a plain docker-compose.yml on git, tailscale and ssh .. just that. All personal files at synology nas, mounted as nfs at self-hosted vm then exported to containers. Love k8s, but only at work, production with daily watch. : ) ---------------------------------------------------------------------- clintkev251 · 2023-12-31 19:41 Kubernetes is overkill for a home environment, so if you're doing it at all, may as well go all the way. >Especially considering they are running all theirs on the same machine. And no, they're not ---------------------------------------------------------------------- niceman1212 · 2023-12-31 19:47 Why do you have only 2 master nodes? I don’t really see how resource allocation is your enemy here. I have over 180 pods running on 3-4 pcs (depending on power prices), with same or less resources than you and don’t have any issue. Can you share a more concrete example of a failure scenario? ---------------------------------------------------------------------- enchant97 · 2023-12-31 19:53 Kubernetes was too much for my homelab. However I still wanted high availability, so went with Docker Swarm with 3 nodes. So far it has worked very well. I still have dedicated machines for home assistant and a nas. ---------------------------------------------------------------------- voodoologic · 2023-12-31 19:56 I came from docker swarm and it’s more straight forward. But that may be because I have experience in docker. K3s was so hard for me that I became an expert at reinstalling it. Upvoted for voicing your frustration, you aren’t the only one. ---------------------------------------------------------------------- jgibbarduk · 2023-12-31 20:00 I did the whole k3s on bare-metal. I spent more time keeping it stable than I did using the apps I ran on it. I decided to go for Hashicorp Nomad with Docker. Never looked back! It’s a dream to run and rock solid. Have it running over 6 nodes 4 at home and 2 in Oracle Cloud (using Tailscale). ---------------------------------------------------------------------- BraveNewCurrency · 2023-12-31 20:01 >Well you need 3 if you want a highly available control plane But why do you want a highly available **control** plane? When the master goes down, your services keep running (you can adjust the timeout on this.) ---------------------------------------------------------------------- -quakeguy- · 2023-12-31 20:05 Well, you are not wrong. Kubernetes overcomplicates things for homelabs and not just that, most enterprise users who are considering kubernetes do not actually really want kubernetes. Is your primary objective to run stateless workloads at a massive scale? If the answer is no, that settles it, you do not want kubernetes. ---------------------------------------------------------------------- clintkev251 · 2023-12-31 20:12 Tons of reasons, sure pods that are already running will continue to run, but jobs will no longer be scheduled (so important things like backups will stop working), any pods that fail will not be rescheduled, no scaling actions will take place, and tons of other little things that degrade the clusters usability, even if the workers are still "up". More importantly, why *wouldn't* you want a highly available control plane? Control plane nodes are quite lightweight, I can't think of many reasons you wouldn't just set up 3 by default, unless you're *very* resource constrained ---------------------------------------------------------------------- SIN3R6Y · 2023-12-31 20:14 k8s, at it's core is designed to be more or less a container centric dynamic configuration and scaling solution. Yes, kinda word salad there, but seriously understand that word salad. The entire core idea is, i have service X and the service really only needs X containers. But, if a bubch of load come up, i want it to auto scale to N containers to handle that load. In a place where you may have thousands of services that fit that model, k8s is the best thing since sliced bread. But at home, where you probably only need one conatiner per app. And you don't really need all the app / base image segregation and config automations. Yes, it's a lot of extra work for no net benefit other than to say you did it. ---------------------------------------------------------------------- dargx001 · 2023-12-31 20:15 OP isn’t running them on the same machine you’re correct, but the commenter you replied to says their cluster is running on top of a R710. ---------------------------------------------------------------------- prime_1996 · 2023-12-31 20:15 Same here, I had a docker compose setup using LXC containers in proxmox. Migrated everything to docker swarm, and I am happy with it. ---------------------------------------------------------------------- clintkev251 · 2023-12-31 20:16 They have 3. I read it as 2 at first too, but they have 1 in a VM, and then two bare metal control plane nodes ---------------------------------------------------------------------- dafzor · 2023-12-31 20:16 From your description seems it's more an issue with the services you're running then k8s. If you want to "go back", you can remove resource declaration and set node affinity on your deployments to basically have the "manually deploy containers on hosts" while retail Ning all the goodness of k8s. Personally I use a 3 master k3s cluster (2vm + 1 pi) for over 3 years and am very happy with how much easier to manage it is vs bunch of machines with docker not to mention the amount of pre packaged things available. ---------------------------------------------------------------------- prime_1996 · 2023-12-31 20:17 Try docker swarm instead. ---------------------------------------------------------------------- clintkev251 · 2023-12-31 20:17 Sure, it makes sense why that commenter is just running 1, and it also makes sense why OP is running 3 ---------------------------------------------------------------------- JTech324 · 2023-12-31 20:21 I switched from proxmox to Harvester, running on 5 mini PCs. Running a rancher VM, using the native harvester drivers for RKE2 clusters, and it's a breeze. Really gives a managed cloud feeling. You can spin up and tear down kubernetes clusters in seconds, scale them, increase RAM and watch the nodes cycle gracefully. Rancher can also expose the storage classes from harvester to your guest kubernetes clusters, for dynamic PV provisioning. The harvester provider in Terraform is also a lot better than the one for proxmox. ---------------------------------------------------------------------- Ohnah-bro · 2023-12-31 20:24 I wanted to learn k8s but I virtualized all my nodes. I built 2 cheap pcs running proxmox plus a truenas with 30tb. The nodes all run as Ubuntu vms and where possible I mount storage in with nfs on truenas. Backup tasks run on truenas and everything works very well. Going on about a year and a half now. ---------------------------------------------------------------------- nukacola2022 · 2023-12-31 20:26 QQ, I’m thinking about clustering Proxmox and want to use tiny PCs. How important is it to have a separate NIC dedicated for replication traffic? And is it fine if that NIC is 1gbps only? Is 1 NIC setup fine for homelab environments? ---------------------------------------------------------------------- Richmondez · 2023-12-31 20:30 Which proxmox terraform provider? I agree the telemate one that pretty much all tutorials on the web use is a bit naff, but the bpg provider seems fairly comprehensive to me. ---------------------------------------------------------------------- wrath_of_reaper · 2023-12-31 20:37 That‘s indeed a good setting, I experienced some replication errors (timeouts) when running over the main switch. When I added some 1G USB Ethernet adapters for direct replication connection, the errors went away. But can be just my semi-pro setup, don‘t know if this is a real Proxmox issue. Single NIC with optional USB Ethernet is totally fine for home labbing. ---------------------------------------------------------------------- fbleagh · 2023-12-31 20:38 This is precisely why I run Nomad at home ---------------------------------------------------------------------- ck_bishop · 2023-12-31 20:39 Docker Swarm. Do that instead. There is absolutely no reason to use k8s in a homelab unless you're making a professional investment in learning it. ---------------------------------------------------------------------- Trustworthy_Fartzzz · 2023-12-31 20:42 I did the same. Now I’m all in on Ansible and Docker. Life is too short to manage a bare metal K8s cluster. ---------------------------------------------------------------------- Sheriff686 · 2023-12-31 20:54 K8s is overkill for typical homedervet applications. And I am someone earing most of my income with k8s. ---------------------------------------------------------------------- JTech324 · 2023-12-31 21:01 The telmate one is the only one I tried - I found it to be buggy and unforgiving towards my slow hardware lol. Lots of issues with timeouts, wanting to recreate VMs when it wasn't necessary, etc. I also tried the proxmox node driver for rancher and was never able to get it to work. VMs would spin up and never register as healthy, so rancher would keep replacing them in a loop. Not knocking proxmox - it was a fantastic hypervisor. I definitely miss the LXC management too. ---------------------------------------------------------------------- Windows_XP2 · 2023-12-31 21:03 Both of the Mini PC's in my cluster have a single 1Gbps NIC, and there doesn't seem to be any noticeable impact with replication. I don't constantly max out the NIC's on them and don't do anything crazy with replication, so your mileage may vary. ---------------------------------------------------------------------- Windows_XP2 · 2023-12-31 21:06 I agree, although I haven't been running it nearly as long as you have, and there was some additional configuration needed as I'm only running two nodes. Overall, it was pretty simple to setup and maintain, and it's pretty reliable. ---------------------------------------------------------------------- adamshand · 2023-12-31 21:23 How are you managing persistent storage for container failover? ---------------------------------------------------------------------- enchant97 · 2023-12-31 21:24 I use GlusterFS replicated over the 3 nodes. ---------------------------------------------------------------------- Particular-Way7271 · 2023-12-31 21:28 Docker swarm here as well. It’s pretty straightforward indeed and for home self hosted environment is doing the job ---------------------------------------------------------------------- adamshand · 2023-12-31 21:34 Thanks. Any complaints with Gluster? Did you try Ceph? ---------------------------------------------------------------------- enchant97 · 2023-12-31 21:35 No complaints so far, self healing seems to work well. I have not tried ceph. ---------------------------------------------------------------------- GoingOffRoading · 2023-12-31 22:07 Share your deployment yamls? K8S has a bit of a learning curve, but it's super easy once you have one deployment configured correctly ---------------------------------------------------------------------- Murky-Sector · 2023-12-31 22:07 For k8s does using configuration managers like ansible help with productivity? ---------------------------------------------------------------------- tperjack · 2023-12-31 22:09 K8s adds a lot of complexity so I personally don't think it is worth it in a home environment unless you're trying to upskill for professional reasons. That said, it sounds like you should have more than enough resources for the services you are running. How much CPU and RAM do you have in aggregate? CPU is "compressible" - if a pod exceeds the CPU limit it'll just be throttled. However, the throttling can cause it to be killed if it is starved to the extent that it fails liveness probes. Memory isn't compressible - if a pod exceeds the memory limit Kubernetes will unceremoniously kill it. When this happens, you'll see the OOMKill status on the pod. One recommendation is to only set requests for CPU, but for memory to set the request and limit to the same value. Bear in mind that a pod can only be scheduled if there is a node with enough capacity to satisfy the request value. So, if you set it incorrectly you could end up putting all your heavy services on one node and if they all exceed their requested resources, services on that node will start being killed. When K8s is deciding which pods to kill, it will first target pods that are designated as "burstable" - i.e. when request != limit. The Kubernetes docs are quite detailed: [https://kubernetes.io/docs/tasks/configure-pod-container/quality- service-pod/](https://kubernetes.io/docs/tasks/configure-pod- container/quality-service-pod/) [https://kubernetes.io/docs/tasks/configure-pod-container/assign- memory-resource/](https://kubernetes.io/docs/tasks/configure-pod- container/assign-memory-resource/) [https://home.robusta.dev/blog/kubernetes-memory- limit](https://home.robusta.dev/blog/kubernetes-memory-limit) https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure- eviction/ ---------------------------------------------------------------------- Varnish6588 · 2023-12-31 22:15 I have a two-node (one master one slave) kubernetes cluster with kubeadm. Thinking that my kubernetes master is a single point of failure, i designed my home Lab to be ephemeral. If that cluster dies someday, which i know it will, i will easily recover by redeploying all the apps again. For the pods that need to persist data, i mount NFS storage from my NAS, and the persistent volumes won't be deleted if my PV resource gets deleted. This way i can make use of the features from kubernetes without having to invest much money in so many servers. This setup has worked well, and i have tested the recovery process from the persistent volumes in the NAS. ---------------------------------------------------------------------- BraveNewCurrency · 2023-12-31 22:30 >I can't think of many reasons you wouldn't just set up 3 by default, unless you're very resource constrained You have a very poor imagination. **Everything** has trade-offs. * Control planes need CPU and disk, they aren't free. What you consider "light weight" may be considered "heavy" by someone else. * High Availability helps your uptimes, but when your HA fails, **it hurts your downtimes --** because now you have a much more complicated system to troubleshoot/fix. (i.e. Parts of [Github was down for a day](https://github.blog/2018-10-30-oct21-post-incident-analysis/) *because* of their HA.) * Maintaining an HA system requires a more complicated runbook. (i.e. Upgrades have many more steps, troubleshooting has more steps, etc.) Home labs (if done right) are usually trivial to "burn to the ground and re-launch" compared to "real" production systems. So if you can take 'some' downtime, it may be easier overall to run a simpler system without HA. ---------------------------------------------------------------------- clintkev251 · 2023-12-31 22:30 I absolutely hated using Ansible with Kubernetes. I love it for lots of things, but for a while I used Ansible for provisioning of k3s nodes and it always felt so fragile. I've since switched to running all my k8s nodes using Talos and that's felt rock solid basically out of the box. Obviously just my own experience though, maybe others have had a better experience ---------------------------------------------------------------------- Murky-Sector · 2023-12-31 22:39 Thank you for the insight! ---------------------------------------------------------------------- clintkev251 · 2023-12-31 22:45 >Control planes need CPU and disk, they aren't free. What you consider "light weight" may be considered "heavy" by someone else. "unless you're very resource constrained" Obviously if you have a very small footprint, HA doesn't make much sense. But then neither does Kubernetes in general. >High Availability helps your uptimes, but when your HA fails, it hurts your downtimes -- because now you have a much more complicated system to troubleshoot/fix. (i.e. Parts of Github was down for a day because of their HA.) Is an HA config more difficult to troubleshoot? Sure. Does it hurt your downtime? It shouldn't if you have anywhere near a reasonable setup. If you loose a control plane node with HA, you still have a working control plane to use while you troubleshoot. And realistically, you should be set up so you can just burn down the failing node and rebuild it from scratch relatively trivially. With a single control plane node you don't have that luxury. If your whole control plane cluster somehow explodes (which is very unlikely), then you're right back in the same position as if you had a single node of having to rebuild it, no worse off. And I'm very familiar with that Github incident. It's a great example of a bad and overcomplicated database replication system being bad and overcomplicated. Not an incitement of HA in general. >Maintaining an HA system requires a more complicated runbook. (i.e. Upgrades have many more steps, troubleshooting has more steps, etc.) >The HA may only kick in once every few years -- which may be longer than the home project lasts.. Yup, of course. "I don't want to" is obviously a completely valid reason in a homelab to not run HA ---------------------------------------------------------------------- lovett1991 · 2023-12-31 23:30 Yeha for homelab (well self hosting environment) I found k8s just wasn’t quite right. It’s great for things I deal with at work but not home stuff which is primarily designed to just be run on a single node and not as a scalable application. As others have said proxmox + lxc might suite better for self hosting. ---------------------------------------------------------------------- red-lichtie · 2023-12-31 23:39 I think that the technological challenge of running an HA kubernetes cluster is not for everyone. You might be better off running a "docker server" with a good back up solution instead of going with Kubernetes. ---------------------------------------------------------------------- threwahway · 2023-12-31 23:52 Am I reading this wrong, or do you only have 2 nodes to run all of your apps? You have 5 nodes total and 3 of them are master? Did you allow scheduling on you master nodes? Did you check to see where all your containers are running?