Selfhosted kubernetes cluster reddit / selfhosted ---------------------------------------------------------------------- Sivarion · 2023-10-01 11:26 I was thinking about migrating my services to self management k8s cluster, mostly for education purposes. I have some low traffic services running on one dedicated server on Scaleway now - there's some nextcloud, portainer, hastebin (sort of pastebin) mysql database, personal wiki, couple of half-dead blogs, stuff like that. Nothing resources heavy, nothing that needs tight SLA. I plan to add a little of my own software anytime soon, but I don't expect it to be very needy as well. I'm planning on going with 3 nodes with min. 2 cores and 4 GB of RAM, and an additional service for storage. Does anyone here had experience with self hostes kubernetes? Do you have any advices? Do you think nodes like these are enough or I should upgrade (or downgrade)? Can you reccomend some providers that could offer this setup at low cost? ---------------------------------------------------------------------- ripnetuk · 2023-10-01 12:33 I used k3s and use the same node as master and runner. I started off with 3 nodes like you are planning, but soon discovered they for home use it just adds overhead in keeping up to date and backing up, and I had to find some way of doing persistent storage without standing up yet another VM as a NFS host, so downsized to a single node. ---------------------------------------------------------------------- timothyclaypole · 2023-10-01 13:07 I’ve three rPi4 8Gb models running K3S with each node being both worker and control plane - works very well and I use keepalived and HAproxy to ensure the control plane remains available if any of the raspberry PIs need to reboot. I’ve a Helios NAS running NFS for shared storage across all of the nodes. It’s all reasonably low power but it has enough capacity to run a bunch of containers for media handling etcetera. ---------------------------------------------------------------------- Sivarion · 2023-10-01 14:18 I thought about RPis cluster as some form of test environment. I'm rather going to stick with cloud providers as I don't have static IP address, my electrical grid is rather sh*tty and is prone to outages and my connection can get a little bit unstable from time to time. As said before my services might not have a tight SLA, but I still want to be able to access my data while I have power loss at home. ---------------------------------------------------------------------- syfr · 2023-10-01 15:42 If your host OS is Ubuntu can use snap and microk8s it’s what I use sudo snap install microk8s —classic There’s lots of different ways to get your cluster up look at the different ones available and make your decision on that Kind Rancher K3s Microk8s Are the ones that come to mind atm ---------------------------------------------------------------------- from-nibly · 2023-10-01 18:17 I use nixos for my cluster. If you aren't familiar with nixos I wouldn't recommend hosting k8s as the way to learn it. If you are it's an amazing way to run k3s. Outside of that I would focus on whatever you can do to get the simplest k8s deployment (k3s et al) and then start focusing your learning on managing the configuration of kubernetes not the deployment. If you want a simple storage solution I would make one node an nfs device and go from there. If you want a magic (but complex) storage solution I would go with ceph. It's bulletproof to everything but you. Cloud providers for k8s are going to be expensive because there's a lot of overhead with running k8s from a hosters perspective. Those machines look fine to get started, so I wouldn't upgrade until you start running into problems, but even then I would focus on one big machine first instead of many little ones. Having many little machines gives you redundancy, and when you are just starting out that doesn't matter. ---------------------------------------------------------------------- Sivarion · 2023-10-01 19:36 Thank you! :) I've heard about NixOS, but that's it - never used it and not even sure what the features are - will read about it for sure. So, if I understand you correctly - your advice is to stay on my dedicated server, setup k3s on NixOS, be prepared for some difficulties along the way and learn from there. Just to clarify - I'm not looking for managed kubernetes from a cloud provider, sorry if I didn't make that clear. I'm looking for VMs or dedicated servers (just discovered they can be cheaper than VMs these days!) to set up and manage clusters myself. I probably wouldn't do much learning on the managed. At the moment I have a server with 16GB RAM and 8 cores - 1TB HDD might be the biggest performance bottleneck here, but I think it will be sufficient for single node cluster. ---------------------------------------------------------------------- maxf123456 · 2023-10-01 19:52 Try Ubuntu & k0s with k0sctl. ---------------------------------------------------------------------- opensrcdev · 2023-10-01 21:47 I just run a single master node with a couple of slave nodes. K3S makes it easy to add new nodes to a cluster. Make sure you have a good data backup strategy though. If you're hosting databases, or other internal application state, and a drive dies, you are going to wish you had backups. For databases, just take a periodic dump of the DB and store it somewhere separate (ie. minio on a different node). For other applications, you'll want to take periodic snapshots of the entire filesystem, where the app stores its data. ---------------------------------------------------------------------- Sivarion · 2023-10-01 22:08 At the moment I use S3 as persistent storage or backup, if Object Storage is not an option. Works acceptably, only nextcloud loads way too slow. I want to check if I would be able to utilize Scaleway Block Storage for this purpose. ---------------------------------------------------------------------- Sivarion · 2023-10-01 22:09 Didn't know k0s, but after initial reading this looks quite good. I'm willing to try it with NixOS, as suggested by someone above. ---------------------------------------------------------------------- AnomalyNexus · 2023-10-01 22:22 I've done this. It works, but isn't all that great. >2 cores and 4 GB of RAM I wouldn't bother with anything less than 3x rasp4 8GB. And you'll need to boot off SSD. SD card is going to be a miserable experience. I'd also consider doing 3x one litre minipcs instead though. ---------------------------------------------------------------------- Sivarion · 2023-10-01 22:24 I was thinking more of VMs in cloud than RPis. But yes, SD card is not an option :) ---------------------------------------------------------------------- AnomalyNexus · 2023-10-01 22:30 eh sorry...must have read the rasp comment from someone above. Yeah I've done that too - the free tier on oracle cloud works for this (sorta). ---------------------------------------------------------------------- Sivarion · 2023-10-01 22:35 Mind sharing caveats? I guess you don't run it anymore - what was the dealbreaker? :) ---------------------------------------------------------------------- M_R_Ducs · 2023-10-01 23:45 Did this over the last month. First I created a 3 node proxmox cluster from my existing homelab, and created 6 vms (2 per node) + 1 for a control plane. I used Debian for the OS and installing kubernetes, docker were fine aside from having to build the docker-cri, if you've never done it before. k8s itself is fine, however if you plan to run things like minio with multiple disks, it may be easier to do it on bare metal. ---------------------------------------------------------------------- clearlybaffled · 2023-10-02 03:07 Im self hosting k8s at home with a single custom build i7 8 core and 32 GB of ram. I just got tired of running after docker compose files everywhere and wanted it all under one roof. Plus work was getting heavy into it for our product so it seemed like a good idea. I'm really happy with my choice, and I haven't really stressed it much at all. Currently running MySQL, postgresql, rook-ceph, and they are probably the heaviest hitters right now. I also built it by hand using just kubeadm (well actually wrote/stole customized a bunch of ansible to manage a bare meta single node kubeadm cluster). ---------------------------------------------------------------------- vdubster007 · 2023-10-02 05:36 Check out kube-hetzner. Great project for self hosting with lots of options. Even supports Hetzners new Ampere servers for lower cost. I deploy three management nodes and 3 agent nodes with longhorn storage. [kube-hetzner](https://github.com/kube-hetzner/terraform-hcloud-kube- hetzner) ---------------------------------------------------------------------- AnomalyNexus · 2023-10-02 09:17 Nope...just checked...still there. 336d uptime lol. Just don't use it. The main gotchas are the free tier means you end up with 4x nodes with 1 vcpu each which I'm pretty sure is below min spec for k8s. Also, oracle has a tendency to delete people's accounts, especially if they're not paid accounts ---------------------------------------------------------------------- Sivarion · 2023-10-02 09:27 Thank you for the heads up :) I won't use it for my cluster, but it's good to know that such free option exists. ---------------------------------------------------------------------- AnomalyNexus · 2023-10-02 09:29 There is also a very generous discount on google's arm instances at the moment. Around 200 bucks a month. Only till march tho