Blog Post: Building a "Cheaper" Kubernetes Cluster at Home reddit / selfhosted ---------------------------------------------------------------------- Hdmoney · 2023-10-03 15:05 This is a project I've been working on for a little while now, mostly on the backburner. It could be useful for others here! I think this would be great to reproduce with the RPi5 and a NAS, instead of just Orange Pi 5s with NVMe drives. Let me know what you think! ---------------------------------------------------------------------- lidstah · 2023-10-03 16:30 Hey, good post, well written (at least for a non-native english speaker like me), without "configuration code blocks bombing" :) MetalLB is great, did you use the ARP or the BGP advertisment? (I use the former at home and the latter at work). Another (and last) question: persistent storage can be quite a hassle on baremetal clusters, have you considered the nfs-subdir-external- provisioner storage class? easy to setup as long as you have a NFS server somewhere in your network, and does the job for a homelab, imho. There's also Piraeus datastore, which uses drbd and linstor under the hood, if you want (or need) to use your nodes' local storage. And as you mentioned, there's Longhorn, Mayastor, rook-ceph and probably a dozen other solutions :) Wishing you the best in your employment search! (Although with posts like this, you should land a devops job quite easily) ---------------------------------------------------------------------- Hdmoney · 2023-10-03 18:00 Thanks! What a wonderful comment \^\^ I use L2Advertisement, so just ARP for now. It would be fun to get an ASN and build out a more intricate network though! Unfortunately I don't have any separate NAS _yet_, and I didn't really look into NFS _anything_, beyond reading up on some comparisons. But, I have Mayastor running with the workers' NVMe drives, which works well enough for now. I'd love to have a bit more separation of concerns -- we'll get there eventually. Y'know, time and money and all that. I definitely need to explore my options once I can afford a NAS. :) ---------------------------------------------------------------------- lidstah · 2023-10-03 19:18 You're welcome! > I use L2Advertisement, so just ARP for now. It would be fun to get an ASN and build out a more intricate network though! It's indeed quite fun - and MetalLB can use both methods at the same time (so, no risks to break anything/going offline while experimenting!) > Unfortunately I don't have any separate NAS yet, and I didn't really look into NFS anything, beyond reading up on some comparisons. NFS can be useful (the easy way) if you want to separate compute and storage. It also have some drawbacks, especially: sqlite and NFS are not really friends - in my experience, jellyfin works well, but navidrome not at all. OTOH, I run several postgres and mariadb databases in cluster with nfs volumes since years without a hitch. Performance on writing lots of small files (i.e lots of random I/Os) is not good at all (even abysmal) so it's not the ideal way to store big git repositories for e.g., but read speed is decent and write speed on big files is quite good too. Note that by default, k8s use nfs v4 to mount NFS shares. But as you say, right now Mayastor (which I should try on a spare week-end (read: week-end without friends at home and my son at grandma's home)) is doing the job for you, and as we say: if it is working, don't change it :). By the way, have fun! ---------------------------------------------------------------------- kmisterk · 2023-10-04 18:23 Thank you for your share! For future reference, we ask that you create a text post with the link to the blog in the body of the text, and a few sentences on why it's relevant to the community. We look forward to future content. Cheers, /r/selfhosted ---------------------------------------------------------------------- Hdmoney · 2023-10-04 18:31 Ah, Rule 6, I see now. I only skimmed the side bar and neglected to click the rule link. Will keep that in mind for the next one :) ---------------------------------------------------------------------- kmisterk · 2023-10-04 18:33 You bet. Thanks for sharing. :)