Need some advices for a 4 nodes centralized storage reddit / selfhosted ---------------------------------------------------------------------- Aekolol · 2023-12-09 23:29 Hello ladies and gentlemen! I’m looking for an advice for my next project, a 4 nodes cluster homelab. Context: These nodes are Intel N100 CPU, with 12 Gb on ship and a 512 Gb NVMe (they are all Chuwi LarkBox X 2023). I plan to self host some services (my website, monitoring stack grafana+victoriametrics, linkwarden, gitea and more to come) into a Kubernetes cluster. To do it, I’ll install Proxmox on every nodes and create a lightweight VM for master and workers nodes of k8s. This part is automated (cloud init, open tofu and some ansible roles). The problem: I can’t find a way to have a centralized storage. I cannot use a NAS nor another PC. It is not possible to add storage on node (only one NVMe port). I had tested ceph into the main disk (create an osd on the free space and mount it into Proxmox, it worked but there’s some glitches and the cluster became unusable. ZFS seems not a solution because ZFS replication is a manual task (sort of). GlusterFS… I had many bad experiences with network issues and corrupted bricks. What can I do to have a centralized storage..? I’m fond of hyperconverged systems and I really want to find a way to do it with this environment. Thanks for your advices :-) have a great day! Edit: Maybe I’ll have to keep one node for a TrueNas server..? ---------------------------------------------------------------------- OtherMiniarts · 2023-12-09 23:44 A proper Ceph setup is the objectively correct answer to your problem (maybe swap the SSDs with something more reliable like Samsung, Solidigm, Micron, etc.) but realistically a NAS is the simplest shared storage solution for clustering. A small TrueNAS or Synology setup would be ideal. Create an NFS share for thin provisioning and call it a day. Also, ProxMox clusters work best with an odd number of nodes due to quorum and split-brain decisions. ---------------------------------------------------------------------- Accomplished-Bad137 · 2023-12-09 23:45 You could use ZFS on all nodes and replicate.... Poor man's solution. I ended up getting two Synology flash stations fs2500 as central storage. Would do it again ---------------------------------------------------------------------- Aekolol · 2023-12-09 23:50 Hey thanks for your reply :). Ceph was working but as you said it will need proper setup (only one OSD per node). Ceph is also very verbose on logging and can be a SSD eater :-(. About quorum, That’s why I use 4 nodes at the moment. ---------------------------------------------------------------------- Aekolol · 2023-12-09 23:52 Hey 👋 is zfs able to replicate on all nodes automatically “like ceph”? It can be a solution! I’m not a fan of NAS storage, because it’s another material to keep up to date and so on :-( ---------------------------------------------------------------------- AxisNL · 2023-12-10 02:20 You get a quorum with 3 or 5 nodes, not 4. With four nodes, if two are down, you get a split brain situation. But you can have cluster with 3 monitors, and use osd’s from 4 hosts of course. ---------------------------------------------------------------------- AxisNL · 2023-12-10 02:23 Although I have no experience with kubernetes, you could create a Ceph vm on each host, and run CephFS across these nodes. Then mount cephfs in your kubernetes vm’s. In this case your proxmox host would not see any shared storage, but your vm’s will. Kind of a poor man’s solution, but if you can’t do Ceph natively because a lack of dedicated disks, this should work fine. ---------------------------------------------------------------------- Aekolol · 2023-12-10 07:53 Interesting, it seems to be the solution… only 3 monitors up and maybe a fourth in stopped state (if one monitor goes down, I’ll start it manually)..? ---------------------------------------------------------------------- trustmeitsfine · 2023-12-10 09:53 If you're going to use Kubernetes and Ceph, consider using Rook Ceph, which runs all of Ceph inside K8S. ---------------------------------------------------------------------- AxisNL · 2023-12-10 13:36 I don't believe Ceph supports standby monitors, and I think three is enough. Depending on your chosen data replication algorithm, Ceph will stop working anyway if 2 nodes go down, because it won't have enough replica's to prevent data loss. So you should be fine with 3 monitors. If you really want more hosts to be able to go down, you could to 5 monitors (have MON-04 and MON-05 on a single node), and do a 4-way replication instead of the standard 3-way replication. ---------------------------------------------------------------------- ElevenNotes · 2023-12-10 18:15 With 12GB RAM you will not get any cluster storage to work. Any chance you can upgrade that RAM to at least 32GB? ---------------------------------------------------------------------- Aekolol · 2023-12-10 18:21 Nope impossible unfortunately. Ram is soldered and cannot be upgraded or replaced. The socket of the Intel n100 support max 16 Gb (low powered cpu and memory) ---------------------------------------------------------------------- ElevenNotes · 2023-12-10 18:22 Then you need a centralized storage solution since with that little RAM available per node you can forget to run any cluster object storage. ---------------------------------------------------------------------- Aekolol · 2023-12-10 18:24 That’s the point. I’ll think I’ll won’t go into a cluster and let hosts individual. And I will work into a better backup solution;) ---------------------------------------------------------------------- ElevenNotes · 2023-12-10 18:29 glhf ---------------------------------------------------------------------- Fluffer_Wuffer · 2023-12-11 00:15 I've used GlusterFS for years, the trick to not getting corruption is make sure you switch on BitRot checking and self-healing etc.. I've had the very corrupt file, but the fix is to remove it from each individual Gluster server (not via where you've got it mounted). I've long had a love hate relationship with it... but for me, it's been much more simple that Ceph, and I use the same cluster to provide storage foe multiple K8S cluster - which I've never been able to do with Ceph Rook. ---------------------------------------------------------------------- Aekolol · 2023-12-11 21:26 Hey nice to hear that, thank you for the note! Ceph is a source of headhache... I'll give a try for GlusterFS soon with the BitRot checking. ;) ---------------------------------------------------------------------- Aekolol · 2023-12-11 21:35 First, thank you all for all your reply, advices and notes. I really appreciate to push forward the reflexion. I finally managed to go on a 3-nodes cluster (instead of 4 initially), install ceph within Proxmox GUI, go with only 1 monitor and 3 manager... and it works! To create the Ceph OSD, I created a partition from my free space of my disk and choose the type "Ceph OSD". After, I created OSD from the Proxmox GUI and.. It works..!!! Yes, performance are not insane and it is not very fault tolerant... But Ceph 18.x is able to auto-configure and auto-tune your ceph cluster. As of today, I have a running PVE 3 nodes cluster with Ceph. This is a homelab, so crashes or low performances are not a problem. Thanks all ;) ---------------------------------------------------------------------- soamsoam · 2023-12-15 14:52 I recommend to try linstor, see https://youtu.be/kMU3JAsFXQk ---------------------------------------------------------------------- Aekolol · 2023-12-17 08:10 Interesting, never heard about it. I’ll check it out, thank you 😊