Suggestions for increasing storage in Proxmox cluster reddit / selfhosted ---------------------------------------------------------------------- Snirlavi5 · 2023-12-08 06:51 Hi all, I've been running Proxmox for about a year. Started with a single server and recently added a second server as a cluster for the ability to do some maintanance on a machine and keep essential services running. One thing I feel I haven't quite figured out is the best way to handle storage. I have 7Tb total on my main server in a ZFS "RAID0" configuration, and 2Tb on my secondary sever. I have configured replication for the most critical services I have running (Home Assistant, Adguard, NPM, Vaultwarded..) but services using lots of storage like Jellyfin or Immich can't be replicated because of the storage diff between the machine. I don't have a great feeling with the data safety here as if one drive fails on my main machine I for one of these I believe I'm screwed. I tried to look at Ceph but from what I gather, it isn't going to work well on my 2 server cluster with 1Gb ethernet. I also haven't fully understood if it's possible for me to add drives to an existing zfspool. Any advice or suggestions on what you guys would do / change to improve this setup would be very much appreciated. ---------------------------------------------------------------------- waywardelectron · 2023-12-08 15:29 Firstly: any type of RAID0 config is considered Extremely Bad because if ANY disk dies, you lose the entire array and will be restoring from backups (you do have backups, right?) Ceph won't play nice without at least 3 nodes. Running ceph on small clusters can be very tricky. Easiest thing to do would be to have a third machine or a NAS that exports NFS or iSCSI. You could perhaps look at something like a shared gluster volume but I'm not sure how well proxmox plays with gluster. You can add a vdev to a zfs pool but zfs won't automatically redistribute your data. It will just start using the new vdev when new data gets written. I know the ZFS project is working on expansion so this might have changed recently or will change maybe soon. ---------------------------------------------------------------------- Snirlavi5 · 2023-12-08 21:23 >shared gluster volume Thanks for your reply. Yes I'm aware of the Raid0 issue and I do have backups :) So for a NAS - 1. Performance wise is it good enough to use instead of local disks ? 2. Wouldn't this still result in a single point of failure but just the NAS server now instead of one of the proxmox nodes? ---------------------------------------------------------------------- Lopsided_Speaker_553 · 2023-12-08 22:40 Have you considered adding 5tb to server 2. I know it sounds simple but... I've tried to manage a cluster with 3 ceph nodes and didn't like the amount of time it all took to keep it running. Then I spoke to these kubernetes guys. They had multiple Proxmox servers running dev and insisted on storage per server and replicating. Upgrading is also easier since you can just install a fresh server with better specs. Good luck! Ps we had 10gb direct links between servers and still performance was mwah ---------------------------------------------------------------------- Snirlavi5 · 2023-12-09 12:41 hm.. Thanks for the input. I might just do that. ---------------------------------------------------------------------- waywardelectron · 2023-12-09 17:17 Sure, the NAS can become a SPOF: even though the disks might be redundant, if the motherboard blows, you're out of luck. It entirely depends on what problems you're trying to solve. If you want either proxmox machine to be able to run anything, having storage outside of them is one of the options. Another option is just to split your workload and replicate a backup between the two proxmox nodes so that you can restore a backup if one of them goes down. This is a more manual process to get things back up and running. Proxmox can do HA but you need min 3 nodes for quorum and such. You're very limited with the number of nodes you have. Which isn't necessarily a bad thing, since it does keep things simple. But if you want higher availability, you'll need to either increase the complexity of your setup, add more nodes, or make other tradeoffs to solve the things you're trying to solve. ---------------------------------------------------------------------- Snirlavi5 · 2023-12-09 18:42 Yes so I'm currently working with 2 nodes and another quorum device on a Raspberry PI and I did set up HA for a few of the critical workloads. I think for now I'll follow the advice below and just stay with replication for now and increase the disk capacity of the second node. ---------------------------------------------------------------------- kmisterk · 2023-12-09 20:14 ## Post Removed Hey, Snirlavi5! Thanks for your submission on selfhosted. Your post has been removed due to a violation of the purpose of the subreddit: **Self Hosting** ## Not Self-Hosted When it comes to posts regarding applications in this subreddit, they must feature a self-hosted tool, or a tool that *can be* self-hosted, or some kind of related information, help request, or otherwise related to a tool that is something that one can self-host. If you believe this post *is in fact* relevant to /r/selfhosted, feel free to [message the mods](https://www.reddit.com/message/compose/?to= /r/selfhosted&subject=Removed For Not Being Self- Hosted&message=[Removed Post](https://www.reddit.com/r/selfhosted/comments/18dh0ef/-/\)) with your concerns.