how to achieve service redundancy or reliability reddit / selfhosted ---------------------------------------------------------------------- pepitorious · 2023-12-29 13:25 Hi there! Right now, for self hosting I have: * Unraid box, an intel 10400 with 64gb of ram, this is my main machine, bunch of docker and a few VMs running here. * A cheap nuc with 8gb of ram, started as a debian but installed proxmox on top to try i out. A couple of services here * Raspberry pi 4 8gb of ram. ​ The nuc and the pi are mainly to have redundancy on the data for the docker containers in my main machine. it backs everything up weekly and syncs those backups. ​ My question is about service redundancy and reliability. How to achieve that if a machine goes down, everything is still online... automatically. so far I have read this can be achieved with: * Kubernetes, I think it's something on top of docker, right? * proxmox cluster * ?? What are the options here? I will probably need to get more hardware to achieve this but I am ok with it. what are the pros and cons of each option? thanks in advance! ---------------------------------------------------------------------- elh0mbre · 2023-12-29 13:46 I run a 4 node kubernetes cluster (3 NUCs and a NAS type machine) for this and love it. The learning curve is pretty steep though. Auth the admin dashboard was especially tricky. It also requires everything you want to run be containerized (not an issue for me as I want it this way). Ive never used proxmox but if it has clustering capabilities and automatic failover and/or load balancing it should work. ---------------------------------------------------------------------- pepitorious · 2023-12-29 13:51 How does it manages replicating data in several nodes? I mean I have one service that connects to a posgres sql which is another container. I assume that service and that posgres would run in each node? ---------------------------------------------------------------------- NiftyLogic · 2023-12-29 13:53 I’m running a Nomad cluster here, works quite well. All services are docker containers, and any of my compute nodes can go down, the containers will just restart on the other node, Fetch storage from the NAS and reconfigure automatically. Takes about 20 seconds until everything is back up again. Setup: Synology 723+ for storage, plus two VMs for proxmox and Nomad/Consul (you need three nodes for HA). 2x Dell Optiplex 3050MFF with 16GB RAM as compute nodes, running Proxmox and an Ubuntu VM. Very happy so far, and I think Nomad is less black magic than K8s. ---------------------------------------------------------------------- pepitorious · 2023-12-29 13:55 Thanks! I'll learn more about nomad, did not know about it! ---------------------------------------------------------------------- NiftyLogic · 2023-12-29 13:59 Storage sits on the NAS and is mounted via NFS. Much easier to do snapshots and backups that way. So Postgres is running on some node, using the DB storage on the NAS. If the Postgres node goes down, Postgres starts on the other node, mount the storage and the Nomad/Consul magic takes care of making the DB transparently available to other services. They don’t care if the DB is running on compute1 or compute2. ---------------------------------------------------------------------- onlyati · 2023-12-29 14:23 You can setup a high availability solution for Postgres like data streaming or logical replication. So each database has its own data and they are sync. Of course this topic is not so simple, worth to read Postgres docs about them. Store data files on NFS which belongs to any RDBMS is rather problematic in my opinion. ---------------------------------------------------------------------- elh0mbre · 2023-12-29 15:26 K8s is storage agnostic, there are a couple of easy to setup highly available, distributed storage solutions available. Ive used longhorn for this. Currently I run postgres on the NAS device with local storage (I use a taint to make sure the NAS device is the only node that can run postgres. Downside is that if that node goes down postgres goes down). I'm replacing the NAS device soon and will be looking at running postgres off of HA storage and/or in a clustered mode. Same minio (for object storage). ---------------------------------------------------------------------- natermer · 2023-12-30 00:32 For my personal stuff I don't mind occasional downtime. If my file server is down for a evening and I haven't noticed it then it isn't obviously that big of a deal. I've done lots of different types of clustering in the past. But then it becomes a maintenance headache as now I not only have to deal with the applications/services I want to run, but also all the infrastructure necessary to make it high availability. Personally I am happy with just making sure backups are working and that stuff comes up correctly after a reboot. --------------------------- If you want to do high availability stuff then look into setting up a 3-node cluster with roughly the same hardware. Having a odd number of machines is helpful because many services rely on maintaining a quorum and quorums typically require a odd number of services to help deal with "split brain" issues. This is because typical services rely on a voting approach were members elect a master process. If you have a even number of systems (like 2 or 4) then you can end up with 2 different candidates with the equal number of votes. Then you have to deal with storage and load balancer. Storage can be dealt with easily enough with something like glusterfs. Ideally you will want the storage cluster to be separate from your compute cluster. This helps with issues regarding memory exhaustion and other sorts of conflicts. But it isn't a requirement. Load-balancer, which is a front-end for your services... is trickier. Most people run a reverse proxy, typically using nginx but there are lots of them, in front of their services. So clients connect to the reverse proxy and the proxy knows the locations of the services. This way clients don't need to be aware of which actual machine the backend service is running on. Obviously having a single reverse proxy isn't "high availability", but since it is so simple and small it doesn't really matter. It quickly can recover. For this I typically run a k3s (kubernetes distribution) inside of virtual machines. Then use metal-lb load balancer. This sets up a "floating IP" or "VIP" that services can use. It then depends on normal level 2/level 2 routing features to route the services to one of the machines in the cluster. One of the machines in the cluster will advertise that IP and clients will connect to it. If that machine goes down then that IP will just move to a different machine and clients will start routing to that one instead. Usually doesn't require any special configuration on the network side. So my vote, if you still want to do it, is to do BOTH proxmox AND kubernetes. But docker-swarm can work just as well. Not all applications work well or are easy to deploy on kubernetes/docker. Having that stuff run in a virtual machine gives you lots of flexibility. For example "Homeassistant" works best inside of a dedicated machine/virtual machine. They have their own distribution and while it can be made to work inside of containers you miss out on a lot of convenience.. ---------------------------------------------------------------------- blind_guardian23 · 2023-12-30 18:12 mostly likely too complicated if you did not already needed it. Proxmox is the easiest, the other nodes should be able to run your stuff, docker only inside VMs. Needs cluster filesystem for automatic start in other node or ZFS and Replication (but manual and you are behind 1min in Data which is the frequency of replication). wouldn't worry about k8s unless you go serious DevOps-route. ---------------------------------------------------------------------- [deleted] · 2023-12-30 21:35 [removed]