Docker Swarm - manager nodes - baremetal or virtual? reddit / selfhosted ---------------------------------------------------------------------- gwolfe28 · 2023-12-25 18:16 Long time reader, first time poster, still very much an amateur! I'd like to set up Docker Swarm mostly for HA purposes for some containers (e.g NGINX Proxy Manager; Gitea; phpIpam) and I understand that it is best to have 3 or 5 managers (with keepalived on the manager nodes to have a VIP to point to). I have 3x Wyse 5070 machines (4 core/4 thread) that I'm planning on using. I am trying to decide whether I use these as 3 baremetal nodes (i.e. 3 manager nodes - which will also perform worker tasks) OR to install Proxmox and have 2 virtual nodes per machine (i.e. 5 manager nodes + 1 worker node). Can anyone speak to the benefit of having more manager nodes vs. the administration & potential performance cost? ---------------------------------------------------------------------- adamshand · 2023-12-25 19:27 I'm only just in the beginning stages of playing with Swarm, others may have better answers than me. As I understand it, the basic trade off with manager nodes is … the more managers you have, the more redundancy you have, but the more overhead there is, which increases latency. For small setups, on a fast local network, I think you could have all managers and it probably wouldn't make a noticable difference, but I haven't tried this yet. Two side notes. I've most of the way through converting all my services over to CapRover. It's basically a simple frontend for Swarm but it does all the reverse proxying for you, has a nice collection of one-click-apps, and also works as a Heroku style PaaS so I can deploy my own code without having to manually build containers. The shitty part of Swarm is there's no built-in way to manage persistent storage for containers in a way that allows failover. If you have a NAS you can use that for homelab nodes via NFS, but that isn't very reliable for SQLite files. I'm looking at using the Rclone plugin for Docker to store persisent volumes on S3 ... but it's a bit of a pfaff. ---------------------------------------------------------------------- mqmq0 · 2023-12-25 20:43 Im running 3 nodes (manager and worker) on 3 Vm on 3 proxmox nodes, all with keepalived. 5th gen i5 NUC. Perk of Proxmox that you can really easily run CEPH (and Ceph FS) so you can have a trully hyper- converged infrastructure. All nodes are equal, any two of them can keep the cluster alive in case of a failure. I recommend Portainer as a management GUI. With GIT auto updates of Stacks, you can have an almost kubernetes-like experience with fraction of the complexity and overhead processing. ---------------------------------------------------------------------- NikStalwart · 2023-12-25 23:18 I see zero advantage in running 2x virtual nodes on the same hardware. The point of multiple nodes is to have redundancy. What redundancy are you getting if both nodes are on the same machine? Lightning strikes your server, Martians invade and steal it, Nancy Pelosi's husband gives the wrong address for his next gay hooker and said hooker smashes your server with a hammer.... No redundancy that way. You can run a hypervisor if you want other services on the same machine besides the docker manager node, but there is no reason to run two manager nodes on the same physical machine. ---------------------------------------------------------------------- gwolfe28 · 2023-12-26 00:33 Interesting - thanks for sharing - so do you run the Proxmox nodes as a cluster as well to be able to leverage ceph? ---------------------------------------------------------------------- gwolfe28 · 2023-12-26 00:49 Thanks - appreciate it - I guess the only redundancy would be if something went awry at the VM OS level - but that is starting to sound silly esp if I am able to deploy with something like ansible - there should be NO reason for one VM to fail and the other not. ---------------------------------------------------------------------- gwolfe28 · 2023-12-26 00:51 Thanks I'll check out CapRover. Re persistent storage I was planning on trying out GlusterFS vs using an NFS share from my NAS. The DB replication issue is something I'm still trying to figure out. Sounds like I'm a few months behind you in your journey!! ---------------------------------------------------------------------- adamshand · 2023-12-26 02:03 GlusterFS or Ceph seem like the best 'serious' options, just don't work very well for my setup. If you haven't found it already, there's some really good info at: https://geek-cookbook.funkypenguin.co.nz/docker-swarm/design/ ---------------------------------------------------------------------- Reverent · 2023-12-26 03:06 Running docker (not swarm) inside a highly available VM will suit your needs, you don't have to double up with container redundancy on top of VM redundancy. ---------------------------------------------------------------------- kayson · 2023-12-26 05:09 I would install proxmox for virtualization, but only have one VM per bare metal machine. You don't really need more than 3 swarm nodes except for load balancing, and I'd be surprised if you have high enough load for that. There are a lot of advantages to virtualization, though, including being able to run other virtual hosts on the same physical one. ---------------------------------------------------------------------- mqmq0 · 2023-12-26 07:14 Yes, 3 node Proxmox cluster. Mainly for CEPH. The only solution for persistent storage for the Swarm that really works. I went through all the usual suspects: NFS, SMB based on my NAS but they all have some problem. Mainly being a single point of failure for the whole setup. So I ended up with Gluster or Ceph. Gluster is easy but somehow does not work reliably. Ceph running on Proxmox is just magic. Literaly like 5 clicks and you are done. The clients are bit tricky but doable. Other benefit of Proxmox even if you run only one VM per node is the Proxmox Backup Server. Both for VMs and the Proxmox Backup Client can be installed anywhere to backup anything you need and restore easily. ---------------------------------------------------------------------- CriticismTop · 2023-12-26 11:46 Why use Swarm? Kubernetes is the standard in that space and has been for years. While it can be difficult to manage, for the sort of set up you're talking about, just use K3s and a single virtual master. Then set up your other boxes at k3s workers. ---------------------------------------------------------------------- NikStalwart · 2023-12-27 05:47 Exactly