Docker Swarm multiple managers and NGINX question
reddit / selfhosted

----------------------------------------------------------------------
Big_Hovercraft_7494 · 2023-11-16 20:17

Hi all...

So I've moved to Docker Swarm to test it in comparison to K3s.  I'm
not a huge fan of the complexity of K3s, so I thought I'd give Docker
Swarm a shot.

The only issue I have so far is this....

I have 3 manager nodes and 3 worker nodes.  However, I have my DNS
server pointing the hosted services to the one manager node that hosts
NGINX for reverse proxy.  My problem is, when one of the manager nodes
goes down, the next one takes over, but all the services are still
pointing to the first manager node's IP for NGINX.

NGINX does move to the other manager, but without DNS pointing to the
right IP, it's useless.  I really don't want to have to go change the
IP in my DNS server anytime this happens and that defeats the purpose
of multiple managers to create HA.

So, what am I doing wrong here?

----------------------------------------------------------------------
murdaBot · 2023-11-16 22:43

You should know that Swarm's ability to self-recover is atrocious.
With 3 manager nodes, you can only lose 1. If you lose 2, the entire
swarm fails. If you want to be able to lose 2, you have to run 5
managers.

As for your behavior, that is normal. You should be able to point to
ANY manager node, whether it's up or down, and communication should
still work. This is also why it totally screws up nginx logs, you'll
always get the IP of the swarm as the client IP.

----------------------------------------------------------------------
nerdyviking88 · 2023-11-17 14:09

The solution around this is 2 fold:

Either use a front end load balancer that sends to the swarm nodes
OR
Configure the swarm nodes with like KeepaliveD to have a virtual IP
address, and then point your DNS to that.

I'm personally using the second one for the past 4 years very happily.

----------------------------------------------------------------------
Big_Hovercraft_7494 · 2023-11-17 14:36

I'm aware of the need for many managers, but I'm not to worried about
that since this is a homelab.  One level of redundancy is enough, and
I have a number of vms and pi's available, so 3 or 5 managers isn't a
problem for me.

Thank you for you input though.

----------------------------------------------------------------------
biswb · 2023-11-17 16:13

This. This is exactly what I do. And it for both my lab which is my
side hustle, and my big boy job. Its perfect!