Nomad - howto prevent Node service port from direct access
reddit / selfhosted

----------------------------------------------------------------------
PressureKiller · 2023-10-26 15:31

Hi everybody,

After having tried several deployment strategies with Nomad docker
containers with traefik on the front, I'm encountering the unresolved
problem of ports on nodes dynamically opened by services that are
directly accessible from internet and with no possibility of closing
them.

So I tried

* mode bridge,
* consul mesh connect
* user network docker
* even if I don't declare a port, one is assign. each time a host port
is opened on the client (node) and be seen inside consul dashboard.

for example, one Use case, hosting a backend and a database behind a
gateway, I don't want these services to be accessible directly from
the node ip but for everything to go through the gateway.

So what do you recommend please? Should I protect each client with a
vpn like tailscale / WireGuard and a firewall that only authorizes
internal requests?

Am I doing something wrong?

Thank you for your advice,

----------------------------------------------------------------------
geo38 · 2023-10-26 17:25

> Am I doing something wrong?

Yes.

----------------------------------------------------------------------
racomaizer · 2023-10-27 03:32

What I do is setup a Wireguard mesh and one loopback interface on the
nodes, setup dynamic routing so nodes can reach others’ loopback via
Wireguard mesh, then config the services to use host networking and
listen only via the loopback. Actually consul and nomad also run
through this private network, I feel that’s straightforward and pretty
easy to maintain.

Of course if your servers are on one provider you can try using their
private network as well, but if you have multiple providers I believe
my solution would be a better choice.

----------------------------------------------------------------------
PressureKiller · 2023-10-27 14:44

Thank you u/racomaizer !

As I understand, I need to play more with network\_interface and cni
networks.

Did you perhaps have some configuration to share please ?

Best regards,

----------------------------------------------------------------------
racomaizer · 2023-10-28 15:05

Which, Consul service mesh or my approach?

For my approach, you set up a flat LAN (like AllowedIPs=10.1.23.0/24)
Wireguard mesh, then you add this into nomad.hcl:
```
client {
  host_network "public" {
    interface = "eth0"
  }
  host_network "private" {
    interface = "wg0"
  }
}
```
Add `host_network=""` to job.group.network.port and use host
networking for every container and done.

I don't know how traefik and consul work together but I do this with
caddy:
```
{{- range services }}
    {{- range service .Name }}
        {{- if .Tags | contains "public-facing" }}

https://{{ index .ServiceMeta "domain" }} {
    import common_headers
    handle {
        reverse_proxy {
            dynamic srv {
                service {{ .Name }}
                proto tcp
                name service.consul
                refresh 30s
            }
        }
    }
    log
}
        {{- end }}
    {{- end }}
{{- end }}
```

For Consul service mesh, I think reading [this](https://www.datocms-
assets.com/2885/1597077859-consul-life-of-a-packet-service-
mesh-v11-digital.pdf) till page 35 will get you started. Basically
after the initial work you slap `service.connect.sidecar_service` to
every container then pull them into traefik's container with
`service.connect.sidecar_service.proxy.upstreams`. Now all your
containers listen on localhost and connect to localhost. This way you
don't even need VPN, everything is protected by the sidecar proxy.

----------------------------------------------------------------------
PressureKiller · 2023-10-29 06:06

Thank you!