[HN Gopher] High Available Mosquitto MQTT on Kubernetes
___________________________________________________________________
High Available Mosquitto MQTT on Kubernetes
Author : jandeboevrie
Score : 45 points
Date : 2025-05-14 20:42 UTC (4 days ago)
(HTM) web link (raymii.org)
(TXT) w3m dump (raymii.org)
| oulipo wrote:
| Wouldn't more modern implementations like EMQx be better suited
| for HA ?
| seized wrote:
| VerneMQ also has built in clustering and message replication
| which would make this easy.
| oulipo wrote:
| Have you tried both EMQx and VerneMQ and would you
| specifically recommend one over the other? I don't have
| experience with VerneMQ
| bo0tzz wrote:
| EMQX just locked HA/clustering behind a paywall:
| https://www.emqx.com/en/blog/adopting-business-source-licens...
| zrail wrote:
| Sigh that's annoying.
|
| Edit: it's not a paywall. It's the standard BSL with a 4 year
| Apache revert. I personally have zero issue with this.
| casper14 wrote:
| Oh can you comment on what this means? I'm not too familiar
| with it. Thanks!
| zrail wrote:
| BSL is a source-available license that by default forbids
| production use. After a certain period after the date of
| any particular release, not to exceed four years, that
| release automatically converts to an open source license,
| typically the Apache license.
|
| Projects can add additional license grants to the base
| BSL. EMQX, for example, adds a grant for commercial
| production use of single-node installations, as well as
| production use for non-commercial applications.
| bo0tzz wrote:
| It is a paywall, clustering won't work unless you have a
| license key.
| zrail wrote:
| Yeah I see that now. Ugh.
| jandeboevrie wrote:
| Would they work as performant and use the same amount of (less,
| almost nothing) resources? I've ran mosquito clusters with tens
| of thousands of connected clients, thousands of messages per
| second, on 2 cores and 2GB of ram, while mostly idling.
| (Without retention, using clean sessions and only QoS 0)...
| jpgvm wrote:
| I built a high scale MQTT ingestion system by utilising the
| MQTT protocol handler for Apache Pulsar
| (https://github.com/streamnative/mop). I ran a forked version
| and contributed back some of non-proprietary bits.
|
| A lot more work than Mosquitto but obviously HA/distributed and
| some tradeoffs w.r.t features. Worth it if you want to run
| Pulsar anyway for other reasons.
| oulipo wrote:
| I was going to go for Redpanda, what would be the pro/cons of
| Pulsar you think?
| andrewfromx wrote:
| when dealing with long lasting TCP connections, why add that
| extra layer of network complexity with k8s? I work for a big IoT
| company and we have 1.8M connections spread across 15 ec2
| c8g.xlarge boxes. Not even using a NLB just round-robin DNS.
| Wrote our own broker with https://github.com/lesismal/nbio and
| use a packer .hcl file to make the AMI that each ec2 box boots.
| Using https://github.com/lesismal/llib/tree/master/std/crypto/tls
| to make nbio work with TLS.
| stackskipton wrote:
| Ops type here who deals with this around Kafka.
|
| It comes down to how much you use Kubernetes. At my company,
| just about everything is in Kubernetes except for databases
| which are hosted by Azure. So having random VMs means we need
| to get Ansible, SSH Keys and SOC2 compliance annoyance. So the
| workload effort to get VMs running may be higher than
| Kubernetes even if you have to put in extra hacks.
| NewJazz wrote:
| You don't need ansible if it is all packed into the Ami.
| avianlyric wrote:
| K8s itself doesn't introduce any real additional network
| complexity, at least not vanilla k8s.
|
| At the end of the day, K8s only takes care of scheduling
| containers, and provides a super basic networking proxy layer
| for convenience. But there's absolutely nothing in k8s that
| requires you use that proxy layer, or any other network
| overlay.
|
| You can easily setup pods that directly expose their ports on
| the node they're running on, and have k8s services just provide
| the IPs of nodes running associated pods as a list. Then rely
| on either on clients to handle multiple addresses themselves
| (by picking an address at random, and failing over to another
| random address if needed), configure k8s DNS to provide DNS
| round robin, or put an NLB or something in front of it all.
|
| Everyone uses network overlays with k8s because it makes it
| easy for services in k8s to talk to other services in k8s. But
| there's no requirement to force all your external inbound
| traffic through that layer. You can just use k8s to handle
| nodes, and collect needed meta-data for upstream clients to
| connect directly to services running on nodes with nothing but
| the container layer between the client and the running service.
| andrewfromx wrote:
| | Aspect | Direct EC2 (No K8s) | Kubernetes (K8s Pods) |
|
| |-------------------------|----------------------------------
| ---------------------|---------------------------------------
| ----------------------------------------------|
|
| | Networking Layers | Direct connection to EC2 instance
| (optional load balancer). | Service VIP - kube-proxy - CNI -
| pod (plus optional external load balancer). |
|
| | Load Balancing | Optional, handled by ELB/ALB or
| application. | Built-in via kube-proxy (iptables/IPVS) and
| Service. |
|
| | IP Addressing | Static or dynamic EC2 instance IP. | Pod
| IPs are dynamic, abstracted by Service VIP. |
|
| | Connection Persistence | Depends on application and OS TCP
| stack. | Depends on session affinity, graceful termination,
| and application reconnection logic. |
|
| | Overhead | Minimal (direct TCP). | Additional latency from
| kube-proxy, CNI, and load balancer. |
|
| | Resilience | Connection drops if instance fails. |
| Connection may drop if pod is rescheduled, but Kubernetes can
| reroute to new pods. |
|
| | Configuration Complexity| Simple (OS-level TCP tuning). |
| Complex (session affinity, PDBs, graceful termination, CNI
| tuning). |
| zrail wrote:
| To preface, I'm not a Kubernetes or Mosquitto expert by any
| means.
|
| I'm confused about one point. A k8s Service sends traffic to pods
| matching the selector that are in "Ready" state, so wouldn't you
| accomplish HA without the pseudocontroller by just putting both
| pods in the Service? The Mosquitto bridge mechanism is bi-
| directional so you're already getting data re-sync no matter
| where a client writes.
|
| edit: I'm also curious if you could use a headless service and
| use an init container on the secondary to set up the bridge to
| the primary by selecting the IP that isn't it's own.
| jandeboevrie wrote:
| > so wouldn't you accomplish HA without the pseudocontroller by
| just putting both pods in the Service?
|
| I'm not sure how fast that would be, the extra controller
| container is needed for the almost instant failover.
|
| Answering your second question, why not an init container in
| the secondary, because now we can scale that failover
| controller up over multiple nodes, if the node where the
| (fairly stateless) controller runs goes down, we'd still have
| to wait until k8s schedules another pod instead of almost
| instantly.
| rad_gruchalski wrote:
| > without the pseudocontroller
|
| I am making an assumption. I assume that you mean the
| deployment. The deployment is responsible for individual pods.
| If a pod goes away, the deployment brings a new pod in. The
| deployment controls individual pods.
|
| To answer your question: yes, you can simply create pods
| without the deployment. But then you are fully responsible for
| their lifecycle and failures. The deployment makes your life
| easier.
| zrail wrote:
| I was referring to the pod running the kubectl loop. As far
| as I can tell (I could be wrong! I haven't experimented yet)
| the script is relying on the primary Mosquitto pod's ready
| state, which is also what a Service relies on by default.
___________________________________________________________________
(page generated 2025-05-18 23:01 UTC)