[HN Gopher] MQTT turns 25
___________________________________________________________________
MQTT turns 25
Author : andypiper
Score : 251 points
Date : 2024-10-22 10:01 UTC (12 hours ago)
(HTM) web link (andypiper.co.uk)
(TXT) w3m dump (andypiper.co.uk)
| Uptrenda wrote:
| The cool thing about MQTT is there's so many servers for it that
| are completely public -- and intentionally so. You can use it for
| real time messages but I think it also has quality modes that
| allow for some persistence.
|
| Some of these servers have stat pages that show how much data
| they transfer. Usually phenomenal amounts of data. It's pleasing
| to me as someone doing a lot of stuff with open source networking
| to have such an establish protocol (instead of say -- 10 crappy
| ones.) MQTT works and it works well.
| roland35 wrote:
| Have any examples of public servers?
| Scarjit wrote:
| HiveMQ has a public one: https://broker.hivemq.com/
| dikei wrote:
| I only know this one
|
| https://www.mqtt-dashboard.com/
|
| But a quick Google search will get you some others.
| Risse wrote:
| Helsinki Regional Transport (HSL) has a public server with
| realtime position of all of their vehicles (buses, trains).
|
| https://digitransit.fi/en/developers/apis/4-realtime-
| api/veh...
| bouvin wrote:
| I have been using MQTT for years in my IoT courses - it has
| proven to be a very versatile tool, and it is convenient that it
| is also supported over web sockets.
| kitd wrote:
| About 15 years ago, Andy Stanford Clark's house became something
| of a news item, in the days before tweeting IoT devices were a
| thing:
|
| https://www.bbc.co.uk/blogs/technology/2009/06/things_that_t...
|
| The core protocol, devised in the days when 1 byte over a
| satellite link cost $1, is incredibly efficient and simple to
| implement.
| baruchthescribe wrote:
| Reposting my comment from five years ago:
|
| I once had a client where the only port available for me to use
| on their firewalls was for MQTT (1883) because that's how we were
| getting sensor data from them. They would not open anything else
| for us no matter how we implored them so I wrote a live TCP
| wrapper over MQTT to get around it. It was a local multithreaded
| TCP daemon that listened for outbound requests on a certain port,
| wrapped them in MQTT and then published them using a unique
| topic. The server daemon would detect these topics and unwrap
| them before forwarding to our server processes. So the client
| machine thought it was making a live TCP connection to our server
| but in the middle was a funky invisible MQTT wrapper. It was
| really elegant once it worked but my goodness was it a pain to
| debug - a couple of months before I got the whole thing right
| because of all the blind alleys I went down.
| ericvsmith wrote:
| I've worked at places where subverting firewall rules like this
| would get you fired. I now refer to such situations where I'm
| forced to twiddle my thumbs while no work gets done as "letting
| the process work".
| exe34 wrote:
| I always kick this kind of decision upwards: we can do it
| this way and break this rule, so I need director level
| approval, or we can wait until the customer blinks.
| n_plus_1_acc wrote:
| Yes. Ask your boss for approval and tell them otherwise
| nothing will get done.
| ocschwar wrote:
| So you basically implemented an MQTT-Sockets protocol. Which
| you could then use to connect to a WebSockets server on the
| other side.
| joezydeco wrote:
| I had a lot of fun on an embedded systems project recently where
| I used MQTT as an interprocess messaging system. Broker and
| clients on the same machine.
|
| If I needed to sniff or debug something it was as easy as putting
| the device on the network and using MQTT Explorer to record or
| inject messages. I could even let coworkers that were working
| remotely work with the system by opening the port outside the
| LAN.
| klysm wrote:
| Haven't seen this done much but it does seem to have some
| desirable properties. My biggest concern when using it as a
| system component has been durability guarantees. I don't have a
| lot of trust on broker implementations not to lose data.
| logifail wrote:
| > My biggest concern when using it as a system component has
| been durability guarantees. I don't have a lot of trust on
| broker implementations not to lose data
|
| (If we ignore retained messages) MQTT brokers are not
| designed to store data, so there's no need to be concerned
| about durability because there simply isn't any...(!)
|
| It is (just) a messaging protocol.
| joezydeco wrote:
| MQTT, by design, isn't going to have the confirmation
| mechanisms and safeguards that you could get from other
| methods or even sockets. But once you release yourself from
| that paranoia, it's pretty liberating.
|
| The nature of my project was that a missed message wouldn't
| be fatal, and message retention was good enough protection
| for a process that crashed and restarted.
| roland35 wrote:
| Would zeromq be an alternative here?
| joezydeco wrote:
| It's an alternative, but there are some aspects to having the
| broker that I liked. If there was a bug I could halt the
| system, attach/subscribe an external dev machine, and grab
| all retained messages from the broker. I don't believe ZMQ
| allows that.
| thebruce87m wrote:
| We use ZeroMQ for this purpose. No broker needed.
| omani wrote:
| this statement is a bit confusing. because with zeromq, one
| side plays the broker (I dont mean the broker pattern). I
| mean it in the sense that someone, somewhere, needs to listen
| on a port. which in the end is the same as having a central
| mqtt broker sitting somewhere listening on a port. hence the
| statement "no broker needed" is a bit confusing.
| thebruce87m wrote:
| > one side _plays_ the broker
|
| Hopefully most people understand that I meant that you
| don't need a stand-alone broker, which on embedded systems
| means one more thing to deploy and worry about the license
| for.
|
| Edit: Also:
|
| > The philosophy of ZeroMQ starts with the zero. The zero
| is for zero broker (ZeroMQ is brokerless)
|
| https://zeromq.org/get-started/
| exe34 wrote:
| I think you can also connect in a random order, so bind
| doesn't need to happen before connect.
| miljen wrote:
| Fun fact: at the same time, the most famous C++ library, Boost,
| is reviewing the `async-mqtt5` implementation
| (https://github.com/mireo/async-mqtt5) to be included in Boost as
| Boost.MQTT:
| https://lists.boost.org/Archives/boost/2024/10/index.php
| formerly_proven wrote:
| Is Boost still something people reach for in newer projects?
| Anecdotally most adoption I've seen happened in the 00s and
| very early 2010s (pre everyone mandating C++0x/C++11), I only
| rarely see it around these days.
|
| edit: boost.org is a blast from the past, still looks exactly
| like I remember it from like 2008. Down to the "Get Boost"
| shopped on an emergency off button!
| miljen wrote:
| That is an interesting question. Here you can see the actual
| popularity of each library within Boost:
| https://grafikrobot.github.io/boost_lib_stats/. Also, see the
| comments at https://www.reddit.com/r/cpp/comments/130bzj8/has
| _boost_lost.... My company has been heavily utilizing
| Boost.Asio and Boost.Beast, so we've definitely reached for
| Boost in newer projects as well.
| spacechild1 wrote:
| AFAICT Boost.Asio is still the go-to networking library.
| (Though Asio is also available as a standalone outside of
| Boost, I didn't know that until recently.)
|
| Apart from that, Boost has quite a few goodies such as
| Boost.Json, Boost.Program_options, Boost.Interprocess,
| Boost.Lockfree, Boost.Unordered, Boost.Dynamic_Bitset, etc.
|
| Some libraries, like Boost.Atomic, Boost.Thread, Boost.Chrono
| and Boost.Filesystem, simply became obsolete with modern C++
| versions. In fact, they served as the blueprint for the
| corresponding C++ standard libraries.
|
| Personally, I have been wary of using Boost in the past
| because it's such huge library, but CMake integration is
| actually quite good these days and I found it pretty easy to
| use. Documentation is also quite good IMO.
| happosai wrote:
| Boost appears to have become some kind of staging area for
| stdlib. Stuff that becomes popular in boost get eventually
| included in stdlib.
| this_user wrote:
| Having recently used MQTT for a project, I can't say that I'm a
| huge fan. There are a lot of options in the protocol where it's
| not immediately clear what they do and why they are important, or
| in what combination they need to be used to make sure things work
| as intended, and the documentation is often not great at
| explaining. Part of this may also be on the Eclipse Mosquitto
| Python client that I had been using. I took me several days to
| figure out that the client was running into a race condition on a
| slower system that resulted in subscriptions to topics silently
| being ignored which caused the associated callbacks to
| malfunction despite following their documentation 100%. Overall,
| this has been one the messiest experienced I have ever had with a
| protocol, certainly one that isn't that old.
| jorl17 wrote:
| My experience using MQTT has been through the paho python MQTT
| library, and, while we definitely have managed to get a lot
| done, it has been a terrible experience.
|
| Everything about it rubs me the wrong way. The API design, the
| poor documentation, even the way they seem to not adhere to
| typical python conventions.
|
| It looks like it's easy to start using it, but then the breadth
| of protocols and implementations start getting to you. At a
| point in time, I remember the only reliable way we had of
| knowing if we had successfully connected to a server was to try
| to subscribe to a topic twice and catch a specific error code
| on the on_connect message (which actually was documented as a
| success code at the time). I know how crazy it sounds and maybe
| there was a better way to do it, but, if there was, I don't
| think it was that easy to find.
|
| It's easy to complain, though. I'm very grateful so many people
| have worked to create this library. If they hadn't, I wouldn't
| have built what we've built with it. I really admire people who
| take on these huge projects.
| DannyBee wrote:
| So your client just sucks here.
|
| My experience with the eclipse clients (paho, etc) is similar
| to yours in both python and C++, - it makes it overly
| complicated and seems very low level. It also is somewhat buggy
| because of the architecture.
|
| I believe (perhaps wrongly), it's all basically maintained by
| one person or close to it (only one person has contributed to
| the C++ client in the past 6 months, for example, and nobody
| has contributed in the past 3), so i kind of understand how
| it's gotten this way over time.
|
| I filed a simple PR (1 word change) to fix an obvious bug and
| it took 2 years to review it and accept it with no changes.
| Again not a complaint, just trying to portray how the state
| feels - a lot of libraries, bugs, and work, and few overworked
| people helping get it all done.
|
| I moved to other clients and the experience is much much better
| in python, rust, C#, and C++.
|
| Most of them have a good combination of high and low level
| API's, so if you just want to send a message on a topic, you
| don't have to worry about acks, retries, etc.
|
| But if you need control, you can get it.
|
| Honestly, i worry at this point that keeping paho/etc alive in
| this state is doing more harm than good - if they were
| officially dead it would at least force the issue. Right now
| you end up with users who have an experience like yours, and
| then either give up on MQTT or assume they are doing it wrong
| :)
| vv_ wrote:
| After reading your post, I went looking for good C/C++
| implementations suitable for embedded development, and
| honestly, I couldn't find many! MQTT is typically
| straightforward to implement, so most companies develop their
| own version to better adapt to the specific systems they use
| for data transmission. That said, maybe this is an
| opportunity to create a well-designed MQTT client
| implementation tailored specifically for embedded devices! :)
| accelbred wrote:
| I work for the team that maintains coreMQTT
| (https://github.com/FreeRTOS/coreMQTT)
|
| Its a C89 MQTT library intended for embedded devices.
| tcshit wrote:
| There is a very nice MQTT client C++ implementation that
| works great on embedded devices too. As it so happens, it
| is right now under formal review for inclusion into Boost.
| It is called async-mqtt5 and can be found at:
| https://github.com/mireo/async-mqtt5/
| ta988 wrote:
| Could you list the clients you are using? Because like many I
| fell into paho and things are not great.
| HankB99 wrote:
| I've used the Paho APIs in Python, C and C++ and moved on to
| not using any of these APIs. Where possible I use mosquitto_sub
| and mosquitto_pub to support the protocol and then just
| read/write standard input/output. It's not a matter of bugs
| (though I have encountered at least one) but rather it's just
| easier to use a program that's been written and tested to
| manage the connection to the broker.
|
| The only place where I was unable to do this effectively was
| with the last will and testament message (and I might not have
| tried hard enough.) It also doesn't work with microcontrollers
| that don't run something like Linux.
| clumsysmurf wrote:
| The eclipse paho MQTT Java client is also terrible, lots of
| open bugs, terrible API, and no release in over 4 years. For my
| case, I was able to use NATS instead.
|
| https://github.com/eclipse/paho.mqtt.java/issues
| qudat wrote:
| While not directly competing with mqtt, https://pipe.pico.sh is
| a great pubsub tool that leverages ssh for communication,
| effectively creating an authenticated, networked *nix pipe
| system.
|
| It tries to be the simplest way to send and receive events.
| ClumsyPilot wrote:
| > There are a lot of options in the protocol where it's not
| immediately clear what they do and why they are important
|
| I totally disagree, your expectations for the protocol seems
| unusual. They are not frameworks, and need to game many
| options.
|
| MQTT specification is like 3 pages.
|
| For comparison try to read specification of HTTP3, or AMQP -
| those are hideously complicated.
| dloranc wrote:
| What do you think about LwM2M in comparison with MQTT?
| vv_ wrote:
| It is not directly comparable because LwM2M was originally
| developed for device management, while MQTT is primarily used
| for transferring telemetry or other opaque data.
|
| My personal gripe with LwM2M is that it uses CoAP (Constrained
| Application Protocol) as its primary transport protocol. Since
| CoAP is based on UDP, you're forced to rely on DTLS for
| authentication and encryption. Managing sessions with DTLS can
| be quite frustrating, and unlike TCP, there's no reliable
| indication of when a session is closed by your peer.
|
| Another issue is that it doesn't strictly follow the client-
| initiated communication pattern and can receive unsolicited
| messages from the server. I have found this to be very
| unreliable in real-world conditions for Device Management use
| cases.
| vv_ wrote:
| There aren't many telemetry transfer protocols suitable for IoT
| besides MQTT (especially for resource constrained systems). It is
| incredibly simple to implement, has numerous libraries available
| in various programming languages, is supported by major cloud
| providers and is designed to operate over TCP (meaning you avoid
| DTLS).
|
| We have shipped many products that use MQTT for telemetry
| transfer to various different vendor servers and it always just
| worked. It was never a pain-point in stark contrast to CoAP or
| some proprietary protocols.
| 8fingerlouie wrote:
| It's such a great little protocol, and besides being "small
| enough" for hobby projects, it also scales enough to be used for
| something like Facebook Messenger [1]
|
| [1]: https://engineering.fb.com/2011/08/12/android/building-
| faceb...
| ErneX wrote:
| Most of my home automations depend on it, it's been great
| honestly. Cheers to mqtt!
| ronameels wrote:
| MQTT is being used a lot more in recent years inside of factories
| for sharing data between machines. Historically it's been used in
| Oil & Gas for SCADA (getting data from remote well sites).
|
| 10+ years ago we added it to Kepware (OPC server) and streamed
| tag values to "the cloud". I was at a conference giving a
| presentation on it when Arlen Nipper, one of the creators of
| MQTT, came up after the presentation and said I did a "decent
| job". It was humbling :). Fast forward to today, and we have a
| new company (HighByte) modeling factory data at the edge and
| sending it via MQTT, SparkplugB (protocol over MQTT), direct to
| S3, Azure Blob, etc, etc.
|
| All that to say, MQTT is a big driver in Industry 4.0, and it's
| cool to see it so heavily used all these years later.
| fartfeatures wrote:
| Is MQTT being used in places where previously Modbus would have
| been used?
| ronameels wrote:
| In some cases yes, controllers are adding in MQTT support
| directly or SparkplugB.
|
| It works in simple cases, but the lack of transactions &
| ability to control what and how often you're reading from the
| client side, can create challenges.
| InDubioProRubio wrote:
| There is also MQTT integrating bus-hardware- you basically
| plug in as bus-hardware replacement which allows to send data
| directly from the bus to the broker. The PLC never know what
| hit it.
|
| https://www.ifm.com/de/de/product/AL1332
|
| Pretty wild times.
|
| PS: You can web GET/POST set and get ios with these. Robot
| experience that tragic irony POST {NOOO} and he just dropped
| it like its hot.
| eska wrote:
| I also work in that industry and am quite puzzled by the
| technological inferiority, scope creep and NIH syndrome of the
| OPC foundation's standards. I wish they just used sparkplug b
| and implemented their specs on top of that for semantics.
|
| Then again the async stuff they're doing now is so over-
| engineered and terrible as well.. I took part in their teams
| meetings some time, only to find they had never read the mqtt
| spec (less than 50 easy to read pages) and didn't understand
| what the headers are for for example (they wanted to put stuff
| in there that actually belongs in the payload). Some microsoft
| guy also took offense at my suggestion to first look at what
| competitors are doing with mqtt, because he wanted to create
| something new rather than copy. My company will just treat opc
| ua at the very edge and isolate it from our tech as much as
| possible on my suggestion.
| jpfr wrote:
| Umm. There is a deep integration between OPC UA and MQTT
| since ca. 2017.
|
| For binary and JSON payloads, and with encryption and group
| key management features.
|
| I see quite a bit of unsubstantiated bashing lately.
| Especially advocating 20% solutions that solve the easy parts
| with less effort.
|
| What are you missing from OPC UA?
| doctoboggan wrote:
| We are currently using the Kepware IoT plugin to stream around
| 800k tags per second via MQTT ultimately into a Victoria
| metrics DB. It's a little clunky and requires more processing
| steps than I would like. I am currently moving away from this
| solution (since Kepware charges a reoccurring license fee every
| year for the IoT plugin), and toward using telegraf to read the
| OPC-UA data directly from kepware without the IoT plugin.
|
| Do/Did you work at Kepware?
| red-iron-pine wrote:
| > MQTT is being used a lot more in recent years inside of
| factories for sharing data between machines. Historically it's
| been used in Oil & Gas for SCADA (getting data from remote well
| sites).
|
| This is where I first encountered it too -- chemical
| production. Also saw a lot of it in aviation and railroad
| control systems.
|
| That said, seeing a lot more Kafka and RabbitMQ crowding into
| MQTT's marketspace.
| NortySpock wrote:
| My most recent foray into MQTT was using the MQTT 3.1.1
| compatibility mode in NATS, which was pretty easy to set up once
| I got the permissions sorted out.
|
| Works great for my smart home needs while also letting me explore
| latency and throughput load testing if I want to have a little
| fun.
| artooro wrote:
| I'm using NATS as well, natively when possible but bridged to
| MQTT for devices that need it.
| gorbypark wrote:
| I think my first project that's still "in production" and used
| daily was taking an existing SVG map of a water distribution
| system (snowmaking and fire suppression
| pipelines/pumps/valves/etc at a major ski resort). I made a MQTT
| topic for each pump, valve and section of pipe along with some
| state for each (water direction, pump or valve on/off, pressures,
| etc). It's a website, so I used mqtt.js and jQuery to update the
| colours and fills of the SVG to represent the various states.
| It's statically hosted and there's a mqtt broker that's been
| running in a container, untouched for nearly 10 years now.
| Mqtt.js is running over a web socket, so if the state is updated,
| it just magically updates for everyone else.
| remram wrote:
| Holy shit Docker is 11 years old.
| brian-gilmore wrote:
| aaah I love me some AJAM ;)
| InDubioProRubio wrote:
| A heap of dead "replacements" at the feets - what more can a
| communication standard want.
| sunshine-o wrote:
| I never understood why we don't use MQTT for simple systems
| monitoring, with a push model.
|
| I mean it was almost designed for this use case.
|
| Also if using NATS things become very simple to setup.
| Evidlo wrote:
| I remember one of the annoying things I found about actually
| using MQTT was that many clients didn't implement any sort of
| connect/disconnect callback and expected you to handle
| reconnection logic (e.g. exponential backoff) yourself. Also it
| didn't support RPC style requests for some reason.
|
| Also since MQTT is TCP-based you can't natively speak the
| protocol from a webbrowser without the broker also listening on a
| websocket.
|
| Ended up switching to WAMP-proto [0], which checked all my boxes
| and just did everything over websockets by default.
|
| 0: https://wamp-proto.org/
| pjbk wrote:
| MQTTv5 now has RPC. You can specify a response topic in the
| request message, and also add 'correlation' data to track the
| context of the call.
| kerblang wrote:
| I don't buy the b.s. around MQTT being so
| lightweight/efficient/etc. It's just using TCP/IP. Maybe by the
| standards of the day that was comparatively special, but I've
| never seen any real evidence for the claim besides this constant
| bragging.
|
| I guess it's nice that MQTT is a standard and so you have a way
| to connect to off-the-shelf devices that are enabled. I still
| think there are better options for pub/sub and message queues,
| esp. if you need to failover on the consumer side.
| bdamm wrote:
| What better options? The neat thing about MQTT, and that almost
| all other pub/sub implementations that I've seen get wrong, is
| that the core data structure in MQTT is the subscribing client,
| not the queues & topics. That means you can map as large an
| address space as you want into the topic tree. The topic tree
| can be trillions and trillions of endpoints, even an embedded
| server could have a topic tree with one endpoint for every ipv6
| address, if you wanted. So the topic tree is rich, your
| subscriptions can be as selective as you need, and the server
| can be fast on a low footprint.
| mmcnl wrote:
| Exactly. Even shorter summary: MQTT broker is stateless thus
| easy to manage and hard to break. This makes MQTT reliable.
___________________________________________________________________
(page generated 2024-10-22 23:01 UTC)