[HN Gopher] BlazingMQ: High-performance open source message queu...
___________________________________________________________________
BlazingMQ: High-performance open source message queuing system
Author : carride
Score : 394 points
Date : 2023-07-27 16:27 UTC (6 hours ago)
(HTM) web link (bloomberg.github.io)
(TXT) w3m dump (bloomberg.github.io)
| eska wrote:
| I checked the docs, but couldn't find anything about mqtt compat?
| fadedsignal wrote:
| A yet another MQ :-) I quickly checked the performance page and
| it looks much better than its competitors. Documentation is
| better as well.
| 29athrowaway wrote:
| 420 messages per second.
| rurtrack wrote:
| Sweet
| 29athrowaway wrote:
| Dude
| i-use-nixos-btw wrote:
| Off topic here but
|
| > Carefully architected and written in C++ from the ground up
| with no dependency on any external framework, BlazingMQ
| provides...
|
| What a weird selling point.
|
| There are pros and cons to having no dependencies. Not long ago,
| it was a common decision for C++ projects because dependency
| management was a mess. But that hasn't been the case for a long
| time (arguably we have the opposite problem - there are so many
| ways of doing it: Conan, vcpkg, bazel, spack, raw cmake, nix,
| etc)
|
| So what would the pros and cons be today and why is it a selling
| point?
|
| For instance, a pro is that everything is bespoke to the task at
| hand, no hammering square pegs into round holes designed for a
| slightly different use case.
|
| A con is that the entire attack surface is managed by one team.
| CVEs identified and solved on another project is a pretty good
| thing when you depend on that project.
|
| The main reason I'm surprised, though, is that there are some no-
| brainer dependencies these days. Fmt, catch2/gtest,
| metaprogramming libraries, etc.
| callbacker wrote:
| Hi, one of the authors here. BlazingMQ depends on two other
| open source C++ libraries: https://github.com/bloomberg/bde and
| https://github.com/bloomberg/ntf-core. I believe documentation
| writer wanted to highlight that BlazingMQ does not depend on
| frameworks like ZooKeeper, etc.
| i-use-nixos-btw wrote:
| Ah, yes, that makes sense! I'm thinking about framework from
| the wrong angle.
|
| I can definitely see why that's a bonus. So many data
| pipeline tools, but so many written in Java. It's for that
| reason that I've always reinvented the wheel. Perhaps I'll
| check this out...
| omeid2 wrote:
| Sending patches to fix a project is hard work, sending patches
| to fix issues in a projects' upstream projects is even harder.
|
| This alone is an important point to avoid dependencies,
| specially if you're anything like Bloomberg.
| i-use-nixos-btw wrote:
| I run into that problem a lot, to be fair.
|
| Though my approach is to fork and maintain the fork until my
| PR is accepted, merged, and in a tagged release.
|
| Another is to apply a diff to the dependency within the build
| system, which is what I do a lot in Nix (mainly to solve
| cmake issues on hermetic builds).
|
| Either way, it isn't unsurmountable. It doesn't really matter
| so much if it's used as an executable rather than as a
| library, though in the case of the latter it's _really_ handy
| to be able to e.g. pass a custom spdlog sink for logging.
| [deleted]
| endisneigh wrote:
| it never ceases to amaze me how the software engineering
| profession has very smart folks reimplement the same conceptual
| primitives over and over again. I'm curious if hospitals (if
| that's even analogous to a company) do similar things by having
| different processes to solve the same problem.
|
| I imagine this is used for Bloomberg (the terminal) and not
| Bloomberg (the website)?
|
| going back to the article - fantastic animations. I'm just as
| curious to how that was made as the queue itself.
| vsareto wrote:
| >it never ceases to amaze me how the software engineering
| provision has very smart folks reimplement the same conceptual
| primitives over and over again.
|
| Well, no one cares, but I'm not saying that to be mean to
| dismissive, it's just that software production as an industry
| (esp. at a company like Bloomberg) can basically do whatever it
| wants because of low capital costs to produce something. Things
| like writing another queue system will have zero impact on the
| industry unless it's adopted widely (and cargo culted) and
| later made worse.
|
| Maybe I'm about to eat my shoe, but if a capable of team at
| Bloomberg wanted to reinvent something considered unnecessary
| in terms of industry need, they probably could if for no other
| reason that management allowing some indulgence in exchange for
| keeping talented people around is Worth It(tm).
| paxys wrote:
| Smart people solving "solved problems" over and over again is
| exactly how the tech industry progresses. Meanwhile other
| industries are stuck with processes from fifty years ago
| because "it has always been done this way", and anyone even
| suggesting the slightest bit of change is a heretic.
| Sparkyte wrote:
| Part of this evolution is from funded development to opensource
| development.
|
| Then there is language variances like flavors so what ends up
| happening is a million different conceptual primitives that
| fundamentally the same.
|
| It isn't the same because of a few things. At the end of the
| day too it is also about how good the support is over the rest.
| If a new MQ can demonstrate better tooling nothing stops a
| company from adopting it.
| kapilvt wrote:
| mainframe emulation is a thing, aka code change (Ala swap mq
| libraries) at enterprise scale, and supporting production
| without a migration path can very much stop adoption.
| Sparkyte wrote:
| That really depends on how the MQ integration is designed
| and whether or not the in queue data is persistent or
| doesn't get stale. In most cases an application doesn't
| like stale data so a long lived queue is rare. People who
| adopt something typically PoC before adoption.
| marginalia_nu wrote:
| I feel like on the contrary there's an enormous cost attached
| to using prefab components that only kinda do what you need
| them to do. Shops that have embrace this philosophy have been
| some of the most inefficient ones I've ever encountered.
| Screens of yaml code was written to implement tasks that could
| have been performed with lines of programming code; and many
| man-years were spent implementing functions that copied fields
| from one object to another nearly but not quite identical
| object.
|
| It's a big driver of both complex and slow software. Often the
| adaptation layer necessary to get it to actually satisfy your
| needs rivals the size and complexity of the piece of existing
| solution code you're trying to leverage.
|
| Software that is built from the ground-up to your exact needs
| is often both faster and slimmer and more maintainable and
| suffers from far less dependency churn.
| ecshafer wrote:
| There is a balance to Not In House syndrome where you build
| everything yourself, and the opposite: everything should be
| vendored in. I am inclined to think that a healthy NIH
| syndrome is the better place to be. Companies which vendor
| everything move along at a snails pace, and often getting
| your outside product to work with internal needs is a huge
| years long endeavor, to the point just writing a decent
| version yourself that is exactly what you need is really
| useful.
| marginalia_nu wrote:
| NIH has shifted in meaning over time. It used to literally
| mean Not _Invented_ Here.
|
| In the original sense, the NIH approach would be to invent
| a rivalling technique to the message broker. That is indeed
| a questionable idea.
| wmfiv wrote:
| Yeah. This works both ways. At small scale or early stage you
| should try like hell to work backwards from pre-existing
| components so that everything does exactly what you need them
| to do by definition. Eventually you outgrow that and it makes
| sense to build things that do exactly what you want because
| the incremental value is worth it.
| BossingAround wrote:
| > I'm curious if hospitals (if that's even analogous to a
| company) do similar things by having different processes to
| solve the same problem.
|
| Of course they do, unless the processes are regulated by a
| country (such as handling highly addictive opiates in the US).
| quercusa wrote:
| In the US, all hospitals are required to use electronic
| health records (EHRs) and most are implemented in Epic, the
| leading EHR system. But they are almost always electronic
| implementations of their old paper record systems, dating
| back to who-knows-when.
| thfuran wrote:
| Yeah, pretty much any medium to large company is going to
| have a bunch of processes for doing things that any company
| that size needs to do and some more that are a bit more
| industry/domain specific but still basically the same as
| those at every other company in the space.
| jrvarela56 wrote:
| Two reasons I can think of:
|
| 1. It is easier to implement from scratch than understand and
| modify: we see this everyday at a way smaller scale with
| libraries/classes/functions. Not saying this is optimal but
| this is a driver for sure.
|
| 2. Recruiting: some devs love doing this. Fancy OSS project to
| put on your resume, some actually believe there's unfulfilled
| need, you get people like us discussing it (otherwise we
| wouldn't be thinking about Bloomberg).
| erulabs wrote:
| It's a somewhat difficult argument to say that it's easier to
| build from scratch, but I hear what you're saying. At any
| rate, it is absolutely _more fun_ to build from scratch.
|
| I've polled developers about what part of software
| development they hate, and in general, it's deployment and
| maintenance (ie: devops). Using a off-the-shelf-component
| means that the developer _only_ gets to do the un-fun part:
| installing, configuring, productionizing. If they code their
| own solution, they get to spend a few weeks coding!
| dboreham wrote:
| What you say is true. But...this is Bloomberg so I think we can
| give them a pass and take a deeper look. I can't think of a
| more relevant company within which to develop messaging
| middleware.
| jdmichal wrote:
| Exactly. Handling message queuing and distribution is likely
| a core capability for Bloomberg, so it makes a lot of sense
| for them to develop something tailored for their needs.
| LinkedIn did the same with the original Kafka development,
| where distributed eventual state consistency was a core
| capability.
|
| If anything, we should be thankful that they've decided to
| share it with the world.
| KnobbleMcKnees wrote:
| It's cool to be cynical
| [deleted]
| fizwhiz wrote:
| > This section explains the leader election algorithm at a high
| level. It is by no means exhaustive and deliberately avoids any
| formal specification or proof. Readers looking for an exhaustive
| explanation should refer to the Raft paper, which acts as a
| strong inspiration for BlazingMQ's leader election algorithm.
|
| So their own homegrown leader election algorithm?
|
| > BlazingMQ's leader election and state machine replication
| differs from that of Raft in one way: in Rafts leader election,
| only the node having the most up-to-date log can become the
| leader. If a follower receives an election proposal from a node
| with stale view of the log, it will not support it. This ensures
| that the elected leader has up-to-date messages in the replicated
| stream, and simply needs to sync up any followers which are not
| up to date. A good thing about this choice is that messages
| always flow from leader to follower nodes.
|
| > BlazingMQ's elector implementation relaxes this requirement.
| Any node in the cluster can become a leader, irrespective of its
| position in the log. This adds additional complexity in that a
| new leader needs to synchronize its state with the followers and
| that a follower node may need to send messages to the new leader
| if the latter is not up to date. However, this deviation from
| Raft and the custom synchronization protocol comes in handy
| because it allows BlazingMQ to avoid flushing (fsync) every
| message to disk. Readers familiar with Apache Kafka internals
| will see similarities between the two systems here.
|
| "a new leader needs to synchronize its state with the followers
| and that a follower node may need to send messages to the new
| leader if the latter is not up to date". I thought a hallmark of
| HA systems was fast failover? If I come to your house and knock
| on the door, but it takes you 10mins to get off the couch to open
| the door, it's perfectly acceptable for me to claim you were
| "unavailable". Pedants will argue the opposite.
| anentropic wrote:
| "deliberately avoids any formal specification or proof"
|
| how is this possibly a selling point?
| scottlamb wrote:
| > "deliberately avoids any formal specification or proof"
|
| > how is this possibly a selling point?
|
| Context.
|
| In an overview doc? The informal version is accessible to
| more audiences.
|
| As the canonical design doc for the system? It's certainly
| not.
| fizwhiz wrote:
| FWIW they mention this at the bottom of their document
|
| > Just like BlazingMQ's other subsystems, its leader election
| implementation (and general replicated state machinery) is
| tested with unit and integration tests. In addition, we
| periodically run chaos testing on BlazingMQ using our Jepsen
| chaos testing suite, which we will be publishing soon as open
| source. We have also tested our implementation with a TLA+
| specification for BlazingMQ's elector state machine.
| callbacker wrote:
| One of the authors here. Thanks for pointing that out. TLA+
| spec can be found here -- https://github.com/bloomberg/blaz
| ingmq/tree/main/etc/tlaplus.
| amelius wrote:
| How is this "deliberately avoiding any formal
| specification or proof"?
| peheje wrote:
| I think they are referring to the specific section below
| the notice.
| KRAKRISMOTT wrote:
| HA aside, is this faster than Aeron?
| renewiltord wrote:
| Very interesting. Thank you for benchmarks. Thank you for
| sharing. In a zero-replication no-leader-election (or all-in-one-
| node) scenario what is min latency observed? It appears
| predominant delay is in ACKing to ensure durability, right?
|
| C++ and Java client examples are very simple which is cool.
|
| Would appreciate if there is hook to enable `SO_TIMESTAMPNS` on
| socket and retrieve `cmsg` to get accurate timestamping and
| permit client code benchmarking.
| 1letterunixname wrote:
| Marketing is all well and good. Let's see the data of p99 latency
| and throughput numbers compared to rabbitmq, kafka, activemq,
| qpid, and hornetq.
|
| Also, it's worth identifying and setting aside use-cases for
| broker-less messaging where 0mq, nanomsg, etc. could be more
| appropriate.
|
| Edit: There's only published performance data of itself, which
| has little value:
| https://bloomberg.github.io/blazingmq/docs/performance/bench...
| notjoemama wrote:
| It doesn't use UDP. So my tongue-in-cheek reply is, it doesn't
| qualify to have "blazing" in the name. Hopefully that's not just
| funny inside my own head. :)
|
| https://bloomberg.github.io/blazingmq/docs/architecture/netw...
| rewmie wrote:
| > It doesn't use UDP. So my tongue-in-cheek reply is, it
| doesn't qualify to have "blazing" in the name.
|
| Arguably, if you start with UDP but you also need to ensure
| retransmission and reordering, you eventually end up
| reimplementing TCP but poorly.
| anonymoushn wrote:
| I've found that for many applications TCP doesn't really make
| optimal choices in these areas or expose knobs for users to
| make better choices.
| jayd16 wrote:
| I guess you could implement QUIC instead.
| lordnacho wrote:
| Check out Aeron, they have a way to multicast (thus UDP)
| reliably and ordered. Based on negative acks. Super fast.
| phishhook wrote:
| Sounds like PGM
| HappySweeney wrote:
| NORM from the Naval Research Labs is another implementation
| of the same concept.
|
| https://github.com/USNavalResearchLaboratory/norm
| jonathanoliver wrote:
| Aeron is another awesome project from Martin Thompson
| (Disruptor, Simple Binary Encoding, Mechanical Sympathy,
| etc.).
|
| That guy really knows his stuff regarding performance.
| amanj41 wrote:
| When I was working at BB I could have sworn they had a fanout
| mode using UDP. Guess not.
| glonq wrote:
| Are they working on other API's besides Java and C++?
| frakkingcylons wrote:
| The python client library is planned to be published. They say
| they need to remove some dependencies and update the workflow
| to something more suitable for OSS.
| callbacker wrote:
| Hi, one of the authors here. We will be releasing a Python
| client library in a few weeks.
| mkl95 wrote:
| How does it compare to NATS.io?
| leetharris wrote:
| It is amazing to me that NATS.io is not more popular. It is so
| much better than Kafka, extremely simple, and extremely light.
| slantedview wrote:
| Nats is simple because it doesn't bother addressing some of
| the scalability problems that Kafka was designed to address,
| like clustering with partitions:
|
| https://docs.nats.io/legacy/stan/intro/partitioning
| coder543 wrote:
| You linked to NATS Streaming, which has been deprecated for
| years. It's not relevant. It can still be used for legacy
| code that still needs it, as the URL even shows, but it is
| not recommended for new applications.
|
| The current standard is NATS JetStream, which absolutely
| addresses scalability, arguably to an even larger degree
| than Kafka, since it can be used to create a globally
| interconnected supercluster of clusters where messages can
| be configured to flow between regions as needed. I doubt
| anyone would run a Kafka cluster that spans multiple
| regions.
| alberth wrote:
| > "Carefully architected and written in C++"
|
| Amazing to see a non-Rust product posted to HN.
| qwertox wrote:
| True, but maybe its age could play some role in it being a C++
| project.
|
| > BlazingMQ is an actively developed project and has been
| battle-tested in production at Bloomberg for 8+ years.
|
| https://github.com/bloomberg/blazingmq
| beanjuiceII wrote:
| don't worry the army of Rust authoritarians are waiting to
| reply as we speak. how software is just not ethical or correct
| unless its in rust
| harerazer wrote:
| Recently, mold 2.0 was posted, which is also C++.
| geertj wrote:
| I've been getting pretty deep in C++ recently and the name
| Bloomberg shows up all across the C++ standards community. They
| seem to be a strong C++ shop.
| tkhattra wrote:
| yep, the authors of "Embracing Modern C++ Safely" [1] are
| Bloomberg folks
|
| [1] https://vittorioromeo.info/emcpps.html
| hkt wrote:
| It has a gorgeous landing page but I'm not really sure why this
| is better than any other MQ. Can someone more knowledgeable
| provide any insight into its comparative advantages?
|
| Edit: I did see the comparisons page, but.. well, there's more to
| life than Rabbit and Kafka!
| traceroute66 wrote:
| > Can someone more knowledgeable provide any insight into its
| comparative advantages?
|
| Well, you know, there is the "Comparison With Alternative
| Systems" page[1]. :-)
|
| Sadly it doesn't include NATS in the comparison. I would have
| been interested to see that. But the usual suspects (Rabbit &
| Kafka) are given a mention.
|
| [1]
| https://bloomberg.github.io/blazingmq/docs/introduction/comp...
| rlonstein wrote:
| I'd also be interested in seeing how it compares to AMQP
| implementations (Apache Qpid for example) and any of the
| proprietary message queue systems (IBM and Microsoft) they
| refer to but don't name-check.
| vyskocilm wrote:
| FJI: There is a faq entry about that
| https://bloomberg.github.io/blazingmq/docs/introduction/comp...
| Cthulhu_ wrote:
| I mean one unsubstantiated armchair take: it's newer than
| Rabbit / Kafka (RabbitMQ was built in 2007, Kafka in 2011),
| since then the learnings from distributed systems, message
| queues and programming languages has improved by leaps and
| bounds, and the existing ones will have backwards compatibility
| / legacy issues.
|
| That said, there are indeed plenty of commercial and
| noncommercial alternatives, so now that it's been published I'm
| sure there will be some thorough people making comparisons
| soon!
| travisgriggs wrote:
| Interesting that there's a comparison (albeit low on details) to
| Rabbit, but not MQTT? Anyone had experience with both MQTT and
| BlazingMQ able to compare/contrast?
| dbrueck wrote:
| Isn't MQTT more of a standard than an implementation?
| sheerun wrote:
| I think he means implementation of MQTT protocol, like
| https://mosquitto.org/
| winrid wrote:
| Yes. They prob mean rabbitmq.
| smarx007 wrote:
| I would be interested in the differences between BlazingMQ,
| AMQP, and MQTT protocols and the invariants afforded to the
| clients, not the implementations per se.
| mizzlr_ wrote:
| LavinMQ is much better.
| konart wrote:
| Better how?
| renewiltord wrote:
| Appears 10^3 times latency for 1 KiB message according to here
| but better at lower 16 B size
| https://www.cloudamqp.com/blog/lavinmq-benchmarks.html
|
| Seems situation dependent.
| phishhook wrote:
| Does that do HA/replication/clustering etc though?
| user6723 wrote:
| I would be interested in trying this out if it were written in
| Rust or Austral.
| Cthulhu_ wrote:
| I don't understand the language preference; it's an
| implementation detail.
| Thaxll wrote:
| Which version of C++ is this built on?
| coder543 wrote:
| One person opined that it is similar to C++03:
| https://news.ycombinator.com/item?id=36898899
| hallfox wrote:
| Hi, dev here. BlazingMQ is written with a baseline of C++03,
| though our docker tutorial builds it with C++17.
| frutiger wrote:
| It targets AIX and Solaris (in addition to Linux) and vendor-
| supported C++ toolchains have spotty support for C++11 and
| newer.
| sitkack wrote:
| TLA 0.1%
|
| Finally! Thank you!
| melenaboija wrote:
| I have been working on an engine to perform heavy computations
| that is distributed and my messaging choice has been gRPC +
| Flatbuffers + proxys and I was wondering if someone could name
| some reasons or use cases to chose message brokers vs a lower
| level approach like gRPC. Is it mostly because of
| synchronous/asynchronous needs or there is something else such as
| latency or even ease of implementation?
| ozarker wrote:
| Depends on your use case. Queueing messages enables the
| competing consumer flow which implicitly allows load-leveling
| which is great if your traffic is spiky and you need a little
| time to auto-scale or other scenarios like that. If you're just
| looking to blast out data to all consumers or just load-
| balancing is fine, approaches like yours are probably better.
| imachine1980_ wrote:
| grpc client a -> grpc server a for client a grpc client b ->
| grpc server a for client b
|
| grpc client a -> message broker(implemen a) -->
|
| grpc client b -> message broker(implemen b) -->
|
| message broker-> your app for the broker this make sense if
| dont make sense manage the quque in your app if ypu want to
| abstract the implementations is worth if you have n to 1 or n
| to n and all works different
| mgaunard wrote:
| I don't understand how you can make queueing high-performance.
| Queueing things is literally the opposite of making things fast.
| thefounder wrote:
| Perhaps it's about efficiency, scalability and
| responsiveness/latency.
| masom wrote:
| > I don't understand how you can make queueing high-
| performance. Queueing things is literally the opposite of
| making things fast.
|
| You likely need to read about message queues, why they are
| used, and why they have performance constraints. Message queues
| are a system, and the system to maintain those queues and
| content has an overhead. The project is building a solution
| that is higher performing than comparable systems.
|
| Queuing things often makes things faster (!) as you are usually
| dealing with limited resources (ex: web and job workers) and
| need to operate on other systems (ex: payments) that have their
| own distinct capacity.
|
| If you have 1000 web workers, and your upstream payment
| provider supports 10 payments per minute, you'll quickly have
| timeouts and other issues with 1000 web workers trying to
| process payments. Your site won't process anything else while
| waiting for the payment provider if all web workers are doing
| this task.
|
| A queue will let you process 1000 web requests, and trickle 10
| requests to your payment provider. At some point that queue
| will fill, but that's a separate issue.
|
| Meanwhile, your 1000 web workers are free to process the next
| requests (ex: your homepage news feed).
| gdy wrote:
| "At some point that queue will fill, but that's a separate
| issue"
|
| Would you recommend something to read about that?
| masom wrote:
| It's a bit difficult to cover as it is highly dependent on
| the queue system you use.
|
| You'd usually want your queue system to fail the enqueue if
| it is full, and you'd want monitoring to ensure the queue
| isn't growing at an unsustainable rate.
|
| It also forces you to think a bit about your message
| payload (rich data or foreign keys the worker loads).
|
| RabbitMQ, Redis-based queues (ex: ActiveJob or sidekiq),
| Gearman, and others will all offer different mechanisms to
| tackle full queues.
| Cthulhu_ wrote:
| What is the faster / more scalable alternative to a message
| queue, or are you hyperfocusing on the word queue?
| therealdrag0 wrote:
| You're misunderstanding the discussion. If you think of queuing
| as "waiting to process" then yes it's slower than not waiting.
| But that's not at all the necessary implication of these
| components. In fact "synchronous" requests are buffered
| (queued?) internally and can be delayed before processing. So
| the distinction is irrelevant to performance. And the
| implementation is what matters.
| LoganDark wrote:
| Queueing things can result in better throughput because you
| don't have to wait for the other side to process your last
| message before you can begin work on the next one.
| jeffbee wrote:
| That's just buffering. A lot of people use Kafka this way, as
| an impedance adapter for a producer with an internal
| architecture that can't tolerate blocking on production. Of
| course this requires you to unrealistically assume that Kafka
| is a zero-impedance message sink.
|
| But what I think the other post is alluding to is the fact
| that in-order delivery is often not a requirement, and can be
| an anti-feature. I know that in every use of Kafka I have
| personally encountered the relative order of messages was
| quite irrelevant but the in-order implementation of Kafka
| converts the inability to process a single record into an
| emergency.
| therealdrag0 wrote:
| There's a lot of brokers without ordering guarantees.
| ActiveMQ for example.
| insanitybit wrote:
| I think there's two really obvious points.
|
| 1. They're referring to the queueing itself. They're not saying
| "this will make some other thing fast" they're saying "this is
| fast at queueing".
|
| 2. Queuing is not the opposite of making things fast.
| hbarka wrote:
| 1,000,000 messages per second isn't high performance?
| say_it_as_it_is wrote:
| Why is Bloomberg approving the open sourcing of this system?
| puchatek wrote:
| Same reasons any company open sources their software
| SkipperCat wrote:
| If it gains wide adoption, then they've got a global community
| of people providing free software engineering for their
| messaging platform. Its a win-win.
| loire280 wrote:
| Why wouldn't they? This technology is not a competitive
| advantage for their company, wouldn't make sense for them to
| sell as a standalone product, and it helps their engineering
| department attract good talent.
| Cthulhu_ wrote:
| Because they're not making money off of their queueing system,
| but what they build with it; to them it's a cog in their
| machine, and open sourcing it means others may adopt it and
| help them maintain and improve it.
| BossingAround wrote:
| Hiring strategy--there are a ton of people who'll take a
| slightly lower salary if they can get a public record of their
| code commits.
| esafak wrote:
| They must not consider it a trade secret, so they can earn good
| will, generate public contributions, and advertise their brand
| by sharing it.
| insanitybit wrote:
| > Supports a unique multi-hop network topology leading to a
| distribution tree for each queue, thereby leading to network
| bandwidth savings for certain use cases.
|
| This is very interesting. One of the benefits of Kafka is
| partition routing so I'm curious to learn how this may compare.
| Sparkyte wrote:
| No idea, but I am interested too. One of the reasons this was
| probably created was because of a network constrained
| environment where data is expensive in more than one way.
|
| Cloud Providers can be expensive.
| eric-hu wrote:
| > Cloud Providers can be expensive.
|
| Do you have a story around this? If so, would love to hear
| more about it.
| lelandbatey wrote:
| Specifically, egress from Cloud Providers (transmitting
| data from inside the cloud to a computer outside the cloud)
| is the usual culprit for being "very expensive". It's part
| of a pricing strategy by cloud providers that encourages
| folks to put all their infrastructure inside of the cloud
| and to have none of it outside the cloud. As one example of
| the ink spilled over this topic, see this discussion from 2
| years ago about AWS's very high egress fees:
|
| Article "AWS's Egregious Egress" (2021)
| https://blog.cloudflare.com/aws-egregious-egress/
|
| HN discussion:
| https://news.ycombinator.com/item?id=27930151
| WJW wrote:
| Bloomberg (who built this) operate a system that makes money
| by transmitting the same data to a huge amount of clients.
| Having one server transmit the same piece of information to a
| million client is approximately 500 times slower than having
| one server transmit it to 1000 proxies who each pass it on to
| 1000 end users.
| pclmulqdq wrote:
| The actual architecture of most of these systems in finance
| relies heavily on UDP multicast, which is a technology that
| big tech has basically forgotten because it can be tough to
| administer at large scale.
|
| Multicast puts the entire load of distributing the data in
| its natural place: the network. The pub/sub queues that the
| rest of the world uses are more complicated and a lot
| worse.
|
| This appears to be the beginning of a "cloud-native"
| bloomberg that can't lean on multicast.
| tetrep wrote:
| Is this implemented with any sort of thought to security?
|
| I don't see anything that implies as such, all FAQ and
| comparisons to other message queues say nothing of security (I
| skimmed and ctrl+f so I could have missed it). This doesn't bode
| well for a network-based application written in a memory unsafe
| language.
|
| edit: I'm surprised so many people are interpeting this as a
| weird cargo cult thing. Application security includes a lot of
| mundane and critical things like:
|
| Authentication, authorization, message signing / authentication
| (e.g. HMAC), encryption, secret/key management, how the system
| handles updates, etc.
|
| You can read a bit more about it here:
| https://en.m.wikipedia.org/wiki/Application_security
| mrits wrote:
| Would you prefer it written in Rust with everything wrapped in
| unsafe block?
| tetrep wrote:
| I care more about methodology than specific methods. I would
| like applications that handle I/O, especially network I/O, to
| be designed and built with security in mind. I asked my
| question because that doesn't appear to be the case here, and
| I think that is dissapointing and concerning from a security
| perspective.
| mikepurvis wrote:
| Rust's guarantees are all about what's happening inside your
| own process with memory ownership. "Security" in the context
| of a middleware like this is almost certainly more about
| external validation/auth-- what prevents some random node
| from injecting messages into the queue, or adding itself as a
| subscriber to the output?
|
| Also, it's a super bad-faith argument to talk about an entire
| program being in an unsafe block. Rust's unsafe is about
| specifically and intentionally opting out of certain
| guardrails in a limited scope of code where it should be
| fairly easy for a human reviewer to understand and validate
| the more complex invariants that aren't enforceable by the
| compiler itself.
| coder543 wrote:
| That is an obviously false dichotomy. The author of the
| comment never mentioned Rust, and wasn't asking about Rust.
| Such a false dichotomy feels like flamebait, which is against
| HN guidelines.
|
| Your false dichotomy also implies that "unsafe" blocks are as
| unsafe as C++, which is not true. "Unsafe" in Rust turns off
| very few checks[0], most of them are still active. No one
| would write a serious Rust program entirely in unsafe
| anyways.
|
| Regardless, asking about security considerations is a valid
| thing to do, even if it were written in Rust. Security is not
| just about memory safety.
|
| [0]: https://doc.rust-
| lang.org/std/keyword.unsafe.html#unsafe-abi...
| touisteur wrote:
| Furthermore, choice of language can have an effect as to
| the actual security of the networking layer.
|
| Having parsers (and serializers) proved absent of runtime
| errors (e.g. with something like SPARK) is a form of
| guarantee I wish I'd see as the default in any 'serious'
| network-facing library or component. It's not even that
| hard to achieve, compared to the learning curve of the
| borrow checker.
|
| Once rust gets plugged into Why3 and gets some industrial-
| grade proof capacity, the question of 'is it written in
| rust?' will be automatic (as in 'why would you do it any
| other way?').
| cogman10 wrote:
| I perused the code and it was written in a C++03 style. Pointer
| chasing and manual memory management in the few files I poked
| through. (IE, no unique_ptr, just new/delete).
|
| This may be perfectly safe... however, I'd give it a few years
| of battle testing before touching it.
|
| Here's an example:
|
| https://github.com/bloomberg/blazingmq/blob/main/src/groups/...
| d4damager wrote:
| I don't see any examples of new/delete there?
|
| Also they do make heavy use of managed pointers in the
| codebase.
| HWR_14 wrote:
| Isn't this opensourced after 8 years of being battletested in
| a very time/error sensitive environment (feeding bloomberg
| terminals)?
| ljm wrote:
| Security purely in terms of the language used, or security in
| terms of cluster access, authentication and so on?
|
| I mean, I don't often see documentation for networked services
| that goes in-depth on how exactly they write memory safe code
| in C or C++, as if that would mean the application is therefore
| secure; it's always much higher level.
| tetrep wrote:
| > Security purely in terms of the language used, or security
| in terms of cluster access, authentication and so on?
|
| I'm personally interested in both, but from a docs
| perspective I'd mostly expect the latter, more "security for
| users of this system" than anything else. I'm a little
| embarassed I didn't remember it until now, but I think the
| general term is "application security."
|
| I'm concerned that I don't see any hints of either in the
| project. I'm not at a computer with access to my Github
| account so I can't easily search the source code for hints of
| obvious signs of care or neglect with respect to security.
| ljm wrote:
| https://github.com/bloomberg/blazingmq/blob/93e6426d474a429
| 3...
|
| Not sure where they are on the roadmap but authentication
| in particular is 'mid-term' but lacks detail.
|
| I expect that if you want to deploy this then you're doing
| it on an internal network; its deployment examples are
| based on Docker so I expect it's relying on what you can do
| with k8s.
| tetrep wrote:
| Thanks! That is the sort of stuff I was asking about.
|
| It looks like it's an after thought but at least on their
| mind now, which is very fair with respect to Bloomberg's
| wants/needs. It'd be nice if they had a bit of a warning
| about using this until it has some basic auth(n) and TLS
| since they're releasing it to the public. I think it is,
| relativley speaking, rude to release insecure networked
| software without giving users a notice as to what sorts
| of insecurity is at least known/expected.
| femto113 wrote:
| Adding a veneer of security isn't necessarily superior to
| leaving it out altogether. Systems of this sort are best
| secured at the network level, i.e. only trusted hosts
| should be able to connect to it. Redis is a good example
| of where this has been tried: it does support password
| based log in, but the password is stored and transmitted
| in plaintext, and a redis server will happily accept
| thousands of auth attempts per second making brute
| forcing a viable attack. Rather than improve the auth
| system Redis has instead doubled down on encouraging
| appropriate network level security by defaulting to only
| being accessible to the local host, so admins have to go
| through an explicit step (with warnings) before they can
| just expose it to the internet.
| ljm wrote:
| Do you really need it? I have no idea why you would
| expose a message queue to the public internet; it's a
| purely internal concern.
|
| > . I think it is, relativley speaking, rude to release
| insecure networked software
|
| Respectfully, I think _you_ are being rude by not being
| forthcoming on the parameters for safe networked
| software.
|
| You should audit the stuff you host in the stack you work
| on and report back on this.
| rewmie wrote:
| > Is this implemented with any sort of thought to security?
|
| There's a hefty load of cargo cult mentality revolving the
| topic of security. Security is not a product, but a process.
| This sort of talk is getting tiresome.
| weekendcode wrote:
| Built in C++, awesome :)
| nodesocket wrote:
| Awesome landing page and looks great. Documentation is very well
| done. Would love to see more clients (python) and a web u/i.
| callbacker wrote:
| Hi, one of the authors here. We will be releasing a Python
| client library in a few weeks.
| fidotron wrote:
| That fan-out functionality is particularly interesting for anyone
| needing to implement the upstream part of a firehose API, which
| of course Bloomberg do.
| arnold_palmur wrote:
| I wonder how this compares to something like Aeron performance-
| wise - though Aeron is strictly messaging (not queueing).
| synergy20 wrote:
| how does this compare against other existing message queuing
| systems?
|
| the github shows its codebase is in c++ with gcc-7.3, that means
| it might be using c++14 in the code, or even c++17, which is
| nice.
| eightysixfour wrote:
| What a phenomenal landing page. Thanks to the animations I have a
| good idea of what this does and how it does it. You barely need
| to know what message queuing is to understand it.
| nine_zeros wrote:
| Agreed. This feels like a page created from enthusiasm, not
| from a place of promo-seeking performance review.
| ElongatedMusket wrote:
| Getting started pages are great too, with guided examples and
| use cases:
| https://bloomberg.github.io/blazingmq/getting_started
|
| BMQ license is apache 2.0 for anyone curious
| gr33nq wrote:
| Does anyone recommend any particular piece of software for
| creating these types of visualizations with relative ease? I've
| always been a fan of animations for helping to understand some
| technical topics.
| erwinkle wrote:
| They use anime.js for their animations https://animejs.com/
|
| As seen in their source code here: view-source:https://bloomb
| erg.github.io/blazingmq/assets/animations/anim...
| Austizzle wrote:
| Lottie.js is a library for animations in JavaScript, and
| after effects can export to it! You're obviously limited in
| what you can export, but if you do a basic after effects
| animation with shapes and text, it works fine.
|
| Pretty sure it was made by Airbnb, and used in tons of apps
| like Google home
|
| Works really well
| jasonlotito wrote:
| Okay, this is going to sound weird, but I've used Keynote in
| the past. I'm sure PowerPoint can do this as well, but
| Keynote has Magic Move, and you can export this animation.
| It's very easy to work with, and do simple animations. I'm
| sure there are much better tools for this, but as someone who
| knew Keynote, this was always effective for me.
| whoisthemachine wrote:
| All the replies mention technologies to help you build one,
| but this chart looks like a modified, animated version of a
| Petri Net to me, which is a very useful chart for visually
| describing the flow of data between systems. I've built
| similar charts using PowerPoint and exporting the PowerPoint
| as a GIF... same effectiveness for internal use, but not
| something you're going to want to brag to your friends about.
| maxwellito wrote:
| As erwinkle mentioned, these animations use anime.js If you
| want to build something but don't really know how to do it, I
| would suggest the following:
|
| 1. Build the visual content in an SVG with Inkscape. It
| should be easy for you to build the layout with assets/item
| to animate. Don't forget to group items and set names so you
| can reference them later.
|
| 2. Once exported, insert the SVG in an HTML page and add
| anime.js. This library will let you animate the content you
| just created (you should be able to reference your named
| items from Inkscape and animate them). The learning curve
| might be tough depending on your experience regarding
| animation (or CSS in general).
|
| Good luck :)
| danielvaughn wrote:
| I've heard Framer can make some impressive animations, but I
| don't know much about how well they export to web. What's on
| display here is really cool, it's all animated SVG.
| munro wrote:
| i've just used html/css/javascript then screen recorded it
| treprinum wrote:
| animatron.com for animated SVG or JS. I used it for some
| algorithm animations myself.
| scottlamb wrote:
| The animations are very slick, but I can still grumpily pick at
| them: the "Clustering and Quorum-based Replication" one shows 4
| total servers (suboptimal number because quorum requires 3/4
| where I'd prefer to require 2/3 or 3/5) and all the followers
| acking before the primary proceeds (the animation could show
| one lagging instead).
| BeeOnRope wrote:
| 4 might be useless compared to 3 from a "required quorum" PoV
| but if you want to stay up with 1 failure and the load can't
| be handled on 3 servers but 4 is OK, it is optimal isn't it?
|
| I.e. selection of ideal server count is a multi-dimensional
| problem.
| scottlamb wrote:
| Worse than useless from a quorum perspective, because
| you've increased the number of servers that have to
| participate in quorum (quickly) without increasing the
| number that don't. Looking at it the other way, you go from
| having problems when 2/3 are slow/broken to having problems
| when 2/4 are slow/broken.
|
| Yes, 4 servers might handle more load than 3 servers [1],
| but so would 5 servers. I'd guess that if you've got enough
| message queue traffic that 3 servers can't handle the load,
| those servers aren't a significant fraction of your total
| cost. You might as well go to 5. Likewise, if 5 aren't
| enough, I'd skip over 6 and go to 7. If 7 aren't
| enough...I'd probably rethink my design entirely...but I'd
| prefer 9 to 8.
|
| Alternatively, quorum systems can support "zero-weight
| replicas" which effectively don't vote but still can be
| useful for eventually consistent reads. Then you have this
| server that helps handle the load but doesn't increase the
| number of servers that have to participate in a round.
|
| [1] if the load is overwhelmingly eventually consistent
| read operations and their multi-hop topology with proxies
| doesn't sufficiently reduce the load on the replicas. It's
| not obvious to me if this scenario is plausible or not.
| amelius wrote:
| Also, why is the middle replica special, in that the message
| is passed through it in the end?
| rewmie wrote:
| I agree, this is perhaps the very best landing page I ever saw.
| What a treat.
| davidkunz wrote:
| It's called BlazingMQ but not written in Rust?!
| gabereiser wrote:
| I'll admit, I went into their repo expecting to find Rust, or
| at the very least go. Pleasantly surprised to see C++ code
| though I would need to look further to see if it's "good" C++
| code. The naming conventions of the source tree lends me to
| suspect it's not.
| beanjuiceII wrote:
| i too determine if code is good based off its naming
| conventions and other things like my linter is set to enforce
| MiXeD_SpOnGeBoB-KeBaB_CaSe as the default casing
| [deleted]
| Cthulhu_ wrote:
| The language is an implementation detail; what makes you think
| Rust would be better suited for such a specialist piece of
| software?
| higherhalf wrote:
| It's a joke, as some Rust projects used to repetitively claim
| to be "memory safe & blazing fast", thus becoming a tongue-
| in-cheek phrase. I also anticipated language to be Rust, but
| it would be way too comical.
|
| https://github.com/mTvare6/hello-world.rs
___________________________________________________________________
(page generated 2023-07-27 23:00 UTC)