[HN Gopher] How to use RabbitMQ in service integration
___________________________________________________________________
How to use RabbitMQ in service integration
Author : olikas
Score : 50 points
Date : 2021-05-17 10:50 UTC (12 hours ago)
(HTM) web link (www.erlang-solutions.com)
(TXT) w3m dump (www.erlang-solutions.com)
| pkb wrote:
| DON'T. RabbitMQ is fragile and breaks under low memory/high swap
| conditions. Usually due to epmd breaking. I know, have to fix it
| (usually by killing epmd) daily.
| haswell wrote:
| Is this a reason to not use it at all, or a reason to ensure
| the environment is properly tuned if you do use it?
|
| Seems your conclusion "DON'T" implies the former, but this
| seems unnecessarily extreme.
| eternalban wrote:
| I've skimmed the comments here and of course the article. Some
| points based on experience with Kafka, RabbitMQ and exploratory
| (and ahead of hype cycle) look at Pulsar:
|
| RabbitMQ is an excellent _messaging_ middleware. But simply
| remember that it is not designed for optimal performance when
| holding on to data. "It is a river, not a lake". Performance is
| very sensitive to the amount of data that is in flight between
| send and receive end points.
|
| Kafka is a "lake", but will not give you the rich routing and
| diverse semantics of Rabbit. If you are building 'event sourced'
| systems, Kafka and similar systems are a better choice.
|
| Pulsar has a highly articulated architecture. It is built on
| Bookkeeper and decouples the persistent store servers from the
| client servicing servers. If you want to avoid the rebalancing
| pain of Kafka, Pulsar is the solution. However, Puslar has many
| more moving pieces.
|
| Both RabbitMQ and Pulsar are authentic 'middle ware', and
| extensible. Kafka is, true to its genesis, a highly performant
| _distributed log_.
|
| Durable, recoverable, and performant distributed
| messaging/journaling is inherently complex. Make sure you know
| what is it that you precisely require, and one of the above
| solutions will likely serve you well.
| rad_gruchalski wrote:
| Mostly agreed but I'd say: Pulsar is a lake, Kafka is a river,
| RabbitMQ is a creek. Kafka still moves fast and once you out of
| retention, difficult to read past. Pulsar, thanks to adding
| bookies to the storage pool, can grow and does not have a
| retention to commit to.
| perlgeek wrote:
| At work, we've been using a RabbitMQ instance as a central
| integration point (for about 7ish years now, I believe), and it's
| been working very well so far.
|
| My observations so far:
|
| * Running a single RabbitMQ is pretty boring in the good sense.
|
| * We haven't managed to switch to a cluster for HA yet; it seems
| that software that deals with RabbitMQ clusters must be cluster
| aware (consume from queues on all instances and the likes), and
| it wasn't worth our effort to fix all the applications.
|
| * In the long run, the lack of tooling is hurting us. Want to do
| green-blue deployments? Canary deployments? When your services
| run on HTTP(S), there are simply tools for that. When your
| services consume from AMQP queues? You have to go searching for
| solutions, possibly build your own plumbing.
|
| In the end, it turns out that we use publish/subscribe far less
| often that direct request/response patterns, so for new stuff I'd
| likely go with HTTPS instead of AMQP today.
| jpgvm wrote:
| RabbitMQ went on my "never again" list of things after dealing
| with it for years as part of OpenStack and projects before that.
| At the time there wasn't many viable alternatives to AMQP that
| were OSS and reasonable.
|
| However many split brains and grey hairs later I decided RabbitMQ
| was almost never worth it regardless of how many of AMQPs
| advanced features you could make use of.
|
| For the longest time I just made do with Kafka but this had
| serious deficiencies when implementing queues because of the
| cumulative ack only nature of Kafka.
|
| Recently I have started using Pulsar which provides selective ack
| and all the best parts of AMQP without the complexity and
| unneeded parts. i.e it has things like scheduled delivery and
| TTLs in addition to the all important shared subscription which
| makes queues "just work" on top of streams.
|
| If you want something like RabbitMQ but with a simpler API and
| are comfortable with JVM services give Pulsar a go. It's not for
| everyone but if you are already using a lot of the big data stack
| it's probably a good fit.
| Serow225 wrote:
| I feel dumb saying it, but as someone whose had a lot of
| experience with messages buses (Rabbit and AMQP1.0), I've
| always struggled to understand what domains/situations Kafka is
| actually the best fit for. It's probably because of the areas
| that I work in which doesn't make it obvious, but I'd love to
| hear what exact scenarios it does actually make sense to use
| Kafka for instead of Rabbit or AMQP1.0 :)
|
| RabbitMQ is one of those things that I've always found better
| to let the experts run (managed SaaS), unless your team is
| really wanting to take on the burden of becoming an Erlang
| distributed system debugger :)
|
| Pulsar seems really interesting... There are now more managed
| Pulsar offerings coming online (StreamNative, DataStax who
| bought Kesque, Pandio, etc)
| rad_gruchalski wrote:
| Well, they're two different things. AMQP 1.0 is everyone to
| everyone messaging where every party can be a consumer and a
| server. RabbitMQ, traditionally, is a queue. You add a
| message, you take it and lock it, if no processing
| confirmation, it is released back for someone else to
| process. Kafka is an append only log. You put a message in
| and consumers just roll over them. Rabbit/amqp is random
| access, Kafka sucks at random access. With amqp, you'll have
| hundreds/thousands of queues, this may be difficult with
| Kafka.
|
| You'd use Kafka more as an unbounded buffer and build
| different paradigms on top of it. It not unusual to ingest
| 100s if mbits of data into kafka, potentially saturating the
| network while also reading that much out. Amqp is better for
| large number of queues where each queue has less messages in.
| Think mqtt, websockets - many, many consumers.
|
| It would be reasonable to use both next to each other.
|
| But I'd never go for rabbitmq. I'd go for azure servicebus or
| artemis with qpid.
| reader_1000 wrote:
| > I'd love to hear what exact scenarios it does actually make
| sense to use Kafka for instead of Rabbit or AMQP1.0 :)
|
| If order of message processing matters, then Kafka is better
| suited then AMQP. For example, In a distributed application
| for money transfers, if AMQP used, message order will be lost
| and some problems will occur in the following scenario:
|
| User A with an accound of $1000 makes order for two transfers
| T1 ($600) and T2 ($500) - Rabbit delivers T1
| to server1, before processing message, server1 enters a full
| GC. - Rabbit delivers T2 to server2 and server2
| processes message immediately, now User A's account have $500
| - Server1 resumes its life after the end of GC, but fails to
| process T1 since account's balance is less than required
| amount.
|
| However, it is T2 that should have failed because User A
| ordered T1 first and T2 after.
|
| In Kafka, when user account identifier is used for
| partitioning key, all User A's messages will be processed by
| same consumer (i.e server1), so even if server1 enters a full
| GC, that is OK, since T2 will be processed after T1.
| robalfonso wrote:
| Wow, and here I was feeling like I was the only one who thought
| this way! Such a similar experience to yours.
| varispeed wrote:
| I know workplaces where you can get sacked just for mentioning
| RabbitMQ...
| closeparen wrote:
| >For the longest time I just made do with Kafka but this had
| serious deficiencies when implementing queues because of the
| cumulative ack only nature of Kafka.
|
| We built a Kafka consumer that's effectively capable of
| selective acks by producing bad messages to separate topics.
| It's a little silly but it works.
| KptMarchewa wrote:
| DLQ is the way to do this. Nothing silly about it.
| closeparen wrote:
| It's a heavyweight custom client to hack Kafka into being
| something it isn't, that other message brokers are. That's
| what's silly.
| suresk wrote:
| Pulsar definitely looks like a great combo of capabilities from
| Kafka + AMQP. I have been wanting to try it out with some of
| our stuff, but inertia/time constraints have made it hard to
| consider moving away from Rabbit.
|
| I'm curious what makes it go on your "never again" list? We've
| definitely had our fair share of issues with it, namely -
|
| - Really easy to misconfigure queues/exchanges, especially
| trying to do something like have a retry + DLQ setup.
|
| - If you have a queue build up to a large number of messages
| (100 million+) for whatever reason, purging it will probably
| bring down the cluster.
|
| Overall, our experience has been mostly positive. It isn't on
| my "never again" list, but I'm definitely wary of some parts of
| it and it is on my list of one of the more difficult pieces of
| our infrastructure to scale.
| [deleted]
| bironran wrote:
| Can only echo parent. 3 places of work in varying sizes, 4
| projects in varying maturity, not a single RMQ administration
| staff that was competent enough to reliably run the cluster.
|
| Which of course leads me to believe the problem isn't with the
| people but with the ridiculously high threshold of knowledge,
| experience and app developer self-control needed to run RMQ
| successfully.
|
| As parent said, many meltdowns later, I'm now firmly in the "No
| Rabbit!" camp. Redis pubsub/queues for immediate lossy
| delivery, kafka / gcp pubsub / aws sqs for less latency
| sensitive flows that require more consistency guarantees.
| datavirtue wrote:
| I concure. I like RMQ because I know how to configure and
| administer it but I would never trust anyone with it. My
| first exposure to it was on a project where the lead
| architect failed to read the documentation let alone
| understand any of it. Several years later I was able to fix
| it and then left it in the hands of some other incompetents.
| jayd16 wrote:
| Hmm, what issues did you run into? I've used it on a few
| projects in a mirrored way and it was always fine. Is the
| clustering the issue?
| reacharavindh wrote:
| Not OP, but that is my experience. It worked like a rock on a
| single server. Clustering brought us issues rooting from its
| complexity. Split brain scenario, corruption of the Mnesia
| database and such. We went back to single server mode.
| Plugawy wrote:
| Same! Never bothered with the so-called HA setup after
| running a cluster for few months. Making all messages
| durable + backups of underlying storage are sufficient,
| while the do not prevent an outage, at least bringing the
| system back to an operational state is fairly
| straightforward
| jayd16 wrote:
| >Mnesia
|
| Is Mnesia just terrible or is there some trick? I did run
| into issues you're talking about with ejabberd clusters.
| 100011_100001 wrote:
| I find that one of the major drawbacks of RabbitMQ is how much
| CPU it requires to run efficiently. If you add clustering on top
| of it it can easily become the biggest CPU hog in a containerized
| cluster. However it does what it says, and it does quite well.
| amirkdv wrote:
| Curious: what kind of load are you throwing at RabbitMQ?
| nesarkvechnep wrote:
| When I proposed to use RabbitMQ for service integration my
| colleagues didn't agree to go with it because it "won't scale"
| and AWS doesn't provide a managed service. Because of an AWS
| fetish and awful development experience the product won't launch
| anytime soon.
| pyepye wrote:
| AWS released the option to have a managed RabbitMQ broker under
| Amazon MQ back in November (2020) https://aws.amazon.com/about-
| aws/whats-new/2020/11/announcin...
| jpalomaki wrote:
| One thing to note is the pricing. Clustering is not supported
| with the micro instance, so if you need it, you must go with
| at least mq.m5.large [1] and you need three of them, so the
| total cost is around $620 per month (or closer to $700 for
| example in eu-central-1).
|
| [1] https://docs.aws.amazon.com/amazon-mq/latest/developer-
| guide...
| cratermoon wrote:
| At a previous employer I had what I thought was close to a slam
| dunk case for messaging with RabbitMQ. While I'm not the most
| experienced person in developing distributed messaging systems,
| and I expected technical objections, the disheartening
| response, and ultimate reason for rejection, was simply
| organizational inertia. In short, "you can't use RabbitMQ and
| messaging for that because we don't use messaging here". I'm
| sure there was some history that nobody mentioned or some lost
| institutional knowledge. The "grandma's ham" of systems design.
| zomglings wrote:
| I can sympathize with your colleagues. Developing the
| distributed messaging system is not where the bulk of the
| cost of running such a system goes - it's the maintenance and
| debugging that soak up time, money, and tears.
|
| While my past experiences with RabbitMQ in production have
| been stellar, I can see why a team would be hesitant to add
| this complexity to their infrastructure.
| datavirtue wrote:
| All of these problems sound like they come from developers
| who don't know how to develop distributed systems. If
| anything this should be simplifying an otherwise
| intractible problem. I have seen people overuse messaging
| and had to fix what I could. I otherwise reach for RMQ as a
| simple solution to horizontal scaling and I write the
| software to easily switch brokers (Aws SQS, on prem RMQ,
| cloud RMQ). It's not something that you choose to adopt and
| then force developers to make every interaction a
| distributed message.
| cratermoon wrote:
| Oh here's the thing though: the company was already using
| RabbitMQ, as well as Kafka (for other things, mostly data
| science), and it was all deployed in Kubernetes and
| replicated across three data centers, including one in
| Amsterdam. The complexity was already there, but only for a
| very restricted set of applications.
| KptMarchewa wrote:
| Why not use SQS if you're bound to AWS requirement?
|
| I've found RabbitMQ difficult operationally, and full of
| footguns (that can make you lose data), so I'm not sure why
| would you want to use it if you don't already have to.
| dsign wrote:
| Not SQS, but Google PubSub: everything was working allright,
| until one day messages started arriving with horrible latency
| and vital parts of our service went down. We spent close to
| two weeks debugging our infrastructure, because _sure it
| could not be PubSub_. Well, _it was_. I guess nothing too
| big, a small bit or tweak that changed in their service that
| didn't propagate to the client library we were using, or god
| knows what. Now we run RabbitMQ for everything; at least we
| control our versions and we can confidently blame ourselves
| when something happens. But RabbitMQ, standalone and
| clustered, has served us well across different systems for
| about fifteen years.
___________________________________________________________________
(page generated 2021-05-17 23:02 UTC)