[HN Gopher] Kafka as an Antipattern
___________________________________________________________________
Kafka as an Antipattern
Author : joshaustintech
Score : 57 points
Date : 2023-09-25 17:35 UTC (5 hours ago)
(HTM) web link (joshaustin.tech)
(TXT) w3m dump (joshaustin.tech)
| aeturnum wrote:
| The way I think about this kind of problem is to remember that
| tools built to deal with huge scaling problems are generally
| dealing with a very complex set of variables. The tool is going
| to be designed to let you choose between all of those variables.
| There's no magic - just configuration whose complexity better
| matches that of your problem.
|
| That being said, if you are not yet in a situation as complex as
| the one your tool is designed to deal with, there is a very good
| chance you will waste some time starting to use such a tool
| "early." You might get that time back later when you scale, you
| might have the right people to set up the complex tool the right
| way for your simple situation, but you are taking a bit of a
| risk. As long as you go into the situation with your eyes open I
| think most people end up ok. The horror stories almost always
| come from people who are working to fulfill needs they do not
| have and don't understand why their work isn't giving good ROI.
| cupofjoakim wrote:
| The company I worked for when GDPR went into action used kafka in
| every configuration possible. I've seen some pretty decent uses
| but also some horrific ones.
|
| I agree with the author to an extent - kafka is overly complex
| for low traffic systems. It can however be a godsend in specific
| cases, but what those cases are can be hard to pinpoint.
| tda wrote:
| The main problem I have with Kafka is that their sales team is
| too good: at my previous employer the CIO was convinced we needed
| Kafka and bought a contract for sever 100k. But we already had
| all our events in a postgres database.
|
| Admittedly that database had some complicated queries with lots
| of business logic to get a useful view on the data. But at first
| I hoped Kafka would somehow make this easier, but of course our
| particular usecase with a low event volume (hundreds per day),
| high latency tolerance (next day reporting was considered good
| enough), highly complex business logic (various computations that
| required knowledge of what was done previously) all made Kafka
| just about the least suitable tool for the job.
|
| Of course the contract was already signed (I was naturally never
| consulted up front), so this resulted in lots of solution looking
| for a problem. No suitable problem was found so I ended up
| leaving enterprise world for a scale-up and the CIO is still
| doing whatever he wants for god knows why
| [deleted]
| davewritescode wrote:
| The anti-pattern here isn't Kafka, it's using Kafka for 7,500
| messages/day. Making your whole system asynchronous for that
| level of load is the textbook definition of over-engineering.
| andrewmutz wrote:
| It's not necessarily over-engineering because
| scaling/performance isn't the only reason to want to be doing
| things asynchronously. In the example given, they could want to
| be decoupling consumers and producers from each other.
|
| For example, they could have one service using CDC (his DB
| source connector) to propagate state out to a bunch of other
| systems and not know which systems are subscribing for changes.
|
| An organization could have many such systems propagating state
| out to many other systems, using a single distributed log
| system.
| pjscott wrote:
| Yes. For perspective, that's about one message every ten
| seconds.
| alexchamberlain wrote:
| Sorry for the naivety/not obvious from the comments: is that
| too much or too little? (I've used RabbitMQ much more than
| Kafka.)
| CookiesOnMyDesk wrote:
| You probably want to be closer to 7,500 messages/second
| before Kafka becomes worthwhile.
| SuperCuber wrote:
| too little
| Tade0 wrote:
| It's almost nothing.
|
| I'm currently maintaining a system that uses Redis as a
| message broker and at a rate of ~10 messages/second it
| marginally makes sense.
| _jal wrote:
| It is using a dump truck to sugar your coffee.
| x0x0 wrote:
| I've used kafka to process data on the order of 50mb to
| 5000mb / second incoming. It has complexities that are
| worth eating for that type of use case.
|
| For a message every 10 seconds, its use is ... hmm. It
| wouldn't be in my top 50 choices.
|
| edit: and to be clear, I'm a huge fan of kafka: it sat
| there and silently just worked. It was great!
| tetha wrote:
| At 7500 records a day, I'd even question RabbitMQ in the
| design.
|
| If I was to judge that at work, my first thought would be
| that one of our busier postgres clusters with 2 read
| replicas is chugging through some 2-3k transactions per
| second without really needing much tuning or rather
| specialized hardware. The more ETL-oriented clusters are
| capable of processing some 100M - 200M rows per second when
| chugging through large queries, and these are just simple 4
| core VMs again on not really specialized hardware. And
| postgres would parallelize these queries more if you gave
| it more cores and the queries aren't horrible.
|
| At 7500 records a day or 3 million a year, you wouldn't be
| able to generate enough data to make one of these databases
| sweat over many years.
|
| Hate me as a DBA, but write some good queries for whatever
| you're doing and run those in a cronjob at that scale.
| steve_adams_86 wrote:
| I once worked on a chat-based system that handled load like
| this, and it was initially built with kafka. I worked out
| that the cost per message was several cents, haha. I
| replaced it with a redis queue, which was all I knew at the
| time, and it ran on a digitalocean droplet for CAD $5 per
| month for around 18 months before they scaled it up. It was
| handling ~100 messages per second peak at the end, which is
| still very low. The cause for concern was that the droplet
| silently failed due to memory issues on a particularly busy
| day, so it seemed reasonable to jump to the next tier to
| avoid that issue for a while.
|
| For what it's worth I never intended for the 1CPU/1GB VM to
| go to production, but I was a consultant and they just ran
| with it. And it worked!
|
| They swore off of kafka forever after the pains they had
| with it. Another consultant built that system for them, so
| it wasn't an internal decision exactly and they had no idea
| what they were getting into. I've heard of similar
| experiences since. I've sometimes hoped to land on a
| project where kafka was well suited to the problem, though;
| I learned a lot about it back then and it seemed incredibly
| cool. I was kind of envious of all these projects fully
| utilizing it!
| yCombLinks wrote:
| Kafka is designed to maximize scalability, millions of
| messages a second. It's a pain in the neck to manage if you
| don't need it
| esafak wrote:
| Too little to need Kafka.
| throw1234651234 wrote:
| Way too little to justify event-driven architecture (let
| alone Kafka specifically), unless you have some specialized
| need like very slow event processing and need to display a
| "message received" notification to user before the
| processing happens. Or you really need the retry
| functionality and can't handle it some other ways.
|
| Most businesses have no (hue hue) business doing event
| driven architecture. There is way too much overhead for
| local testing and overall complexity, especially when you
| want to properly handle errors.
|
| "But, every developer should be able to set up their
| local." Yea, great, explain to the manual QA who may be
| amazing but just started 3 months ago.
| alexchamberlain wrote:
| I think we need to separate load from event driven
| architecture: what if you want to reload data on a
| client's view, even if that data only gets refreshed once
| a month? The load is very low (1 message/month), but
| still requires an event to be pushed to the client to
| refresh their data.
| cbsmith wrote:
| Sometimes you do want it to be asynchronous anyway, it just
| doesn't make sense to make it asychronous with a horizontally
| scalable distributed computing streaming platform...
| Philip-J-Fry wrote:
| Exactly, handled way higher loads than that with a basic table
| and a poller.
| lallysingh wrote:
| You need _something_ , but the premise of adding Kafka for this
| load is pretty weak. Use something you already have, because
| nearly anything will work.
|
| Ex: - Your filesystem. - Someone else's
| filesystem (e.g. s3). - Your database. - REDIS.
| WaitWaitWha wrote:
| For those who came here for Franz Kafka, this article and posts
| are about Apache's "distributed event store and stream-processing
| platform".
|
| https://en.wikipedia.org/wiki/Apache_Kafka
| bob88jg wrote:
| Amazing reply
| SanderNL wrote:
| 8000 messages a day, tops? That's 5 a minute. Does that warrant
| "infrastructure"? I think a gameboy's Z80 could handle that load.
|
| I don't want to be dismissive, but I often see these big numbers
| being posted, like "14M messages" or "thousands of messages" and
| then adding "per year" or something, which brings it down to toy
| level load.
|
| Even the first "serious" example is about "thousands of messages"
| per minute. Say 5K a minute. That's 83 per second, say 100. That
| seems .. not that interesting?
|
| Am I being too dismissive? I think I am. I am not seeing
| something right. Can anybody say something to widen my
| perspective?
| wayfinder wrote:
| Yeah that's nothing.
|
| My busy discussion forum built in PHP running on a toaster of a
| server was handling way more load than that 15 years ago.
| baz00 wrote:
| I think you're probably being slightly dismissive. It's not
| necessarily about load but various other concerns like
| durability, delivery latency and how failures are handled.
| There's a big difference between messaging and reliable
| messaging. I have messaging systems that take 10-20 messages a
| day but _must_ deliver those messages and do it on a deadline.
| For that you do need infrastructure (and no that isn 't a queue
| inside a SQL database).
| slt2021 wrote:
| you don't need "systems" for 10-20 messages a day. it all
| could be replaced with S3 buckets and aws-cli with even
| better durability and delivery latency and error handling
| than anything you would be able to engineer yourself
| anthonyskipper wrote:
| Your point is good, but that stack wouldn't win any latency
| awards. Many of the people I know using kafka need
| latencies in the millisecond range.
| slt2021 wrote:
| kafka is not for latency, it is used for high throughput.
| By design kafka shines in high throughput workload due to
| consumer and producer concurrency (consumer group),
| broker concurrency (multiple nodes and partitions).
|
| for latency sensitive you will probably need redis
| pub/sub or something in-memory
| KaiserPro wrote:
| but kafka isn't fast. Most things are backed by real
| files, so when you hit limits or something ejected from
| cache, it gets slow real fast.
|
| Kafka isn't the right choice for most things.
|
| SQS, MQTT, NATS, rabbit if you're wanting a lot of admin
| are all better (plus the crap that azure and google make)
| baz00 wrote:
| I am completely dumbfounded by this reply.
|
| You're suggesting we engineer something on S3 and aws-cli,
| while complaining about engineering something ourselves
| when AWS offers a perfectly good queue service that
| requires no engineering?
|
| Uff. I'm going to buy a hut in the woods and live in it.
| slt2021 wrote:
| I used s3 just as an example of a service with very good
| track record of availability for a very low cost.
|
| perfectly fine with using SQS, just it will have worse
| availability guarantees than S3 - people should
| understand tradeoffs
| The_Colonel wrote:
| Then you have dependency on a specific proprietary API /
| technology available from a single company. Doesn't look
| like a good trade off.
| slt2021 wrote:
| S3 API has become lingua franca and is supported by open
| source (minio), storage vendors (QNAP), as well as
| plugins that translate S3 API calls to APIs for competing
| cloud providers (s3proxy).
|
| all this is done because S3 provides unmatched durability
| and reliability at a dirt cheap cost of
| $22/terabyte/month of storage (with the first 50Tb/mo
| free!).
|
| Try to beat that reliability guarantees with whatever you
| handrolled, and I bet you will never be able to beat the
| cost of S3, even match the durability, reliability,
| availability guarantees at any reasonable cost at all
|
| from https://aws.amazon.com/s3/storage-classes/:
| Key Features: Low latency and high throughput
| performance Designed for durability of
| 99.999999999% of objects across multiple Availability
| Zones
|
| have you ever built anything with 11 nines? (as in eleven
| nines)
| herbstein wrote:
| > S3 API has become lingua franca
|
| S3 API support sounds great until your costumer builds a
| system with an "S3 compatible object storage" product.
| Soon you discover that many "S3 compatible" solutions
| aren't actually that compatible when pushed.
| baz00 wrote:
| S3 is fine until you want your data to leave AWS.
|
| Then it costs $92 / TB to get it out again.
|
| Also S3 has durability guarantees but it's very difficult
| to do a durable transactional write to S3. Try it a few
| million times and see. The API is a defacto shitty
| standard.
|
| These two facts are rather interesting when it comes to
| doing a restore from your supposed backup or wonder why
| consistency guarantees between external metadata services
| (DB) and what is in S3 don't always line up.
| slt2021 wrote:
| and why would you ever take raw data out of AWS ?
|
| if it is for migration: it is one time cost that anyone
| can swallow easily if they decided to leave AWS for
| something else.
|
| If your data is worth < $94/tb - it is really not worth
| pulling it out of AWS. Just let it sit there.
| rewmie wrote:
| > 8000 messages a day, tops? That's 5 a minute. Does that
| warrant "infrastructure"? I think a gameboy's Z80 could handle
| that load.
|
| The blog post is quite clear in stating that their pain points
| had nothing to do with scaling or throughput. The author
| explicitly mentions idempotency, custom headers, and
| authentication.
|
| I think you're ranting about a strawman you put up.
| jshen wrote:
| The author's talked about their pain points, but they didn't
| really outline the downside of going to this approach in a
| comprehensive way. What I've found with event based systems
| is that they are much worse if your operational
| maturity/excellence is low. To make it worse, operational
| excellence can get worse over the years so you can't simply
| base the decision on how well you do it today.
| SanderNL wrote:
| I think you might be right. I'll take the hit.
|
| Still, I think talking about Kafka without load feels like
| Christmas without tree. I thought that was the main point of
| it. Handling massive loads (by distributing them).
| rewmie wrote:
| I think that the author just leveraged the existing
| messaging infrastructure they were already using for other
| services, and also reused their know-how. Their new
| microservice will barely register in the overall traffic
| volume, but they don't need to either deploy dedicated
| infrastructure or onboard onto yet another technology just
| to have message listeners.
| Zandikar wrote:
| FTA's conclusion: "If you are handling thousands of messages a
| day, a simple database-driven queue might be better than
| Kafka."
|
| They're not trying to say "thousands of messages a day" is a
| lot, but rather not. Or at the very least, they're saying that
| at that scale, it is not significant enough to merit the
| complexity they were dealing with.
| wolfi1 wrote:
| ok, pretty embarrassing I thought of Kafka the author first and
| wondered why he would have been an antipattern ...
| jcims wrote:
| I've worked at two large companies now with a mature managed
| Kafka offerings. The 'platform' engineering team handles all of
| the engineering, implementation, security and compliance,
| upgrades, observability etc. and have self-service onboarding
| with lots of recipes and sample integrations. My team moves about
| 5B messages a day through two topics and we're not putting a dent
| in the overall volume. It just enables use to move so much more
| quickly than we would if we had to deal with all of that
| ourselves.
|
| So in our case it's clearly not an anti-pattern, but the right
| tool for the job.
| browningstreet wrote:
| Agreed -- have used it in a bank. It was very suited for that.
| jpgvm wrote:
| Anything is deceptively deep if you understanding never goes
| beyond skin deep.
|
| Also Avro is great but like Kafka you were probably holding it
| wrong.
|
| I do prefer Protobuf in these particular scenarios as Protobufs
| features more closely align with svc <-> svc RPC style
| communication patterns while Avro shines in longer lived
| scenarios where messages need to be archived and you don't want
| to come up with your own framing for your Protobufs.
|
| This is because Avro has the Avro Object Container Format which
| is a simple block based file format, which allows for relatively
| efficient seeking, block based compression etc. Protobuf
| unfortunately doesn't define any standard file formats or even
| wire protocol framing. If you need to do more than simply store
| and scan/read in bulk you might want to use Parquet instead
| though.
|
| Reading this blog post was probably a waste of my time, hopefully
| this comment actually helps someone though.
| sdfghswe wrote:
| Deceptively deep topic, and yet such a superficial post.
| chrsig wrote:
| It seems a lot of the complaints weren't about kafka itself, but
| rather seemed to stem from internal communication problems.
| Custom kafka message headers could very well be custom http
| headers, and the problem is the same. Kafka is just coincidental.
|
| Looking at the volume though, kafka is overkill. They most likely
| could have just used the database and reaped the benefits of
| doing everything in a single transaction, with easier row level
| locking. The post acknowledges this.
|
| I do think it highlights the need for a small scale kafka,
| though. It's conceptually great to have everything work off of
| logs, but kafka does add a non trivial operational burden.
| iamwpj wrote:
| I used bash and netcat for a queue like this once. I stashed
| the on disk if the database was down and read them back if far
| end was down.
|
| The think Kafka really brings to the table is trusting the pipe
| -- in a case where writing to disk queues would occur often
| enough, I would run into reliability building my own system,
| instead Kafka handles that type of indexing.
| slowmovintarget wrote:
| We've had "small-scale" Kafka for a long time. It's an append-
| only log, and there are a number of ways to implement it, but
| it's essentially that.
|
| The thing that makes Kafka interesting is the technique of
| operating from a linux disk write-buffer. That's the trick that
| makes it fast and scale to huge volumes. But if you don't have
| the scale, you can stand up a table, or RabbitMQ, or anything
| that manages append-only ordered log entries. There doesn't
| need to be a new thing... Kafka was the new thing.
| chrsig wrote:
| Yes, there's nothing novel about an append only log. What's
| missing (or unbeknownst to me) is a library or small server
| that provides a good general purpose implementation.
|
| It's not just a matter of "write to log, done!". Ensuring
| persistence, keeping track of consumer offsets, transparent
| compression, waking up consumers on new message availability,
| support for transactions...
|
| It's not just _a_ append only log that 's wanted, it's a
| system for managing append only logs, without the
| complications like leadership election, replication,
| partitioning, etc.
| rewmie wrote:
| > Looking at the volume though, kafka is overkill.
|
| Overkill in what sense? The blog post seems to suggest Kafka
| was already pervasive in their organization, and that they
| leveraged the existing infrastructure and simply added.a couple
| of topics. How is this overkill?
| imachine1980_ wrote:
| >small scale kafka, though. It's conceptually great to have
| everything work off of logs, but kafka does add a non trivial
| operational burden.
|
| does something like that exist ???
| chrsig wrote:
| Not that I'm aware of. I've been very tempted to write my
| own.
| ssrc wrote:
| Depending on the meaning of "small-scale kafka", both
| RabbitMQ and redis do support streams.
| chrsig wrote:
| One of my desires would be for it to be persistent.
| Hopefully with the option of different storage tiers, so as
| logs became older they could be moved to less costly medium
| and transparently fetched when requested.
|
| Having an event sourced system doesn't make much sense
| unless you maintain messages from the start of the system.
| You can snapshot state and resume in order to quickly
| rebuild from a known good state. That doesn't help if there
| was a logic error corrupting every state from the start,
| and a full rebuild is required.
|
| I'm unsure how redis streams behave with regard to cache
| eviction, nor am I familiar enough with rabbitmq to comment
| on it's behavior. It's been 10 years since I used either,
| and at the time neither were good solutions for a log based
| system.
| mu53 wrote:
| I think it'd be very easy to write your own. I used postgres
| subscribe/listen built in combined with a database table to
| get a distributed message system.
|
| Writing a distributed, scalable system is really hard, and
| beyond the API, that is the real value for kafka
| local_crmdgeon wrote:
| >I used postgres subscribe/listen built in combined with a
| database table to get a distributed message system.
|
| Every single person I know who's done this says it was a
| fantastic decision, and the "eventually I'll have to
| migrate to X" never came.
| chrsig wrote:
| It's relatively easy -- removing any networking
| requirements drastically simplifies the problem. There's
| still some non-trivial bits that vary depending on
| granularity for concurrency.
|
| It's a weekend project to demonstrate the concept, maybe a
| few weeks to really flesh it out and iron out quirks. I
| imagine if you're willing to use sqlite as a backend for
| persistence, it gets a bit easier.
| phendrenad2 wrote:
| Kafka (or, broadly, distributed systems) are only an anti-pattern
| if you embrace the design model when you don't need to. Then it's
| hell. But considering that most tech teams are run top-down by a
| lone senior engineer who has barely heard about RabbitMQ before
| falling onto the Kafka bandwagon, best of luck to ya'll.
| no_wizard wrote:
| Seems like a lot of what I read about Kafka really makes it sound
| like using it is quite, well, _Kafkaesque_
|
| Why do so many engineers end up having such a struggle with an
| event sourcing system, yet the system itself remains highly
| popular I don't know. I theorize the following:
|
| - Its flexible enough to do things like receive events (messages)
| and sending downstream events from those received
|
| - it can ingest events fast. A well tuned instance is very fast
| and can handle a lot of volume
|
| - it often allows a middleware log point (or other work types)
| for things happening throughout your whole system
|
| Perhaps all of these things (and more) are hard to attain using a
| different technology
| marcinzm wrote:
| The simple approach mentioned in the article gets annoying if
| you have micro-services that don't share a DB. You could add a
| shared DB or a nosql DB but then you may as well just add
| Kafka. Of course the key question then shouldn't be kafka or
| not-kafka but if you over engineered on micro-services.
| quickthrower2 wrote:
| - developer tries to get those jobs paying $50k/y more "what
| kafka, event sourcing and microservices experience do you have"
| mrkeen wrote:
| > Why do so many engineers end up having such a struggle with
| an event sourcing system, yet the system itself remains highly
| popular I don't know.
|
| It's the mental model that's simple: some services write down
| what happened, other services 'do their own thing' with that
| information.
|
| I can write the core of the system in 2022, with events like
| 'Joe wants to buy a bike', 'Joe owes us $200', 'Joe paid us
| $200', 'Send Joe the bike', etc.
|
| In 2023 I want to build the book-keeping service, in 2024 I
| want to build the inventory management system, and in 2025 I
| want to hook it up to a CRM and see if we can try to sell some
| bike parts to Joe.
|
| Why couldn't I just use REST for that? Because the recipients
| of the rest calls didn't exist yet.
|
| The bad part of Kafka (in my opinion) is how opinionated the
| consumer logic is (oftentimes by necessity, because of the
| whole distributed system thing). Sometimes I just wanna ask
| "what offset are you up to?", but end up in API hell, and am
| unable to do it.
| convolvatron wrote:
| I agree that Kafka is alot of machinery for a fairly limited
| gain. but I really dislike this whole notion of 'antipattern', as
| if we can look at thing and assign a decontextualized thumbs up
| or down. that building systems is just a matter of assembling the
| right patterns, and avoiding the antipatterns.
| BoorishBears wrote:
| Sometimes a technology or pattern can be poisonous in a very
| specific way that warrants a label: when they're most alluring
| to those least equipped to leverage them.
|
| Just like microservices, you cross the activation energy to
| want Kafka very easily because it's appealing on resumes,
| sounds like a hedge against scale,etc.
|
| But there's a huge asymmetry in understanding the drawbacks to
| them. When you spin up these systems, the drawbacks don't hit
| you immediately, it feels like they're solving the problem you
| had, and it's not until you've invested immense amounts of
| sweat capital (and literal capital) that you discover how badly
| you screwed up.
|
| You need some way to match that low effort value prop with a
| low friction warning: this is not a panacea for your problems.
| It only seems simple, it's not simple, it will hurt you unless
| know it will hurt you and simply have the resources and scale
| to play through that hurt.
|
| --
|
| To me that warning is what's implied by "antipattern", it's not
| never use this, it's never use this _unless you know why you
| should never use this_.
| jshen wrote:
| I agree completely. Kafka, and event based architectures, are
| highly overused, but are also the right thing sometimes. It's
| FAR easier to manage an architecture where systems call into
| the source of truth for a given piece of data via APIs, and you
| should only switch to an event model if you truly need to.
| [deleted]
| tgma wrote:
| This was not explicitly addressed in the post, but the big "Kafka
| antipattern" out there is building "microservice infrastructure"
| and using a stateful message broker between services where you
| should be using RPC/look-aside load balancing with deadlines and
| retries.
|
| Some morons even write books and blog posts about this. The funny
| thing is this sort of shit is done in the name of scale, but the
| big folks never operate this way. Large scale infrastructures
| actively disdain keeping buffers and state in the middle of the
| request flow. They cannot afford the cost and latency of such
| systems. They do it the sane way[1].
|
| [1] https://www.usenix.org/conference/osdi23/presentation/saokar
| waffletower wrote:
| Curious what language the OP and their team was using to
| integrate with Avro. Binary serialization can be a bit awkward,
| but Avro is a very stable API and it isn't difficult to find/ and
| or build abstractions to work with it. Perhaps I am spoiled
| coming from a Clojure perspective?
| tetha wrote:
| This is what I've been starting to think about on a more abstract
| level: Introducing a new technology, a new system into a design
| isn't like putting a piece into a jigsaw puzzle, or even worse,
| trying to mold and force the system to fit whatever hole your
| design has. Many more specialized systems - and Kafka is one of
| them - should solve some problem, but they should also change
| your mental model of the system and you should look for the
| easiest way to introduce these heavy hitters.
|
| For example, if you use Kafka or streaming solutions like Flink
| or Spark, you should change your mental model to (possibly
| large), (possibly resplayable) streams of events and look for
| simple ways to get these event streams going and good ways to
| consume them. And then you need to let the design push you where
| it wants you to go.
|
| Like, at work, we recently had a discussion how it was so
| storage-expensive for a project to store all events of a day and
| how the query to count all of these events per tenant was taking
| so long. While they are using a streaming event processor in
| front of it. Like, what the hell - think in streams, tally up
| these events on the fly and persist that every hour?
| bosky101 wrote:
| Immediately starts to doubt OP's assumption/implementation of the
| "where it works great"
|
| Jokes aside, agree with others. For the 7500/day, I would just
| push these into an S3/minio folder. And then dequeue 100 or N
| once every 10/30/60/T seconds. play around for the right N,T.
|
| Then again am sure there maybe reasons/context/constraints
| unaware to us.
|
| Eg - the ingestion is spiky, with the possibility of all 7500 in
| a few secs/minute, you would want to first make sure that the
| http traffic can scale before getting to the point where it can
| actually connect and push to the queue
|
| Possible reason#2 - an intern who just finished up their first
| Kafka task; just got freed up
|
| #3 - or this was the only infra available and a choice had to be
| made with the time available at hand
|
| #4 - or this was an experiment to see for yourself
|
| A majority may not agree with your view, but none of us really
| are in your shoes. So I applaud you for sharing your thoughts
| anyway.
| opportune wrote:
| Oh, I've seen much worse than this. I truly believe system design
| interviews and Confluent marketing/sales have made Kafka a midwit
| trap:
|
| 1. You cannot just use Kafka for free. It will take dev time to
| set up itself, dev time to code sources and sinks, dev time to
| handle commonly glossed over but utterly important details like
| idempotency, retries, duplicate messages, consumed-but-not-
| committed (or whatever the term is in Kafka world) network
| interruptions or restarts of consumers.
|
| 2. You cannot just continue to use Kafka for free. Running it has
| an operational cost; this is mitigated using it as a PAAS but not
| fully solved as you'll still need to twiddle configurations and
| scramble to deal with things like "We had no idea we'd need to
| handle idempotency or use deadletter queues, fixed it, but now
| need to deal with old data before our fix".
|
| 3. There are many ways you can implement async producer:consumer
| patterns that are less complex and less costly than Kafka. For
| example, you can write data to S3. Or you can store records in a
| regular RDBMS. Kafka isn't worth it unless you really need "real-
| time" ingestion but can't/won't/shouldn't implement an actually-
| real time (synchronous) system instead, like if you get large
| spikes and are ok with ingestion going from O(seconds) to
| O(minutes) when that happens.
|
| 4. There's a good chance you don't need an async queue at all. If
| consumers can horizontally scale quickly (like with Lambda) why
| not synchronously invoke them over HTTP/RPC and only use async
| queues (or a file, etc) for messages that fail multiple retries?
| Since external users usually aren't directly writing to your
| Kafka topic, and thus you have a degree control over your
| ingestion and consumption, why not just combine the two services
| (since you can experience data loss from external world to
| ingestion service anyway, and in fact this is a pretty likely
| source of failures, your queue may not even be solving the
| problem you think it is). If you don't need ordering or partition
| and are using queues for eg config update propagation, why not
| just synchronously update consumers or implement basic polling in
| your consumers?
|
| 5. A lot of Kafka/Confluent "features" like retries or logging
| you can get with so many other tools and services but for some
| reason these can be the actual selling point more so than the
| fact it's an async queue (that also has these features).
|
| Yes, in a FAANG design interview where it only costs you 5seconds
| to say "and we'll use an async queue like Kafka between these
| components to handle variable load and partially consumed data"
| it's a great tool that saves you a lot of time. And when you
| pretend integration and maintenance costs don't exist, and don't
| even know what idempotency means, and are sitting across from
| some slick Confluent salesperson telling you Kafka can be THE
| database of everything your company does with all these nice
| features, it sounds great to midwit managers and hasbeen
| architecture astronauts. In reality? The dumb unsexy alternatives
| probably solve your actual problem more simply
| slt2021 wrote:
| from your reply it seems like you have not worked with high
| throughput workloads that FAANG deals with daily.
|
| your suggestion of single node rdbms as a replacement to kafka
| suggest you dont have experience with workloads that cannot be
| served by a single machine, yet you still need a single
| architecture.
|
| agree that Confluent took a gread product that works for high
| load use case, and then tries to shove it to each and every
| average Fortune1000 enterprise use case with 100 users and
| traffic that could be well served by SQLite/Postgres on a
| single machine
___________________________________________________________________
(page generated 2023-09-25 23:02 UTC)