[HN Gopher] Show HN: Walrus - a Kafka alternative written in Rust
___________________________________________________________________
Show HN: Walrus - a Kafka alternative written in Rust
Author : janicerk
Score : 126 points
Date : 2025-12-01 15:28 UTC (3 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| roncohen wrote:
| As someone who myself worked on a hobby-level Rust based Kafka
| alternative that used Raft for metadata coordination for ~8
| months: nice work!
|
| Wasn't immediately clear to me if the data-plane level
| replication also happens through Raft or something home-rolled?
| Getting consistency and reliability right with something home-
| rolled is challenging.
|
| Notes:
|
| - Would love to see it in an S3-backed mode, either entirely
| diskless like WarpStream or as tiered storage.
|
| - Love the simplified API. If possible, adding a Kafka compatible
| API interface is probably worth it to connect to the broader
| ecosystem.
|
| Best of luck!
| seanhunter wrote:
| It says on the github page " It provides
| fault-tolerant streaming with automatic leadership rotation,
| segment-based partitioning, and Raft consensus for metadata
| coordination."
|
| So I guess that's a "yes" to raft?
| zbentley wrote:
| GP asked about data plane consensus, not metadata/control
| plane.
| EdwardDiego wrote:
| They asked about data plane replication - e.g., leader ->
| followers. Unless I misunderstood them.
| nubskr wrote:
| Hi, the creator here, I think its a good idea to have S3 backed
| storage mode, its kinda tricky to do it for the 'active' block
| which we are currently writing to, but totally doable for
| historical data.
|
| Also about the kafka API, I tried to implement that earlier, I
| had a sort of `translation` layer for that earlier, but it gets
| pretty complicated to maintain that because kafka is offset
| based, while walrus is message based.
| EdwardDiego wrote:
| TBH I don't think anyone can utilise S3 for the active
| segment, I didn't dig into Warpstream too much, but I vaguely
| recall they only offloaded to S3 once the segment was rolled.
| zellyn wrote:
| The Developer Voices interview where Kris Jenkins talks to
| Ryan Worl is one of the best, and goes into a surprising
| amount of detail:
| https://www.youtube.com/watch?v=xgzmxe6cj6A
|
| tl;dr they write to s3 once every 250ms to save costs.
| IIRC, they contend that when you keep things organized by
| writing to different files for each topic, it's the Linux
| disk cache being clever that turns the tangle of disk block
| arrangement into a clean view per file. They wrote their
| own version of that, so they can cheaply checkpoint heavily
| interleaved chunks of data while their in-memory cache
| provides a clean per-topic view. I _think_ maybe they clean
| up later async, but my memory fails me.
|
| I don't know how BufStream works.
|
| The thing that really stuck with me from that interview is
| the 10x cost reduction you can get if you're willing and
| able to tolerate higher latency and increased complexity
| and use S3. Apparently they implemented that inside Datadog
| ("Labrador" I think?), and then did it again with
| WarpStream.
|
| I highly recommend the whole episode (and the whole
| podcast, really).
| nubskr wrote:
| s3 charges per 1,000 Update requests, not sure how it's
| sustainable to do it every 250ms tbh, especially in multi
| tenant mode where you can have thousands of 'active'
| blocks being written to
| zellyn wrote:
| Guess it beats doing it every 250ms for every topic...
| k_bx wrote:
| There's also Iggy https://github.com/apache/iggy
|
| Never tried it, but looks promising
| tormeh wrote:
| Looks like it has a solid amount of contributors. Exciting!
| Some other attempts like Fluvio seem to have lost momentum.
| spetz wrote:
| Thank you for the mention! BTW, we're currently working on VSR
| (Viewstamped Replication) to provide the proper clustering :)
| carverauto wrote:
| iggy is amazing
| oulipo2 wrote:
| Nice! How does it compare to Redpanda, NATS, etc?
| Zambyte wrote:
| Funnily enough I have been toying around with implementing a
| NATS compatible server using the new Zig Io interfaces
| (including the 0.16.0 preview) :)
| teleforce wrote:
| For Kafka alternative written in C++ there's Redpanda [1],[2].
|
| Redpanda claim of better performance but benchmarks showed no
| clear winner [3].
|
| It will be interesting to test them together on the performance
| benchmarks.
|
| I've got the feeling it's not due to programming language
| implementation of Scala/Java (Kafka), C++ (Redpanda) and Rust
| (Walrus).
|
| It's the very architecture of Kafka itself due to the notorious
| head of line problem (check the top most comments [4].
|
| [1] Redpanda - A Kafka-compatible streaming platform for mission-
| critical workloads (120 comments):
|
| https://news.ycombinator.com/item?id=25075739
|
| [2] Redpanda website:
|
| https://www.redpanda.com/
|
| [3] Kafka vs. Redpanda performance - do the claims add up? (141
| comments):
|
| https://news.ycombinator.com/item?id=35949771
|
| [4] What If We Could Rebuild Kafka from Scratch? (220 comments):
|
| https://news.ycombinator.com/item?id=43790420
| nubskr wrote:
| In the current benchmarks, I only have Kafka and rocksdb wal,
| will surely try to add redpanda there as well, curious how
| walrus would hold up against seastar based systems.
| chaotic-good wrote:
| I don't see any mentions of p99 latency in the benchmark
| results. Pushing gigabytes per second is not that difficult
| on modern hardware. Doing so with reasonable latency is
| what's challenging. Also, instead of using custom benchmarks
| it's better to just use the OMB (open-messaging benchmark).
| EdwardDiego wrote:
| > It's the very architecture of Kafka itself due to the
| notorious head of line problem
|
| Except a consumer can discard an unprocessable record? I'm not
| certain I understand how HOL applies to Kafka, but keen to
| learn more :)
| thinkharderdev wrote:
| > Except a consumer can discard an unprocessable record?
|
| It's not the unproccessable records that are the problem it
| is the records that are very slow to process (for whatever
| reason).
| drob518 wrote:
| Or it's I/O-bound.
| jeberle wrote:
| That was my first reaction. It's not like Java is terribly
| slow, so rewriting it in a slightly faster language seems
| like an empty exercise.
| pixelpoet wrote:
| Yeah but how is a C++ based project supposed to hit the HN
| frontpage?
| lionkor wrote:
| Fun anecdote; a couple years ago I started writing a Kafka
| alternative in C++ with a friend. I got pretty far, but abandoned
| the project.
|
| We called it `tuberculosis`, or `tube` for short; of course, that
| is what killed Kafka.
| sgt wrote:
| Imagine talking to your clients about tech stacks and "we're
| running tuberculosis" comes up... while people are dying from
| it.
| lionkor wrote:
| You just say "well, the alternative was Kafka" and they'd
| surely get it. Or not. Either way we imagined it to be
| hilarious.
| ramses0 wrote:
| t10s, pronounced "tios" or a stuttering "t- tents" on your
| geo. :-D
| kylecazar wrote:
| "Consumption" works too :)
|
| Assuming topics are consumed in your version, a la Kafka.
| gethly wrote:
| I never understood the popularity of Kafka. It's just a queue
| with persistent storage(ie. not in-memory queu with ram-size
| limited capacity) after all.
| sumtechguy wrote:
| Most of the other ones at the time it was pop and the data was
| gone. You had to jump thru some hoops to make it work as
| persistent. Not 'hard' but just more annoying. Kafka has that
| out of the box. Where kafka starts to come apart is how to set
| it up. Its configuration is a bit tedious to setup.
| mrkeen wrote:
| A queue with persistent storage is like a ledger whose entries
| don't vanish when you read them, or a git branch whose commits
| stick around for longer than 24-72 hours.
|
| It's popular because it didn't have any competition while it
| built up its ecosystem. And even though there's competitors
| now, I haven't had time to check them out, and they still brand
| themselves as "Kafka-alternatives".
| fareesh wrote:
| coo coo ca choo
| Barathkanna wrote:
| Walrus isn't trying to replace Kafka, but it does beat Kafka in a
| few narrow areas. It's a lightweight Rust-based distributed log
| with a fast WAL engine and modern I/O (io_uring), so the
| operational overhead is much lower than running a full Kafka
| stack. If you just want a simple, fast log without JVM tuning,
| controllers, or the entire Kafka ecosystem, Walrus is a lot
| easier to run. Kafka still wins on ecosystem, connectors, and
| massive scale, but Walrus is appealing for teams that want the
| core idea without the complexity. Really impressed by the
| direction here, great work!!.
| alexmorley wrote:
| IMO for those requirements Redpanda has been the go-to for the
| last 5 years or so but I agree this is still a nice take and
| potentially even lighter.
| ertucetin wrote:
| We need Rust alternative not written in Rust
| enether wrote:
| For Rust-based Kafka alternatives, I like Tansu[1]. It at least
| provides Kafka API parity, and critically also gives users a
| pluggable backend (embedded SQLite, S3 for low cost diskless type
| workloads and Postgres because just use Postgres)
|
| It's nice to try and out innovate Kafka, but I fear the network
| effect can't be beaten unless the alternative is 10x better.
|
| Something like Warpstream's architecture[2] had a shot at
| dethroning Kafka, but critically even they adopted the Kafka API.
| Sure enough, Apache Kafka introduced a competing feature[3]
| within two years of warpstreams launch too.
|
| [1] - https://github.com/tansu-io/tansu [2] -
| https://www.warpstream.com/ [3] -
| https://topicpartition.io/blog/kip-1150-diskless-topics-in-a...
| hexo wrote:
| Why. Just why. Rewrite for sake of rewrite. I'm clapping. It's
| enough.
| WD-42 wrote:
| Are those dips in the write throughput comparison for Kafka GC
| pauses?
___________________________________________________________________
(page generated 2025-12-04 23:01 UTC)