[HN Gopher] Iggy.rs - building message streaming in Rust
___________________________________________________________________
Iggy.rs - building message streaming in Rust
Author : lukastyrychtr
Score : 221 points
Date : 2024-01-04 15:29 UTC (7 hours ago)
(HTM) web link (blog.iggy.rs)
(TXT) w3m dump (blog.iggy.rs)
| ramon156 wrote:
| Looks neat! Excited to play around with it when I get the time
| spetz wrote:
| Thank you, feel free to ping us anytime and/or join our open
| Discord!
| iddan wrote:
| Love the idea, and the blog post writing. The author really comes
| off humble, honest and a constructive project leader. Good luck!
| spetz wrote:
| Thank you, I'd like to mention that the team is really awesome,
| as they've decided to join me in my efforts, just to have some
| fun as well :)
| cwales95 wrote:
| Would love to give it a go at some point (but I'll first need to
| learn Rust). On a side note, I love the aesthetic of the site.
| cosmic_quanta wrote:
| The blog post mentions SDKs for other programming languages, so
| you may be able to use this without learning Rust
| spetz wrote:
| Exactly, we have multiple SDKs, and the blog uses Rust Zola
| engine :)
| bsaul wrote:
| How fast can one realistically expect to be able to build decent
| quality system-level rust code, provided the person is already
| familiar with C-style programming languages and has 0 rust
| experience ?
| ddoolin wrote:
| It has gotten a little easier over time. IMO, a month or two
| depending on your aptitude.
| spoiler wrote:
| If you have a _little_ bit of experience with any higher level
| language (eg Python, Haskell), then I 'd say fairly quicky
| (say, a week to a week and a half for the books and rustlings
| course), if you want an opinionated (and good enough, I
| recommend the book!) intro to production code theres the
| zero2prod Rust book. You can read the zero2prod book while
| working on one or two pet project (anything more complicated
| than a calculator would probably do) to help inform decisions.
| That should be another week or two, depending on how quickly
| you pace through the book.
|
| That should get you started to writing decent code, maybe not
| aways idiomatic or neatly structured. That might take a few
| more weeks or months (depending on your interest domains and
| their inherent complexities). But like with all skills, it's an
| endless journey; you'll keep getting better as you're
| increasingly more familiar with the tools.
| steveklabnik wrote:
| The closest thing I have to a non-anecdotal answer to this
| question is from Google:
| https://opensource.googleblog.com/2023/06/rust-fact-vs-ficti...
|
| > All survey participants are professional software developers
| (or a related field), employed at Google. While some of them
| had prior Rust experience (about 13%), most of them are coming
| from C/C++, Python, Java, Go, or Dart.
|
| > Based on our studies, more than 2/3 of respondents are
| confident in contributing to a Rust codebase within two months
| or less when learning Rust. Further, a third of respondents
| become as productive using Rust as other languages in two
| months or less. Within four months, that number increased to
| over 50%. Anecdotally, these ramp-up numbers are in line with
| the time we've seen for developers to adopt other languages,
| both inside and outside of Google.
|
| > Overall, we've seen no data to indicate that there is any
| productivity penalty for Rust relative to any other language
| these developers previously used at Google.
| spetz wrote:
| I had 0 Rust experience and close to 0 of system programming
| experience (except playing with some lower level communication
| protocols etc. but nothing fancy), but I'd say that something
| like message streaming isn't the system programming - it's
| close to building the database, but nowhere close to making
| direct usage of low level system APIs.
| bsaul wrote:
| Yes i didn't know exactly how to call it. Let's say <<
| middleware programming >>
| spetz wrote:
| That's a good name! :)
| emmanueloga_ wrote:
| Looks like a direct competitor to JetStream [1]? Impressive
| progress in less than a year of work!
|
| --
|
| 1: https://docs.nats.io/nats-concepts/jetstream
| spetz wrote:
| JetStream, Kafka, Redpanda, RabbitMQ Streams, Fluvio - quite a
| few message streaming solutions out there. Thanks!
| the_duke wrote:
| I'm not quite sure how this compares to Kafka and fluvio [1], a
| Kafka competitor also written in Rust?
|
| Is it more of a message queue like rabbitmq?
|
| [1] https://www.fluvio.io/
| spetz wrote:
| It's a message stream, so more like Kafka, Redpanda or RabbitMQ
| Streams (plugin). Fluvio is more mature (it's an actual product
| & company behind it), but we have our own ideas how to make
| Iggy a competitive message streaming solution :)
| esafak wrote:
| Doesn't Fluvio aim to replace both Flink and Kafka? I just
| heard about it so I am trying to understand.
| mattmmatthews wrote:
| i'm a bit confused as to what exactly this is; a low-level,
| messaging system? what are the implications of it and the
| comparison of competitors? I'm obviously missing something. Could
| someone elaborate?
| spetz wrote:
| This is a message streaming infrastructure, similar to Kafka,
| Redpadna etc. You can think of it as a kind of a simple
| database (so-called Write Ahead Log), being responsible for
| appending the messages in a highly performant manner.
| vlovich123 wrote:
| I'm surprised about the selection of monoio. AFAIK it requires
| the usage of the nightly compiler which I've found to not be a
| great choice when maintaining a project.
| steveklabnik wrote:
| It does require nightly, but it only uses five features, one of
| which could be removed by adding an external crate, and the
| others are like, not anything super wild, for the most part. I
| haven't looked too deeply into the code but they all make sense
| to me, for example one of them is a standard library API for
| creating uninitialized containers, which allows you to
| eliminate a copy.
|
| I have not compared monoio with glommio, which does run on
| stable. Would be interesting!
| vlovich123 wrote:
| It's not that the features are unreasonable but more that
| anything that uses it as a dependency also inherits the
| nightly requirement which makes things a bit more unstable
| because nightly is a floating version.
|
| I did compare with glommio a bit and disliked the file API in
| monoio but that's probably more a matter of taste.
| steveklabnik wrote:
| Yeah I hear you. I much prefer staying on stable for that
| reason. Sometimes (though much more rarely than non-Rust
| users seem to think) it is unavoidable though, and once you
| do go into nightly, you can kinda go two ways:
|
| * only use what is absolutely necessary
|
| * go wild and add tons of stuff
|
| As well as there being two kinds of nightly features:
|
| * stuff that's not stable but has a clear path to becoming
| stable on some time frame
|
| * stuff that's not stable but who knows when that will
| change, it may never
|
| And to me, this dependency feels kinda closer to choosing
| the first option from both lists, which is the most
| conservative way of using nightly.
| spetz wrote:
| As already mentioned, we've decived to use monoio, just
| to have something to begin with, it will take at least a
| few months to rewrite the core parts to make use of
| io_uring and thread-per-core approach (if feasible), so
| staying on the bleeding edge is more like a sandbox for
| us - we might decide to go with another runtime, like
| glommio, mfio or something else.
| steveklabnik wrote:
| Ah cool! Just to be clear, I think you should do whatever
| you feel is best for your project. I don't see this in
| any sort of moral terms, purely engineering ones, and
| there are good reasons to use nightly just like there are
| good reasons to use stable.
| spetz wrote:
| Monoio seems to be the most performant runtime, and actually
| easy to use - we have decided to go with "bleeding edge"
| approach, as it will still take at least a few months to
| implement io_uring and other optimizations, as we'll have to
| rewrite some of the core parts and most likely shift towards
| thread-per-core architecture.
| vlovich123 wrote:
| For what it's worth I don't believe it meaningfully
| outperforms glommio. There are some spots where it
| outperforms glommio but that's just a missing optimization
| that could be added. The biggest advantage monoio really has
| is that it currently has support for other platforms.
|
| Not saying it's the wrong choice, just highlighting the trade
| offs.
|
| I'm not sure what you mean by io_uring as glommio/monoio
| should be hiding the io_uring details behind the runtime.
| spetz wrote:
| Of course, they hide the details behind io_uring, it's just
| that monoio seemed even easier to work with. It's not set
| in stone, though, actually there are new runtimes being
| developed as we speak, for example mfio - we'll see what
| will be our final choice, but we've decided to start with
| monoio.
|
| Speaking of the other platforms, having the fallback to
| epoll or kqueue (they've even announced some Windows
| integration) is nice to have, however, at some point we
| might just purely focus on Linux development anyway
| (io_uring only), if there'd be any issues e.g. when it
| comes to the code design to provide the compatibility
| across multiple OS.
| jauntywundrkind wrote:
| It seems like such a sharp, smart move to start with QUIC.
|
| It provides really useful SCTP-like multi-streaming, which is a
| great base to start from. There's already so many good libraries
| the project or others can work from, and it'll just keep getting
| better (and better optimized, likely).
| https://github.com/xileteam/awesome-quic?tab=readme-ov-file#...
|
| This seems like such a natural place where of course having a
| somewhat better transport protocol than we've had is a huge win.
| Looking forward to the next decade of QUIC.
| spetz wrote:
| I've started with QUIC, as I wanted to try out something new.
| Still, TCP protocol we have in place is a bit faster than QUIC,
| but it might be due to the lack of some additional tuning or
| so. On a side note, QUIC on MacOS is slow, when compared to
| Linux.
| dochtman wrote:
| Is TCP + TLS still faster than QUIC, or do you mean TCP
| without encryption? (I'm assuming you're using Quinn with
| rustls, so encryption is always enabled.)
|
| I'm also curious about your macOS vs Linux numbers. As a
| Quinn maintainer, we're always happy to hear feedback
| including about our performance!
| spetz wrote:
| Right, I meant TCP without TLS, and when it comes to QUIC I
| only did benchmark using the "dangerous" mode (disabled
| local cert validation). I don't have the exact numbers, but
| AFAIR on MacOS raw TCP was a few times faster than QUIC,
| while on Linux (e.g. PopOS distro) TCP was maybe around
| 20-30% faster?
|
| Still, there are plenty of options to be configured in
| Quinn (buffer size, send/receive window etc.), so it might
| be that with some tweaking the performance would be similar
| or QUIC would be even faster than TCP - this is also one of
| the things that I'd like to spend more time with in the
| future. We expose most of these options in the server
| configuration file, so it's easy to adjust.
|
| Thank you for contributing to such a great library, really
| easy to work with :)
| amath wrote:
| These types of stories are what drove me to software in the first
| place. The ideal of a group of people working towards the same
| goal, for sometimes different reasons, but not for the sole
| purpose of monetary reward.
|
| Good luck with the project!
|
| I also echo some of the others, it would be great to see a sort
| of comparison with the alternatives so that we can better
| understand how the project fits in.
| spetz wrote:
| Thank you, that's exactly how it all started! :) At some point,
| we will try to incorporate some benchmarks and comparisons with
| the other tools.
| scxsky wrote:
| I keep my fingers crossed, the project looks great and
| interesting!
| spetz wrote:
| Thank you :)
| internet101010 wrote:
| Is this project kind of an amalgamation of Kafka and something
| like Flight/nats?
| spetz wrote:
| Can't say for sure, we share some of the same concepts that can
| be found in Kafka or RabbitMQ Streams, but at the same time,
| try to build something from the ground up and focus on low-
| level I/O improvements :)
| worik wrote:
| Definitely wins a prize for cute dog picture
|
| Love it
| spetz wrote:
| Thanks, I do agree, they're both lovely! :D
| thibauts wrote:
| Built something similar in Go with a friend a few years ago.
|
| https://github.com/thibauts/styx
| spetz wrote:
| Nice one, looks quite similar indeed! May I ask why did you
| discontinue the further work?
| lysecret wrote:
| Quite an aspirational read especially about how the team came to
| be. I love open source.
| spetz wrote:
| I did some dotnet OSS in the past, but there's something
| special about Rust community and I guess that if I'd picked
| another language instead, I wouldn't gather such an amazing
| team :)
| tizz98 wrote:
| Really cool idea and project! There are 2 things I would need to
| understand before trying it out though:
|
| 1. How can I run more than one instance of the server?
|
| 2. When running more than one instance of the server, how would
| the filesystem interaction work between servers?
___________________________________________________________________
(page generated 2024-01-04 23:00 UTC)