[HN Gopher] Show HN: SwimOS Rust - A framework for real-time str...
___________________________________________________________________
Show HN: SwimOS Rust - A framework for real-time streaming data
applications
Author : swimos
Score : 62 points
Date : 2024-07-23 12:22 UTC (5 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| swimos wrote:
| The Swim Rust SDK contains software framework for building
| stateful applications that can be interacted with via multiplexed
| streaming APIs. It is built on top of the Tokio asynchronous
| runtime and a Tokio runtime is required for any Swim application.
|
| Each application consists of some number of stateful agents, each
| of which runs as a separate Tokio task and can be individually
| addressed by a URI. An agent may have both public and private
| state which can either be held solely in memory or, optionally,
| in persistent storage. The public state of the agent consists of
| a number of lanes, analogous to a field in a record. There are
| multiple kinds of lanes that, for example, lanes containing
| single values and those containing a map of key-value pairs.
|
| The state of any lane can be observed by establishing a link to
| it (either from another agent instance or a dedicated client). A
| established link will push all updates to the state of that lane
| to the subscriber and will also allow the subscriber to request
| changes to the state (for lane kinds that support this). Links
| operate over a web-socket connection and are multiplexed, meaning
| that links to multiple lanes on the same host can share a single
| web-socket connection.
|
| There's a number of example applications available here:
| https://github.com/swimos/swim-rust/tree/main/example_apps If
| you're interested in getting started with it a developer guide is
| available here https://www.swimos.org/server/rust/developer-
| guide/ as well as reference documentation here
| https://www.swimos.org/server/rust/
|
| Previously on HN:
|
| - SwimOS: Distributed platform for building stateful, real-time
| streaming apps https://news.ycombinator.com/item?id=22920764
|
| - Real-time traffic light status in Palo Alto powered by swim.ai
| https://news.ycombinator.com/item?id=19234286
| 01HNNWZ0MV43FF wrote:
| Doesn't run on bare metal though?
| swimos wrote:
| It doesn't unfortunately but this is something that we have
| discussed internally about targeting. We would be interested
| to hear about your use-case for it!
| duped wrote:
| How is it real time if you are using tokio? What latency and
| deadline guarantees are you able to provide?
| filleokus wrote:
| "real time" as in figma multiplayer not as in control system
| for a car (I'm assuming, never used SwimOS)
| jitl wrote:
| The README tells me mechanically what swim does but even with the
| example it's quite abstract and I don't know why I would choose
| to build with swim, what it's similar to, or what it's especially
| good at.
|
| I think with frameworks you should make the value proposition
| very clear otherwise it's a tall ask for someone to adopt a way
| of building. Versus a library which is easy to drop into existing
| software to solve a specific, scoped task.
|
| I guess a bunch of what I'm looking for is discussed in the main
| repo of your project, but even there it's a bit heavy on the
| "what" and too light on the "why" for me. This is a deep
| framework. I work on a realtime reactive app and am currently
| building stateful services on Kafka + Typescript + SQLite, so I
| think I'm squarely in your target customer demographic.
|
| It's not until I found the website that the value proposition is
| clearly described. I encourage you to copy-paste the headline of
| your website to all the components of your framework, so it's
| easier for a casually interested person on HN to grok what it's
| all about.
|
| EDIT: there's like a 1% chance I actually do convert, since I
| realize I could run my existing Typescript business logic in a
| Rust agent via Deno... perhaps the scattered documentation hunt
| on a Sunday is good marketing strategy after all.
| the__alchemist wrote:
| I had a similar reaction, and assumed I'm not the target
| audience. (Including skimming the examples folder and not being
| sure where I would use one of them) I have a hard time
| understanding the purpose of these abstract libraries; I assume
| people who have had the problems they solve recognize their
| utility.
|
| As a reference point for rust libs I've worked with recently: I
| see things like EGUI: This clearly is a library that allows you
| to add a GUI to programs. Or Bio: This lets you read and write
| FASTA format DNA etc sequences, find matches in sequences etc.
| Bincode: Provides a "derive" that allows you to easily
| serialize data to and from binary formats. Or the various
| embedded infrastructure libs that provide high-level APIs for
| performing hardware options. (I/O, ADCs, send a packet over USB
| or a radio etc) By contrast to these and every other lib I've
| worked with, Swim is abstract.
|
| I start wondering if I'm too dumb or otherwise incapable of
| reasoning abstracting to understand the Async part of Rust's
| ecosystem.
| swimos wrote:
| Thank you for the feedback! We could definitely improve the
| documentation and we will expand on the "why?"; we will get an
| updated readme up that expands on this in more detail. Could
| you please tell me which part of the website clearly explained
| it to you such that you understood the value proposition?
| written-beyond wrote:
| I am someone who initially really struggled with
| understanding whatever the Kafka team thinks their
| explanation is. Even after working on streams and pub sub
| messaging I still struggle with their explanation.
|
| However, I completely resonated and understood what you guys
| have built. We actually built something very similar but it
| was over websockets with protobuf3 just to emulate this exact
| behaviour. We we're using it for IoT-esque devices which send
| their data constantly but we needed to keep track and update
| certain fields when devices would connect and disconnect.
|
| We'll be open sourcing our implementation soon. I will
| happily want to rebuild it with this, and maybe add a
| sprinkle of protobuf3 to make sure messages are type safe.
| benatkin wrote:
| I think the examples could answer that question. They're linked
| at the end of the explanation in the README. However, I think
| each example ought to have a README.
|
| Here's the first one I looked at. I'm wondering when to use an
| event instead of a command. https://github.com/swimos/swim-
| rust/tree/main/example_apps/v...
| swimos wrote:
| Many of the example applications are corresponding code to
| our developer website which contains guides for them:
| https://www.swimos.org/server/rust/ We will get this updated
| as it isn't clear
| nine_k wrote:
| Having to study an example before getting a clear enough idea
| what to look for definitely narrows down the potential
| audience. Examples are very useful and important, but, unless
| they fit in 5-7 lines, they are the second step.
|
| The first step should be a few bullet points listing the key
| reasons behind the framework and the key benefits it
| provides.
| worldmerge wrote:
| This is super cool!! Back many years ago now I had a project that
| dealt with realtime data + geofences so I used tile38. I'm super
| curious to check this out!
| swimos wrote:
| We hope that it goes well and we would love to hear your
| feedback!
| alexmorley wrote:
| Maybe stating the obvious but took me an embarrassingly long time
| to clock this:
|
| SwimOS is an existing streaming framework (Java-based) targeted
| at full stack / app development.
|
| This announcement is a new Rust SDK for the above - not a new
| framework.
| swimos wrote:
| That is correct. We're working towards replacing our Java
| implementation with the Rust version and then provide bindings
| to Java
| qwerty456127 wrote:
| How does this compare to Erlang?
| thoughtpoet wrote:
| I've worked with Erlang/OTP in the past, although it's been a
| while since I last used it. I have fond memories of working
| with it, and I appreciate its unique strengths. One notable
| similarity between Erlang and SwimOS is the actor model, but
| their use cases are quite different. Erlang excels at fault
| tolerance, robust supervision, and lightweight isolated
| processes, with a focus on message passing (which, if I recall
| correctly, involves significant data copying underneath).
|
| In contrast, SwimOS is designed for high-performance streaming,
| boasting a heavily optimized WebSocket implementation that
| leverages multicast, multiplexing, and differential dataflow
| (delta encoding) to efficiently handle massive data volumes.
| SwimOS utilizes zero-cost abstractions and lock-free mechanisms
| to achieve its goals. Its primary use cases are data
| observability and real-time collaboration applications, with an
| emphasis on in-memory processing and recent data. This
| processing window is similar to a retention window in
| distributed log technology like Kafka, where the focus is on
| handling large volumes of data in real-time to enable timely
| decision-making, without necessarily storing all the data long-
| term.
|
| While both technologies share some similarities, their design
| and use cases make them suited for very different problem
| domains.
| VyseofArcadia wrote:
| Why do so many products that aren't an operating system calling
| themselves WhateverOS?
| Sunscratch wrote:
| Looks very similar to Akka, where agent is actor and Lifecycle is
| Behaviors.
|
| An interesting project for a complex task - streaming stateful
| processing.
___________________________________________________________________
(page generated 2024-07-28 23:04 UTC)