[HN Gopher] Turmoil, a framework for developing and testing dist...
___________________________________________________________________
Turmoil, a framework for developing and testing distributed systems
Author : zbentley
Score : 206 points
Date : 2023-08-17 15:28 UTC (7 hours ago)
(HTM) web link (tokio.rs)
(TXT) w3m dump (tokio.rs)
| merlincorey wrote:
| How does this compare to Jepsen?
| levzettelin wrote:
| Can someone explain what this does under the hood? Did find the
| text on their website very helpful. (Also, I don't know Rust)
| OJFord wrote:
| I also thought the example could do with fleshing out, but
| from:
|
| > Testing distributed systems is hard. Non-determinism is
| everywhere (network, time, threads, etc.)
|
| I assume that, at a very high level, what it does is: introduce
| random delay between components, give them slightly different
| fake times when they ask for system time, put them in different
| threads & perhaps introduce (additional) pre-emption etc., and
| anything else to simulate 'etc.'.
| jupp0r wrote:
| While deterministic execution for concurrent/distributed systems
| is great, there's also value in fuzzing the timing and order of
| asynchronous events to reveal race condition bugs in testing.
| outside1234 wrote:
| Very cool - the Tokio team just keeps cranking out good stuff!
| Thank you!!!
| danielvaughn wrote:
| This seems to be specifically for distributed systems _written in
| Rust_. Though it 's not entirely clear from the post so not sure?
| speedgoose wrote:
| It may also be some kind of small CloudSim alternative but in
| Rust instead of Java. I'm not sure.
|
| https://en.m.wikipedia.org/wiki/CloudSim
| mcches wrote:
| This is built on the tokio.rs ecosystem so it is Rust-specific.
| Dowwie wrote:
| > Turmoil is a framework for testing distributed systems [written
| in Rust]. It provides deterministic execution by running multiple
| concurrent hosts within a single thread. It introduces "hardship"
| into the system via changes in the simulated network. The network
| can be controlled manually or with a seeded rng.
| DylanSp wrote:
| Always good to see more examples of this sort of project; making
| it straightforward to test how systems behave with unreliable
| networks [1] should make building resilient systems easier.
|
| I came across Microsoft's Coyote project [2] a while back, which
| seems similar to this, though for C#. Has anyone here tried that
| out? If so, what have your experiences with it been like?
|
| [1] Also known as any network ever.
|
| [2] https://github.com/microsoft/coyote
| Fiahil wrote:
| See Also: Loom (https://github.com/tokio-rs/loom), a concurrency
| permutation testing tool.
| jeffreygoesto wrote:
| Used it to check lock- and wait-free algorithms. Great tool!
| One really tough bug however needed TLA+ to be found, running
| loom over multiple days was apparently not enough.
| hnjst wrote:
| Leslie didn't steal his Turing award ;)
| chc4 wrote:
| I used https://docs.rs/shuttle/latest/shuttle/ (which was
| directly inspired by Loom) for testing a concurrent garbage
| collector I wrote in Rust. It was really useful to find
| pathological thread orderings that broke GC invariants, and I'd
| definitely use a similar system for any concurrent or
| distribute code I'd write in the future - these types of
| exercising systems and deterministic reproducers are great.
| woile wrote:
| Nice library!
|
| I found myself struggling with this in python with asyncio, is
| there something similar in that space?
|
| Every time I wish I was using rust
| levzettelin wrote:
| With asyncio you can run multiple servers/clients/whatever in
| the same event-loop/thread/process already. Although, I don't
| understand what this library does at all, I don't think in the
| asyncio case you would need anything special. I've written
| tests for multiple distributed applications by using the
| asyncio approach in a relatively straightforward manner.
| pjmlp wrote:
| And me thinking it was a speccy game.
|
| https://www.wikipedia.org/wiki/Turmoil_(1984_video_game)
| mamurphy wrote:
| I, in turn, was reminded of the (quite fun, for 1 play-through)
| 2016 game about mining oil:
| https://en.wikipedia.org/wiki/Turmoil_(2016_video_game)
| sporksmith wrote:
| Cool, will be interested to see how this develops! tokio's loom
| framework has been a big help in testing some tricky concurrency
| code I've worked on.
|
| Folks interested in this space might also be interested in the
| system I spend most of my time working on: Shadow. It also
| performs deterministic simulation of a network of hosts, but it
| intercepts network and system interactions at the syscall level
| via seccomp. As such it can work with binaries compiled from ~any
| language, usually without any code modification or special
| compilation. https://shadow.github.io/
| tveita wrote:
| FoundationDB has published some informative content on how they
| test their actor based distributed system in C++, might be worth
| a watch for inspiration.
|
| https://www.youtube.com/watch?v=4fFDFbi3toc
| kaymanb wrote:
| I've been working on implementations of classic algorithms in
| distributed computing, and used Turmoil for testing correctness
| in message-passing / HTTP systems [0].
|
| Overall, my experience has been positive. When it works, it's
| great. A pattern I've been following is to have a single fixture
| that returns a simulation of the system under a standard
| configuration, for example N replicas of an atomic register, so
| that each test looks like: 1. Modify the simulation with
| something like`turmoil::hold("client", "replica-1")`. 2. Submit a
| request to the server. 3. Make an assertion about either the
| response, or the state of the simulation once the request has
| been made. For example, if only some replicas are faulty, the
| request should succeed, but if too many replicas are faulty, the
| request / simulation should time-out.
|
| One of the things I have found difficut is that when a test
| fails, it can be hard to tell if my code is wrong, or if I am
| using Turmoil incorrectly. I've had to do some deep-dives into
| the source in order to fully understand what happens, as the
| behavior sometimes doesn't line-up with my understanding of the
| documentation.
|
| [0] https://github.com/kaymanb/todc/tree/main/todc-net
| mcches wrote:
| That's great to hear that you've been using turmoil for this
| type of work. I'm one of the authors and we'd love to hear
| about your experience and what we can do improve things. Either
| a github issue or reaching out on discord works great.
|
| We've discussed improving the tracing experience, and even
| adding visualizations, but it hasn't been prioritized yet.
___________________________________________________________________
(page generated 2023-08-17 23:00 UTC)