[HN Gopher] Rust Stream API visualized and exposed
       ___________________________________________________________________
        
       Rust Stream API visualized and exposed
        
       Author : PaulHoule
       Score  : 141 points
       Date   : 2024-04-25 12:51 UTC (10 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | withoutboats3 wrote:
       | The problem here isn't with the concept of Streams (which are
       | good) but with specifically the "buffered stream" APIs provided
       | by the futures crate (i.e. the buffered and buffer_unordered
       | methods). Their lack of concurrency with processing before or
       | after is a known problem as the blog post alludes to at the end;
       | I would discourage users from using these APIs without
       | considerable care.
       | 
       | I've explored this subject on my blog, including possible
       | solutions to the problem with these APIs:
       | 
       | https://without.boats/blog/futures-unordered/
       | 
       | https://without.boats/blog/poll-progress/
       | 
       | https://without.boats/blog/the-scoped-task-trilemma/
       | 
       | Also, the rendering and visualization aspect of this is very
       | cool!
        
         | gdcbe wrote:
         | Please keep on blogging like you do withoutboats, your articles
         | are a gem that I learn something new from every time.
         | 
         | Due to the work of you and others I do have hope it will all be
         | better in future.
         | 
         | That said, might be my low standards due to many scars from my
         | c++ background, but I'm already plenty happy with what we have
         | today, so the fact that it will get even better in the next
         | years is like cherry on the cake for me.
        
         | zamalek wrote:
         | Is there any reason why one couldn't?
         | stream::iter(0..10)           .map(async_work)
         | .map(|t| spawn(t))           .buffered(3 - 1) // The line above
         | act as a buffered slot           .map(unwrap_join)
         | .filter_map(async_predicate);
        
           | demurgos wrote:
           | The poll_progress post linked above explains the situation.
           | When polling the overall stream, you alternate between
           | awaiting in the buffered interface or in the subsequent
           | adapters. This is because the different futures are not peers
           | with regard to the executor, but there's a chain of futures
           | and `FilterMap` only calls `poll` on its parent when it's
           | done with the current item.
           | 
           | This post was also helpful to understand the issue:
           | https://tmandry.gitlab.io/blog/posts/for-await-buffered-
           | stre...
        
       | low_tech_punk wrote:
       | Good visualization is worth a thousand words! I wonder if Rust
       | stream can contain streams themselves, i.e. higher order streams
       | as in seen in RxJS? I found it very difficult to visualize
       | anything that is of higher order. The RxJS marble diagram was
       | helpful to some extent but they are static.
        
         | macawfish wrote:
         | Yes, higher order streams are possible in Rust. I appreciate
         | that in Rust they are also typed. In JavaScript it's sometimes
         | tricky to reason about higher order streams without types.
        
       | renewiltord wrote:
       | This is an incredible animation engine. I'm going to check it
       | out.
        
         | roland35 wrote:
         | Bevy is a full-blown game engine, which is an awesome idea for
         | visualizing rust programs. Maybe it would be good for
         | generating advent of code diagrams next year... (Who am I
         | kidding, I barely get to day 12 most years!)
        
       | 0x457 wrote:
       | So, buffer and buffer_unordered only make sense at the end of
       | stream and only if the receiving side is slower than the rest of
       | the "pipeline"?
        
       ___________________________________________________________________
       (page generated 2024-04-25 23:00 UTC)