[HN Gopher] Show HN: Lfi - a lazy functional sync, async, and co...
___________________________________________________________________
Show HN: Lfi - a lazy functional sync, async, and concurrent
iteration library
Hey HN! Roughly 4 years ago I started building a lazy functional
iteration library for JS/TS. I had a few goals for the library: -
Supporting sync, sequential async, and concurrent async iteration
- Limiting it to a small number of orthogonal concepts that compose
beautifully to solve problems - Making it fully tree-shakeable I
built it for myself and have (mostly) been its only user as I
refined it. I've used it in lots of personal projects and really
enjoyed it. I recently decided it would be nice to spread that
enjoyment so I created a documentation website complete with a
playground where you can try out the library. I hope you enjoy
using it as much as I do! Looking forward to hearing your thoughts
:)
Author : tomeraberbach
Score : 91 points
Date : 2024-12-11 20:07 UTC (2 days ago)
(HTM) web link (lfi.dev)
(TXT) w3m dump (lfi.dev)
| deanebarker wrote:
| Call me an idiot, but nowhere on the home page does it say it's
| for JavaScript. The code samples looked like JS, but I don't know
| every language, so I wasn't sure. It's not until I got into the
| docs and saw "npm" was a sure we were talking about JS...
| Chris2048 wrote:
| should probably say "JS" in the title too - there are all sorts
| of devs on HN afterall
| ComputerGuru wrote:
| I came to comment the same thing, having clicked through from
| the link and not seen the OP's text. In my case, I searched for
| and found the GitHub link at the bottom of the landing page and
| saw the language in the repo details.
|
| Please add "for JavaScript" to the title!
| egorfine wrote:
| > nowhere on the home page does it say it's for JavaScript
|
| Yeah, and it is not mentioned in the post title. It's like JS
| is considered the default programming language.
| tomeraberbach wrote:
| That's fair! Agreed it would be good to mention on the home
| page :)
|
| I did mention in the HN post description, but probably should
| also say so in the title
| tomeraberbach wrote:
| Actually, looks like HN doesn't allow me to update the HN
| title at this point. Sorry about that
| akdor1154 wrote:
| Was gonna post some snark about assumed languages, countries,
| etc, now i think there could be an interesting post in that..
| revskill wrote:
| Should be renamed to lfi.js
| VeejayRampay wrote:
| it says "lfi is a lazy functional sync, async, and concurrent
| iteration library for JavaScript and TypeScript" when you click
| on the link though
|
| has it been added since?
| tomeraberbach wrote:
| Yup, I added it in response to the feedback :)
| WorldMaker wrote:
| Were you aware of IxJS [0]? I've used that to good success over
| the years. It's relationship to RxJS keeps it familiar.
|
| Any thoughts on what Lfi does better/different than IxJS?
|
| [0] https://github.com/ReactiveX/IxJS
| tomeraberbach wrote:
| I was not aware of it!
|
| As far as I can tell, it has some similar ideas (and seems
| pretty nice!), but I noticed the following differences:
|
| - I don't _think_ it supports "concurrent" iteration [1]
|
| - It doesn't have a "reducer" concept with composition and what
| not [2]
|
| - It doesn't have a concept of "optionals" [3]
|
| [1] https://lfi.dev/docs/concepts/concurrent-iterable
|
| [2] https://lfi.dev/docs/concepts/reducer
|
| [3] https://lfi.dev/docs/concepts/optional
| WorldMaker wrote:
| > I don't _think_ it supports "concurrent" iteration
|
| I believe what you are calling a "concurrent" iteration is
| one of the use cases for Observables, so would be on the RxJS
| side (the "parent" project).
|
| > It doesn't have a "reducer" concept with composition and
| what not
|
| Interesting. I'll read further on what you are doing there.
| Seems possibly similar to Lenses/Prisms in FP, with different
| rules.
|
| > It doesn't have a concept of "optionals"
|
| Good point. So far I've not found a JS approach to optionals
| I'm entirely happy with. I've had some suggest I should give
| Effect [0] a deeper look, but given I'm mostly happy with
| RxJS the bulk of Effect doesn't appeal to me. The dual of an
| Option being an iterator with 0 or 1 values is something I've
| seen before and something to keep in mind.
|
| [0] https://effect.website/docs/data-types/option/
| tomeraberbach wrote:
| > I believe what you are calling a "concurrent" iteration
| is one of the use cases for Observables, so would be on the
| RxJS side (the "parent" project).
|
| Ah, got it. I don't think I quite understood the
| relationship between this project and RxJS. I did know
| about RxJS's observables, yes. I do sort of hint at the
| relationship between concurrent iterables and the
| observables [1], but I tried to make concurrent iterables a
| bit more specialized to what you'd normally use an iterable
| for.
|
| > Interesting. I'll read further on what you are doing
| there. Seems possibly similar to Lenses/Prisms in FP, with
| different rules.
|
| I think I've heard of lenses (but not prisms), but don't
| think I ever actually learned what they are. Would be
| curious if you find a connection between those and my
| reducers :)
|
| > Good point. So far I've not found a JS approach to
| optionals I'm entirely happy with. I've had some suggest I
| should give Effect [0] a deeper look, but given I'm mostly
| happy with RxJS the bulk of Effect doesn't appeal to me.
| The dual of an Option being an iterator with 0 or 1 values
| is something I've seen before and something to keep in
| mind.
|
| Yeah, I liked representing optionals as iterables because
| you get a lot of functionality for "free" [2]
|
| [1] https://lfi.dev/docs/concepts/concurrent-
| iterable#:~:text=A%....
|
| [2] https://lfi.dev/docs/concepts/optional#why-use-
| iterables-to-...
| WorldMaker wrote:
| Prisms are Lenses through Options. They compose very
| similarly to Lenses and it is easy to convert a Lens to a
| Prism to compose Prisms from existing Lenses. Every Lens
| is a very simple "degenerate" Reducer (a Lens is just a
| getter and a setter from one "state" to another), but
| Prisms feel even more like "real" Reducers because of the
| optional states that can be involved. So Prisms and
| Lenses are very common composable Reducers in some styles
| of FP.
| stevedekorte wrote:
| Looks useful and powerful. Nice work!
| tomeraberbach wrote:
| Thank you!
| uwemaurer wrote:
| great library! I just tried it out and compared it with iter-
| tools
|
| I like Lfi better, great documentation!
|
| Here is the example I use to compare:
| https://stackblitz.com/edit/stackblitz-starters-ia9ujg6m?fil...
| conartist6 wrote:
| Love to see `iter-tools` come up, though obviously less so for
| getting beat.
|
| Still, I love seeing innovation in this field as much as
| anyone! And it's true that I've been dragging my feet on
| building concurrency support for iter-tools...
|
| That said, iter-tools still has some killer APIs like peekerate
| that I would find it quite hard to live without, and for iter-
| tools 8 I have some major tricks planned like making the
| sync/async divide go away completely.
| rasso wrote:
| Just so I understand: Could this library be used for iterating
| large collections without blocking the main thread? Basically as
| a replacement for a worker with synchronous iteration?
| Mustachio wrote:
| Having had a quick read of the concept, I believe it's
| concurrent in the sense that Promise.all() is compared to
| awaiting in a for loop?
|
| In other words, put on the stack at the same time but not a
| different thread.
|
| Do correct me if I misunderstood though.
| conartist6 wrote:
| Yes, but I'd add a caveat.
|
| It will be very efficient when the async operations are few,
| and slower. It will not be very efficient when the async
| operations are many and fast.
|
| That's because the await keyword itself blocks the main thread
| every time a function call is made. It has to because the
| `await` keyword is defined to return to the event loop and
| resume processing only during the next "tick" and after other
| queued ticks have run.
|
| For a large collection the overhead on the event loop can be
| calculated as: COLLECTION_SIZE * FUNCTION_CALLS_PER_ELEMENT.
| Since function calls per element goes up as you wrap layers of
| helper functions, even with this library you would face a
| strong perverse incentive to avoid using (async) function calls
| or nested iterators (if at all possible) when handling large
| collections, especially with high desired throughput,
| especially in situations where you want the event loop to stay
| unclogged so that you can, for example, redraw the UI.
| conartist6 wrote:
| In the most ridiculous scenario you try to process two high-
| throughput async data streams concurrently and you end up
| with this nightmarish "bouncing" where each "thread" of
| processing can only process until it sees an await keyword
| before being forced to cede control back the other thread. At
| this point it would be reasonable to expect that the amount
| of overhead would exceed the amount of real work being done,
| and in such a way that the text of the EcmaScript spec makes
| impossible to optimize or fix.
| conartist6 wrote:
| There's a rather famous blog post that cuts to the heart of
| all this: https://blog.izs.me/2013/08/designing-apis-for-
| asynchrony/
| rasso wrote:
| Thank you for taking the time to explain! While the details
| seem to be well over my head, I think I'll stick to a worker
| then, as the specific thing I was talking about was taxonomy
| faceting of a pretty large dataset. Thank you again!
| fergie wrote:
| I don't understand (but am open to be persuaded), isn't iteration
| in js now trivially easy and actually quite powerful? Especially
| since we can now put 'await' everywhere? What are the killer use
| cases for this lib?
| conartist6 wrote:
| for the record sprinkling await everywhere is not something you
| should take lightly. await is still probably the single most
| high-level high-overhead construct in JS, and putting it inside
| tight loops is a recipe for perf disaster
| dogboat wrote:
| Depends what you are waiting for.
|
| Maybe an in memory cache hit---that you explicitly code---
| causes a sync resolve to happen, woohoo!
|
| But if it goes to IO and you are waiting for the event loop
| to schedule you back in you could be in for a shock even if
| the IO op is tiny. Especially doing it alot.
|
| Only matters probably on a high CPU usage server which
| hopefully you avoid by scaling up on cpu and using all the
| cores.
|
| On the other hand... you are being cooperative :)
|
| But tight loop IO is an antipattern ... as is doing a lot---
| millions---of IO ops from node (per request or job) that you
| need to worry about this (use a different language for your
| DB server!).
| tomeraberbach wrote:
| I think there are two things the library makes easier than what
| you're suggesting:
|
| - Some people like/prefer writing iteration in a functional
| style, which this library enables
|
| - It's pretty easy to create unintentional async/concurrency
| bottlenecks with the simple `Promise.all` approach (see the
| home page example)
| darkest_ruby wrote:
| Another fp-ts
| tomeraberbach wrote:
| I'm somewhat familiar with fp-ts, and I think lfi is pretty
| different.
|
| fp-ts seems more focused on "pure" functional programming in
| the style of languages like Haskell. It's much more opinionated
| on how you should write your code, including the data
| structures you should use. Plus, it's not really concerned with
| concurrency in the way that lfi is.
|
| I think lfi is a lot less invasive/opinionated on how you write
| your code.
| Hugsun wrote:
| How does this compare to Effect, or Fluture? There have been so
| many attempts at improving these things in JS that it's hard to
| keep track.
| agos wrote:
| easy differences: Effect does everything and is huge, this does
| iteration only and is tiny
| tomeraberbach wrote:
| Yeah, that's true.
|
| I think it just has completely different goals. Read through
| the home page examples and getting started and you'll see
| it's pretty different.
| bmcahren wrote:
| I'm not sure I know the use case for all of this logical
| complexity. Is there a specific use-case you had in mind?
|
| Split the array of elements to process. Set up a promise.all
| against a function that handles sync/async function calls against
| each chunk and accumulates an array of results. This isn't the
| end of the world for any of the practical use cases I can think
| of. The efficiency loss and memory consumption is miniscule even
| across hundreds of thousands of records. This is as concurrent as
| anything else - the event loop is still a mandatory part of the
| processing loop. You can even defer every 1000 records to the
| event loop. Javascript is syncronous so there is no concurrency
| without threading. I can't think of an IO or non-IO function that
| would create the issues described of not yielding. We used to
| have chunks of 1000000+ records that would freeze the event loop.
| We solved this with structured yields to the event loop to allow
| other processes to continue without dropping TCP packets etc.
|
| If the results are objects, you're not even bloating memory too
| much. 800KB for 100,000 records seems fine for any practical use
| case.
|
| I like the idea of an async generator to prevent over-pre-
| calculating the result set before it's needed but I couldn't
| justify pivoting to a set of new functions called pipe, map, etc
| that require a few too many logical leaps for your typical
| ECMAScript developers to understand.
|
| What's the win here? It must be for some abuse of Javascript that
| should be ported to WASM. Or some other use-case that I've
| somehow not seen yet.
| gavmor wrote:
| pipe and map are too complex for "ECMAScript devs?" give me a
| break!
|
| > should be ported to WASM
|
| valid.
| tomeraberbach wrote:
| To clarify, the point of it isn't _just_ performance.
|
| It's a combination for writing the iteration code in a
| functional style, which some people like/prefer, while
| retaining certain performance characteristics that you'd get
| from writing more imperative code.
|
| For example, the "naive" approach to functional programming +
| concurrency results in some unintentional bottlenecks (see the
| concurrent iteration example on the home page).
| fermigier wrote:
| "lfi is a lazy functional sync, async, and concurrent iteration
| library for JavaScript and TypeScript"
|
| Looks good (I understand all the word and even the whole
| sentence).
|
| But what is it good for? ("What's in it for me?")
|
| What are the use cases? How is it difference / how does it
| improve on / when should I use it instead of / ... lodash, ramda,
| functional.js, RxJs, etc.?
| tomeraberbach wrote:
| > But what is it good for? ("What's in it for me?") > What are
| the use cases?
|
| I think the examples on the home page and in the "getting
| started" answer these questions to be honest. Do you feel it's
| unclear or something is missing?
|
| > How is it difference / how does it improve on / when should I
| use it instead of / ... lodash, ramda, functional.js, RxJs,
| etc.?
|
| I think some of that info is implicitly there in the docs, but
| I think this is good feedback. I should probably have a page
| comparing lfi vs other libraries.
|
| To answer here though, for most of the libraries you mentioned,
| they don't provide support for _concurrent_ async iteration
| (and some of them don't support even sequential async
| iteration).
|
| RxJS is the exception there I think. Although, I think lfi has
| a bit of a simpler API (that also matches the sync versions of
| the APIs) that's primarily designed for iteration rather than
| generalized observables. Plus, I don't _think_ RxJS has the
| same performance characteristics around tree-shakeability.
| fricze wrote:
| Interesting. How is the concurrency achieved? I don't see any
| info about concurrency implementation on the website
| tomeraberbach wrote:
| Some info on this page:
| https://lfi.dev/docs/concepts/concurrent-iterable
|
| Recommend reading it all, but this part is probably most
| useful: https://lfi.dev/docs/concepts/concurrent-iterable#how-
| does-i...
___________________________________________________________________
(page generated 2024-12-13 23:02 UTC)