[HN Gopher] Piscina - The Node.js Worker Pool
___________________________________________________________________
Piscina - The Node.js Worker Pool
Author : AquiGorka
Score : 55 points
Date : 2021-03-22 10:54 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| revskill wrote:
| Could i use it on a serverless platform ?
| timmit wrote:
| I assume u can, but how it performs, it really dependents on
| the virtual CPU of the, platform.
|
| I did something similar
|
| https://github.com/tim-hub/pambdajs
|
| but I haven't done the comparison on aws lambda yet
| cjdell wrote:
| What would be nice is a way of preventing the same job running
| multiple times concurrently. Like if I start a job and a job with
| the same parameters was already started milliseconds ago then it
| automatically awaits the already running job rather than starting
| another.
| sbarre wrote:
| You could build a reasonably lightweight supervisor pattern
| that uses a parameter-derived hash for comparison to handle
| this kind of situation in your application too.
|
| Might be easier and more flexible than asking the library to do
| it?
| tomwojcik wrote:
| That's exactly what I do in python with celery. I order all
| dicts/lists and hash this final object. If hash is not found
| in the db, I set hash as the task id. With `track started`
| setting I never run the same task twice.
| fendy3002 wrote:
| If you can use redis, redlock can be a nice implementation.
|
| Example:
| https://github.com/fendy3002/NodeLearn/blob/master/Redlock/s...
| timmit wrote:
| u remind me my own researching project too.
|
| https://github.com/tim-hub/pambdajs
| freeqaz wrote:
| Does anybody use this or anything similar? If so, what problems
| are you solving?
| rektide wrote:
| Knex, the SQL query builder, uses Tarn.js[1] for connection
| pooling to the db.
|
| I've been using Tarn a bunch at work recently. We're doing some
| batch jobs, and I'm queuing work at each stage in Tarn.js
| pools. I created my own enqueue function that waits until the
| pool is less than a high-water mark in size before enqueueing.
| Then the pool has however many workers running.
|
| Neither of these are off-thread pools. But they help a lot of
| for managing multiple async operations.
|
| [1] https://github.com/vincit/tarn.js/
| Etheryte wrote:
| The worker API in Javascript is quite a pain to use, but
| needless to say multithreading is invaluable in many contexts,
| both in the browser and in Node. I haven't used this library
| but it seems to solve a similar problem to other ones in the
| same space -- make writing multithreaded code sane, allowing
| you to avoid writing a bunch of repetitive boilerplate.
| vorticalbox wrote:
| At work I use https://github.com/pioardi/poolifier
|
| We have an api used for generating reports from mongo dB to
| csv, this let's the report process in the background leaving
| the report api to still handle requests.
| Skhalar wrote:
| This is useful for processing large chunks of data like audio
| files (look at Superpowered sdk) but breaking em down or when
| processing multiple files.
| barefeg wrote:
| I'm guessing the fact that node is single threaded
| nobleach wrote:
| JavaScript is single threaded, Node is most certainly not.
| gbrits wrote:
| Yes it surely is
| zbentley wrote:
| My "htop" display surely disagrees. So do my node
| programs that lag out waiting for disk IO thread pool
| slots. Sounds does node's documentation:
| https://nodejs.org/api/worker_threads.html
| hmcdona1 wrote:
| It's not exactly. Certain operations like I/O are
| threaded in Node. libuv has it's own threadpool that it
| uses for a lot of these types of tasks.
| ddoolin wrote:
| We use worker threads directly to process large unorganized
| (for the browser) datasets and do some deductions before it
| hits the store.
|
| We also have a worker thread blocked on a redis channel that
| acts as a queue.
| [deleted]
| niftylettuce wrote:
| Yeah. I built Bree @ https://jobscheduler.net
|
| It has support for worker threads + cron + human strings
|
| GH: https://github.com/breejs/bree
| ddoolin wrote:
| This is cool. Our use case was very simple so we went with
| node-schedule but this may have more upsides. I've had
| trouble with the former. Thanks.
| niftylettuce wrote:
| I helped maintain all the Node.js node/cron/agenda
| libraries. Just switch to Bree and your troubles will be
| gone.
| searchableguy wrote:
| I use workers in deno to evaluate user facing code which may
| take a lot of time to finish and need sandboxing. Another place
| it's used in where I want to keep context and reuse it for
| subsequent code execution for each user.
|
| In deno, you can restrict what the code inside worker can do by
| passing a map of allowed permissions. I have built a simple
| privilege system on top of this to allow users different access
| level.
|
| This is cheaper and faster than spinning up container.
| domenicd wrote:
| I maintain a command-line utility which assembles an eBook from
| scraped .xhtml files. It uses a similar package, workerpool, to
| process multiple chapters in parallel.
|
| https://github.com/domenic/worm-scraper/blob/master/lib/conv...
|
| https://www.npmjs.com/package/workerpool
| curben wrote:
| AFAIK this is a wrapper of worker_threads API.
| https://github.com/tuananh/camaro utilises this for multi-
| threading processing of XML input. Hexo (static site generator)
| is attempting to utilise this library
| (https://github.com/hexojs/hexo/issues/4355).
| maxrev17 wrote:
| Might be quite nice for keeping tokens refreshed?
| hfktk4nrn wrote:
| Is it just me, or do I see a trend in naming projects using
| romance language words (Italian/Spanish/France)?
|
| Does it sound more exotic? Are these words less crowded?
| andreynering wrote:
| I'm curious on why you call these languages "romance"
| languages.
|
| "Piscina" is a Portuguese word for "pool", by the way.
| olakease wrote:
| https://en.wikipedia.org/wiki/Romance_languages
| age008 wrote:
| To quote Wikipedia, "Romance languages (less commonly Latin
| languages, or Neo-Latin languages) are the modern languages
| that evolved from Vulgar Latin between the third and eighth
| centuries." "Piscina" is also "pool" in Romanian, which is,
| you guessed, a Romance language.
| andreynering wrote:
| Got it.
|
| In Portuguese they're only known for "Latin languages",
| which explains my question. ;-)
| Kaze404 wrote:
| Not really. I've personally heard the term before.
| https://pt.wikipedia.org/wiki/L%C3%ADnguas_rom%C3%A2nicas
| gdsimoes wrote:
| I haven't. Maybe it's not a popular term in Brazil.
| mayank wrote:
| > I'm curious on why you call these languages "romance"
| languages.
|
| That's what they're called:
| https://en.wikipedia.org/wiki/Romance_languages
| [deleted]
| [deleted]
| rodrigodiez wrote:
| Romance languages are those derived from vulgar latin. They
| are called romance because Rome (Roma)
|
| Portuguese is also a romance language, as Spanish or Italian.
| Piscina also means pool in Spanish by the way :)
|
| https://simple.wikipedia.org/wiki/Romance_languages
| zdragnar wrote:
| Maybe it is just an English thing, but Romance languages are
| anything evolved from latin (predumably spread by the Roman
| empire, hence "romance"). Portuguese is included in the list,
| fwiw.
|
| https://en.m.wikipedia.org/wiki/Romance_languages
| th3h4mm3r wrote:
| Also in Italy, we say "piscina" in the same way.
| G4BB3R wrote:
| I think it's because most of english names are already taken.
| ericlewis wrote:
| Neat! I'm friends with the creator of this and teased him a bit
| about the name (so many of these projects have weird names now a
| day)
|
| The reason is: don't wanna be boring from what I could glean.
| 29athrowaway wrote:
| Common mistake in stream code: -- .on('end')
| ++ .once('end')
___________________________________________________________________
(page generated 2021-03-23 23:01 UTC)