[HN Gopher] Writing A Job Runner (In Elixir) (Again) (10 years l...
___________________________________________________________________
Writing A Job Runner (In Elixir) (Again) (10 years later)
Author : rhgraysonii
Score : 113 points
Date : 2025-05-23 10:41 UTC (12 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| rhgraysonii wrote:
| I wrote this up after having written a tutorial on doing this in
| GenStage around a decade ago, and thought it was interesting to
| have the two of them side by side to consider. I linked the
| original in the document linked to here.
|
| Overall, I'm wildly impressed at how this Elixir code held up,
| and it was a joy to revisit this.
| freedomben wrote:
| Great write up, thanks for sharing! Nothing against Oban, but
| it _is_ nice seeing someone in the Elixir community not just
| say, "run Oban" and drop the mic
| dangerousnight wrote:
| As someone who has been learning Elixir on and off for over a
| year, this looks really exciting on first skim through. Excited
| to give this a deep read this weekend!
| rhgraysonii wrote:
| If you have any feedback or anything is unclear feel free to
| open an issue. I am thinking I am going to take this and expand
| the concepts to start as a beginners primer going through the
| primary concepts in brief akin to Elixir School, and then
| expand it into building this and a web service that is using it
| and offering some real time features.
| sbrother wrote:
| FYI if anyone is looking for a production ready job runner in
| Elixir, I strongly recommend Oban. They have a paid pro version
| but the open source version is still absolutely fantastic.
| victorbjorklund wrote:
| Just to add to the parent the Open Source version and the Pro
| Version just differs in that the Pro Version has a few plugins
| that the normal version has (for example workflows where you
| can have multiple workers working together) but it is not
| something you need for most use-cases.
| pikdum wrote:
| It's more than just a few - even more basic things like rate
| limiting or concurrency controls are gated behind Pro. It
| works extremely well, but I've been reluctant to use it in
| open source projects because there's quite a bit in there I'd
| need to rebuild.
| throwawaymaths wrote:
| im curious about your use case -- it seems weird (to me) to
| use it in an open source project unless its some kind of
| turnkey full app -- is there a way to just release it and
| encourage people to bring their own oban keys? that way it
| looks good for the elixir ecosystem that it has found a way
| to support hybrid open source libraries and expands the
| obam ecosystem
| pikdum wrote:
| My open source projects are generally more applications
| than libraries, yeah.
| rhgraysonii wrote:
| Oban is great! This is definitely just met to be a learning
| exercise. But it could go quite a ways on its own in prod.
| freedomben wrote:
| Do you worry about enshittification down the road?
| cpursley wrote:
| I'd love to see one for Elixir built around the Postgres pgmq and
| pg_crom extensions similar to what pgflow is doing:
| https://www.pgflow.dev/
| rhgraysonii wrote:
| That would be super cool. Pretty much everything Supabase is up
| to is awesome.
| cpursley wrote:
| Yeah, their realtime thing is powered by Elixir (listens to
| Postgres WAL).
| jumski wrote:
| pgflow can be used with any language/runtime, I just started
| with TypeScript and Supabase, as that's what I'm using.
|
| The worker is stateless and "dumb" by design (currently it runs
| on serverless functions) - it just calls SQL functions:
| "poll_for_tasks" to get some tasks from the queue and then
| either "complete_task" or "fail_task" after executing user code
| - that's it, nothing more, so it should be relatively easy to
| adopt it to other runtimes.
|
| I have written a small architecture primer on pgflow if anyone
| is interested in its simple but flexible design
| https://www.pgflow.dev/concepts/how-pgflow-works/
| jamal-kumar wrote:
| The absolute facility with which you can do distributed
| applications in Elixir is the big thing that sold me in the first
| place. Nice to see this tutorial!
| hmmokidk wrote:
| Incredibly well written. You clearly put a lot of thought and
| effort into this. Thank you.
|
| Also FOR UPDATE SKIP LOCKED is interesting.
| rhgraysonii wrote:
| You also could use an advisory exact lock, which would
| serialize the order of input and force everyone to wait in
| line, but the advantage here is that we are allowed shared
| concurrent reads while maintaining our lock. Thank you for the
| feedback. It was fun to revisit after 10 years. The initial
| piece came after a Columbus, OH Ruby meetup that Jose attended.
| It was quite fun to see him put it together as a whole and I
| ran with things a bit further than his first ideas with this.
| sarchertech wrote:
| SKIP LOCKED is amazing for this kind of thing. I used it to
| build a transaction outbox a few years ago.
|
| One thing worth looking into if you do this in production is
| adding a way to add partitions such that each partition is
| single threaded. It's the only way to guarantee ordering if
| your jobs are doing anything non-deterministic.
| boruto wrote:
| We have a system where each pod spins up around 30
| scheduled job instances of one job each processing a
| "partition", then transaction outbox is queried with hash
| of identifier equating it to partition.
|
| We increased partition counts on sale days and it works
| well for us.
|
| Couple of gotchas we had were.
|
| 1) Using hashtext from postgres is sketchy.
|
| 2) Increasing partiton count is an orchestra which requires
| stopping the partition.
| oxidant wrote:
| FOR UPDATE SKIP LOCKED is great, but it needs to be in a
| transaction. In the example code it won't "do" anything because
| it selects for update then immediately loses the lock.
|
| Claude says you can use a CTE to select and the run your update
| with the locked rows, but I have only ever used transactions.
| huqedato wrote:
| Years ago I built something similar, a Pub/Sub notification
| system: https://github.com/huqedato/qnotix. It is still running
| (with small modifications), in production, at my ex-customer.
|
| Such a pity the industry (customers) reacts with skepticism every
| time I propose solutions based in Elixir/Erlang. I always hear:
| "Elixir, what? We want Java/.Net/Python/php"
| jbverschoor wrote:
| It means they don't trust your judgement, aka you're just there
| to execute their plans. You're the floor guy, not the
| architect, owner, or developer of a house.
| huqedato wrote:
| Not really. I am usually the one they pay to solve their
| problems. It's not a matter of trusting my judgement, rather
| they are trapped in an institutional/corporate mindset (old
| patterns are most suitable, 'best practice', 'reliable,
| proven tech' and such)
| stonemetal12 wrote:
| > I am usually the one they pay to solve their problems.
|
| In the context that doesn't really say a lot, they pay the
| floor guy and the architect to solve problems, just
| different problems.
|
| >they are trapped in an institutional/corporate mindset
| (old patterns are most suitable, 'best practice',
| 'reliable, proven tech' and such)
|
| While true, it is also correct. From their perspective
| Elixir is an exotic material, the next "one they pay to
| solve their problems" is not going to know how to work with
| it, scrap it, and start over. Therefore the Elixir solution
| isn't a good solution for them.
|
| As a business man once told me, "I never saw a feature I
| liked so much that I was wiling to pay for it twice."
| icedchai wrote:
| The problem is finding other people to work in less mainstream
| languages. Probably not an issue right now, but during better
| times...
| freedomben wrote:
| I've run into this a few times before too. Sometimes discussing
| maintenance/ongoing costs and how minimal they are with an
| Elixir/Phoenix app can be compelling, though not always. I
| prefer to be over-transparent with clients so I'll usually add
| that if they plan to switch contractors often then it's
| probably not (tho sometimes still is) a good idea to go with
| Elixir. When I compare the amount of ongoing effort to keep a
| PHP, Node, or Python app running securely in prod with the
| ongoing effort with a Phoenix app, it's usually a better deal
| for them.
|
| The one Danger Zone area though is when they want something
| between WordPress and a "real" app. A lot of the clients I've
| talked to already have WordPress and are really trying to
| stretch the framework to places it isn't really made to go.
| Those customers often think "I should be able to get a full app
| built for <=$10,000" and often shop around until they find
| someone who will accept it, and then they usually end up
| getting a bad/buggy product that is slow as dirt. At that point
| it's often too late as well because nobody wants to pay to
| start over. For those clients though I wouldn't recommend
| Elixir because they would almost rather spend more in
| maintenance costs than in initial development because the
| numbers feel smaller when spread out, even if they add up to
| more in the long run and with a much worse product.
|
| That is not to say of course that _any_ Elixir app is going to
| be better - there are bad Elixir devs just like any stack,
| though I think Elixir 's relative lack of popularity actually
| leads to an overall more skilled dev pool.
| udkl wrote:
| Just want to say thank you for the detailed write up. I am
| enjoying the article a lot!
| rhgraysonii wrote:
| Thanks!
___________________________________________________________________
(page generated 2025-05-23 23:01 UTC)