[HN Gopher] Bridging Elixir and Python with Oban
___________________________________________________________________
Bridging Elixir and Python with Oban
Author : sorentwo
Score : 114 points
Date : 2026-02-19 11:07 UTC (11 hours ago)
(HTM) web link (oban.pro)
(TXT) w3m dump (oban.pro)
| cpursley wrote:
| Very nice, Oban is great. I effectually found a similar approach
| with pgflow.dev (built around pgmq) - but the stateless deno
| "workers" are pretty unreliable and built an elixir worker
| (https://github.com/agoodway/pgflow) that can pick up and process
| jobs that were created by pgflow's supabase/typescript client. So
| maybe there's an opportunity also with Oban to have a
| TypeScript/Node client that can insert jobs that Elixir/Python
| Oban can pick up. Also, I wonder if another approach vs the
| python workers picking things up is to have elixir workers
| call/run python/lua, etc code or is that too limiting?
| elitepleb wrote:
| that's easy with https://hexdocs.pm/pythonx/Pythonx.html and
| https://hexdocs.pm/lua/Lua.html and works well too
| cpursley wrote:
| btw, a lot of postgres envs are not going to have pgmq, so just
| use Oban and don't reinvent the wheel like I did ;)
| languagehacker wrote:
| I feel like if you need to utilize a tool like this, odds are
| pretty good you may have picked the Wrong Tool For the Job, or,
| perhaps even worse, the wrong architecture.
|
| This is why it's so important to do lots of engineering _before_
| writing the first line of code on a project. It helps keep you
| from choosing a tool set or architecture out of preference and
| keeps you honest about the capabilities you need and how your
| system should be organized.
| whalesalad wrote:
| What leads you to this conclusion
| Arubis wrote:
| It's almost as though choosing a single-threaded, GIL-
| encumbered interpreted scripting language as the primary
| interface to an ecosystem of extremely parallelized and
| concurrent high-performance hardware-dependent operations
| wasn't quite the right move for our industry.
| markstos wrote:
| Ha. The question now is whether the ML industry will change
| directions or if the momentum of Python is a runaway train.
|
| I can't guess. Perl was once the "800-pound gorilla" of web
| development, but that chapter has long been closed. Python on
| the other hand has only gained traction since that time.
| geooff_ wrote:
| I disagree, using python for a web-server and something like
| celery for background work is a pretty common pattern.
|
| My reading of this is it more or less allows you to use
| Postgres (which you're likely already using as your DB) for the
| task orchestration backend. And it comes with a cool UI.
| languagehacker wrote:
| That's not the sort of architecture I'm referring to. I'm
| specifically talking about splitting your application layer
| between Elixir and Python.
| victorbjorklund wrote:
| Wait until you find out about some people not writing pure
| python apps but also have some code in JavaScript. Crazy to
| mix more than one language in one machine.
| victorbjorklund wrote:
| Strange opinion. Plenty of apps have more than one language. I
| might end up using this.
|
| Why? Because my app is built in Elixir and right now I'm also
| using a python app that is open source but I really just need a
| small part of the python app. I don't wanna rewrite everything
| in Elixir because while it's small I expect it to change over
| time (basically fetching a lot of data sources) and it will be
| pain to keep rewriting it when data collections needs to change
| (over a 100 different sources). Right now I run the python app
| as an api but it's just so overkill and harder to manage vs
| just handling everything except the actually data collection in
| Elixir where I am already using Oban.
| markstos wrote:
| Sometimes the "right tool for the job" philosophy leads to
| breaking down a larger problem into two small problems, each
| which has a different "right tool".
|
| Choosing a single tool that tries to solve every single problem
| can lead to its own problems.
| ananthakumaran wrote:
| We have a similar use case. All Elixir code base, but need to use
| Python for ML libraries. We decided to use IPC. Elixir will spawn
| a process and communicate over stdio. https://github.com/akash-
| akya/ex_cmd makes it a breeze to stream stdin and stdout. This
| also has the added benefit of keeping the Python side completely
| stateless and keeping all the domain logic on the Elixir side.
| Spawning a process might be slower compared to enqueuing a job,
| but in our case the job usually takes long enough to make it
| irrelevant.
| dnautics wrote:
| I have one vibecoded ml pipeline now and I'm strongly
| considering just clauding it into Nx so I can ditch the python
| flippant wrote:
| I did exactly this in early 2025 with a small keyword tagging
| pipeline.
|
| You may run into some issues with Docker and native deps once
| you get to production. Don't forget to cache the bumblebee
| files.
| dnautics wrote:
| No problem. It's an SLM, I have a dedicated on-prem GPU
| server that I deploy behind tailscale for inference. For
| training, I reach out to lambdalabs and just get a beefy
| GPU for a few hours for the cost of a Starbucks coffee.
| markstos wrote:
| Is this part of a web server or some other system where you
| could end up spawning N python processes instead of 1 at a
| time?
| rozap wrote:
| I use a similar strategy for python calls from elixir. This
| is in a web server, usually they're part of a process pool.
| So we start up N workers and they hang out and answer
| requests when needed. I just have an rpc abstraction that
| handles all the fiddly bits. The two sides pass erlang terms
| back and forth. Pretty simple.
| ananthakumaran wrote:
| No, it's a background job. We can easily control the Python
| process count by controlling the job queue concurrency on the
| Elixir side.
| barrell wrote:
| Similar use case as well. I use erl ports to spawn a python
| process as well. Error handling is a mess, but using python as
| a short scripting language and elixir for all the
| database/application/architecture has been very ideal
| Kaliboy wrote:
| Honestly you saved yourself major possible headaches down the
| line with this approach.
|
| At my work we run a fairly large webshop and have a ridiculous
| number of jobs running at all times. At this point most are
| running in Sidekiq, but a sizeable portion remain in Resque
| simply because it does just that, start a process.
|
| Resque workers start by creating a fork, and that becomes the
| actual worker.
|
| So when you allocate half your available RAM for the job, its
| all discarded and returned to the OS, which is FANTASTIC.
|
| Sidekiq, and most job queues uses threads which is great, but
| all RAM allocated to the process stays allocated, and generally
| unused. Especially if you're using malloc it's especially bad.
| We used jemalloc for a while which helped since it allocates
| memory better for multithreaded applications, but easiest is to
| just create a process.
|
| I don't know how memory intensive ML is, what generally screwed
| us over was image processing (ImageMagick and its many memory
| leaks) and... large CSV files. Yeah come to think of it, you
| made an excellent architectural choice.
| kzemek wrote:
| We also had a similar use case, so I built Snex[0] -
| specifically for Elixir-Python interop. Elixir-side spawns
| interpreters with Ports managed by GenServers, Python-side has
| a thin asyncio runtime to run arbitrary user code. Declarative
| environments (uv), optimized serde with language-specific
| objects (like `%MapSet{}` <-> `set`), etc. Interpreters are
| meant to be long lived, so you pay for initialization once.
|
| It's a very different approach than ex_cmd, as it's not really
| focused on the "streaming data" use case. Mine is a very
| command/reply oriented approach, though the commands can flow
| both ways (calling BEAM modules from Python). The assumption is
| that big data is passed around out of band; I may have to
| revisit that.
|
| [0]: https://github.com/kzemek/snex
| jbott wrote:
| This might be of interest to others: Last night I stumbled
| across Hornbeam, a library in a similar vein from the author of
| Gunicorn that handles WSGI / ASGI apps as well as a specific
| wrapper for ML inference
|
| https://erlangforums.com/t/hornbeam-wsgi-asgi-server-for-run...
| https://github.com/benoitc/hornbeam
| jongjong wrote:
| I don't see the point of Elixir now. LLMs work better with
| mainstream languages which make up a bigger portion of their
| training set.
|
| I don't see the point of TypeScript either, I can make the LLM
| output JavaScript and the tokens saved not having to add types
| can be used to write additional tests...
|
| The aesthetics or safety features of the languages no longer
| matter IMO. Succinctness, functionality and popularity of the
| language are now much more important factors.
| HorizonXP wrote:
| So I know these are just benchmarks, but apparently Elixir is
| one of the best languages to use with AI, despite having a
| smaller training dataset:
| https://www.youtube.com/watch?v=iV1EcfZSdCM and
| https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/ma...
|
| Furthermore, it's actually kind of annoying that the LLMs are
| _not_ better than us, and still benefit from having code
| properly typed, well-architected, and split into modules
| /files. I was lamenting this fact the other day; the only
| reason we moved away from Assembly and BASIC, using GOTOs in a
| single huge file was because us humans needed the organization
| to help us maintain context. Turns out, because of how they're
| trained, so do the LLMs.
|
| So TypeScript types and tests actually do help a lot, simply
| because they're deterministic guardrails that the LLM can use
| to check its work and be steered to producing code that
| actually works.
| dnautics wrote:
| I don't think LLMs benefit from having code properly typed
| (at the call definition). It's costly to have to check a
| possibly remote file to check. The LLM should be able to
| intuit what the types are _at the callsite_ and elixir has
| ~strong conventions that LLMs probably take advantage of
| baseonmars wrote:
| llms benefit greatly from feedback and typing/type errors
| are one of the fastest and easiest methods of feedback to
| give to an llm.
| dnautics wrote:
| Think about fitts law: the fastest place to click under a
| cursor is the location of the cursor. For an LLM the
| least context-expensive feedback is no feedback at all.
|
| I think codebases that are strongly typed sometimes have
| bad habits that "you can get away with" because of the
| typing and feedback loops, the LLM has learned this.
|
| https://x.com/neogoose_btw/status/2023902379440304452?s=6
| 1
| cloud8421 wrote:
| > I don't see the point of Elixir now. LLMs work better with
| mainstream languages which make up a bigger portion of their
| training set.
|
| I can't say if it works better with other languages, but I can
| definitely say both Opus and Codex work really well with
| Elixir. I work on a fairly large application and they
| consistently produce well structured working code, and are able
| to review existing code to find issues that are very easy to
| miss.
|
| The LLM needs guidance around general patterns, e.g. "Let's use
| a state machine to implement this functionality" but it writes
| code that uses language idioms, leverages immutability and
| concurrency, and generally speaking it's much better than any
| first pass that I would manually do.
|
| I have my ethical concerns, but it would be foolish of me to
| state that it works poorly - if anything it makes me question
| my own abilities and focus in comparison (which is a whole
| different topic).
| jakejohnson wrote:
| LLMs work great with Elixir. Running tsc in a loop while
| generating code still catches type errors introduced by an LLM
| and it's faster than generating additional tests. Elixir is
| also succinct and highly functional. If you can't find a
| specific library it's easier than ever to build out the
| barebones functionality you need yourself or use NIFs, ports,
| etc.
|
| https://dashbit.co/blog/why-elixir-best-language-for-ai
| WolfeReader wrote:
| I'm starting to see a new genre of post here in the AI bubble,
| where people go to topics that aren't about AI at all, and
| comment something like, "this doesn't matter because it's not
| AI". This is the third I've seen in a week.
| dnautics wrote:
| > Succinctness, functionality and popularity of the language
| are now much more important factors.
|
| No. I would argue that popularity _per se_ is irrelevant: if
| there are a billion examples of crap code, the LLMs learn crap
| code. conversely know only 250 documents can poison an LLM
| independent if model size. [Cite anthropic paper here].
|
| The most important thing is _conserve context_. Succinctness is
| not really what you want because most context is burned on
| thinking and tool calls (I think) and not codegen.
|
| Here is what I think is not important: strong typing, it
| requires a tool call anyways to fetch the type.
|
| Here is what I think is important:
|
| - fewer footguns - great testing (and great testing examples) -
| strong language conventions (local indicators for types,
| argument order conventions, etc) - no weird shit like
| __init__.py that could do literally anything invisible to the
| standard code flow
| techpression wrote:
| Your code doesn't run anywhere? Running on the BEAM is
| extremely helpful for a lot of things. Also, I review my LLM
| output, I want that experience to be enjoyable.
| perrygeo wrote:
| > Succinctness, functionality and popularity of the language
| are now much more important factors.
|
| Not my experience at all. The most important factor is
| simplicity and clarity. If an LLM can find the pattern, it can
| replicate that pattern.
|
| Language matters to the extent it encourages/forces clear
| patterns. Language with more examples, shorter tokens,
| popularity, etc - doesn't matter at all if the codebase is a
| mess.
|
| Functional languages like Elixir make it very easy to build
| highly structured applications. Each fn takes in a thing and
| returns another. Side effects? What side effects? LLMs can
| follow this function composition pattern all day long. There's
| less complexity, objectively.
|
| But take languages that are less disciplined. Throw in
| arbitrary side effects and hidden control flow and mutable
| state ... the LLM will fail to find an obviously correct
| pattern and guess wildly. In practice, this makes logical bugs
| much more likely. Millions of examples don't help if your
| codebase is a swamp. And languages without said discipline
| often end up in a swamp.
| mrcwinn wrote:
| I absolutely love Elixir, but if this is the bridge you need to
| cross, just write it in Python in the first place.
| dnautics wrote:
| It's 2026 and the LLMs score high on elixir, just write it in
| python and patch it over to elixir gradually
| Towaway69 wrote:
| Or patch it over to python, I assume LLMs are even better at
| python.
| dnautics wrote:
| Don't assume. Empirically, they are not. (This post Feb
| 2026 may change in future yadda yadda)
|
| See: autocodebench
|
| https://github.com/Tencent-
| Hunyuan/AutoCodeBenchmark/tree/ma...
| Towaway69 wrote:
| Reading that made me think how much that might be related
| to Elixir being very similar in syntax to Ruby. Do LLMs
| really differentiate between the two?
|
| Specific studies, as the one quoted, are a long way from
| original real world problems.
| ch4s3 wrote:
| Having written a lot of both languages, I'd be surprised
| if LLMs don't get tripped up on some of Ruby's semantics
| and weird stuff people do with monkey patching. I also
| find Ruby library documentation to be on average pretty
| poor.
| Towaway69 wrote:
| > I also find Ruby library documentation to be on average
| pretty poor.
|
| That surprises me :)
|
| From my time doing Ruby (admittedly a few years back), I
| found libraries were very well documented and tested. But
| put into context of then (not now), documentation and
| testing weren't that popular amongst other programming
| languages. Ruby was definitely one of the drivers for the
| general adaption of TDD principles, for example.
| ch4s3 wrote:
| I think they're often very well tested, but the
| documentation piece has always been lacking compared to
| Elixir.
|
| I used to frequently find myself reading the source code
| of popular libraries or prying into them at runtime.
| There's also no central place or format for documentation
| in ruby. Yes rubydoc.info exists, but it's sort of an
| afterthought. Sidekiq uses a github wiki, Nokogiri has a
| dedicated site, Rails has a dedicated site, Ruby itself
| has yet another site. Some use RDoc, some don't. Or look
| at Devise https://rubydoc.info/github/heartcombo/devise/m
| ain/frames, there's simply nothing documented for most of
| the classes, and good luck finding in the docs where
| `before_action :authenticate_user!` comes from.
| ricketycricket wrote:
| Here are some thoughts on it from Jose Valim:
| https://dashbit.co/blog/why-elixir-best-language-for-ai
|
| LLMs absolutely understand and write good Elixir. I've
| done complex OTP and distributed work in tandem with
| Sonnet/Opus and they understand it well and happily keep
| up. All the Elixir constructs distinct from ruby are well
| applied: pipes, multiple function clauses, pattern
| matching, etc.
|
| I can say that anecdotally, CC/Codex are significantly
| more accurate and faster working with our 250K lines of
| Elixir than our 25K lines of JS (though not typescript).
| d4mi3n wrote:
| I suspect this is partly due to the quality of
| documentation for Elixir, Erlang, and BEAM. The OTP
| documentation has been around for a long time and has
| been excellently written. Erlang/Elixer doc gen outputs
| function signatures, arity, and both Elixir and Erlang
| handle concepts like function overloading in very
| explicit, well-defined ways.
| bnchrch wrote:
| Thats a large reason for sure!
|
| I'd layer in a few more
|
| * Largely stable and unchanged language through out its
| whole existance
|
| * Authorship is largely senior engineers so the code you
| train on is high quality
|
| * Relatively low number of abstractions in comparisson to
| other languages. Meaning there's less ways to do one
| thing.
|
| * Functional Programming style pushes down hidden state,
| which lowers the complexity when understanding how a
| slice of a system works, and the likelyhood you introduce
| a bug
| OkayPhysicist wrote:
| I suspect the biggest advantage Elixir has is the
| relative quality of the publicly available code.
| Approximately no one has Elixir as their first
| programming language, which keeps a lot of the absolute
| trash-tier code that we all make when first learning to
| program out of the training set. If you look at languages
| that are often people's first (Python, JavaScript, Java),
| only Java has an above average score. Of those three,
| Java's significantly more likely to be taught in a
| structured learning environment, compared to kids winging
| it with the other two.
|
| (And Elixir's relationship to Ruby is pretty overstated,
| IMO. There's definitely inspiration, but the OO-FP jump
| is a makes the differences pretty extreme)
| Towaway69 wrote:
| Agree with the quality level but there are other
| languages where that is also the case: Erlang for example
| is probably one of those languages.
|
| > Elixir's relationship to Ruby is pretty overstated
|
| Perhaps I am actually am over thinking this. Elixir has
| probably diverged enough from Ruby (e.g. defmodule, pipe
| operators, :atom syntax) for LLMs to notice the
| difference between the two. But it does open the
| question, though, how does an LLM actually recognise the
| difference in code blocks in its training data.
|
| There are probably many more programming languages where
| similarities exist.
| dnautics wrote:
| Here are my thoughts:
|
| https://m.youtube.com/watch?v=YZa5GqrZeao
| victorbjorklund wrote:
| So if your app is 99% elixir but 1% is easier in Python because
| a lib you should rewrite the whole app? Makes no sense. Do you
| think Python devs rewrite everything in C if they have a small
| part that needs to use C instead of Python?
| rekoros wrote:
| Oban is great!
| Kaliboy wrote:
| This is a similar concept to Faktory, which uses a built in Redis
| server to manage shared job state.
|
| You then implement workers in your language of choice and
| subscribe to queues.
|
| Very interesting though, the article mentioned a few things I
| hadn't considered before like shared access to one database from
| multiple (different) apps.
|
| I wonder how database schema state is handled in a case like
| that. And CI/CD.
___________________________________________________________________
(page generated 2026-02-19 23:01 UTC)