[HN Gopher] Embedding Python in Elixir, it's fine
___________________________________________________________________
Embedding Python in Elixir, it's fine
Author : arathunku
Score : 231 points
Date : 2025-02-25 12:53 UTC (10 hours ago)
(HTM) web link (dashbit.co)
(TXT) w3m dump (dashbit.co)
| pmarreck wrote:
| Looks like a very cool way to interop with Python from Elixir
| without maintaining a separate Python stack (which is a PITA)!
| ddanieltan wrote:
| "Your scientists were so preoccupied with whether or not they
| could, they didn't stop to think if they should"
|
| just kidding, this is pretty cool.
| qwertox wrote:
| Great and informative article. Also nice to get an explicit
| mention that this isn't just a subprocess call, but running in
| the same process.
|
| The only thing I'd would have like to see in added would be
| calling a function defined in Python from Elixir, instead of only
| the `Pythonx.eval` example.
|
| The `%{"binary" => binary}` is very telling, but a couple of more
| and different examples would have been nice.
| jwbaldwin wrote:
| I love the initial decision to grow Elixir's ML foundations from
| scratch, but I also love that we now have a really ergonomic way
| to farm out to the fast-moving python libraries
|
| > Also, it conveniently handles conversion between Elixir and
| Python data structures, bubbles Python exceptions and captures
| standard output
|
| Sooo nice
| behnamoh wrote:
| Elixir has some features I wish Python had:
|
| - atoms
|
| - everything (or most things) is a macro, even def, etc.
|
| - pipes |>, and no, I don't want to write a "pipe" class in
| Python to use it like pipe(foo, bar, ...). 90% of the |> power
| comes from its 'flow' programming style.
|
| - true immutability
|
| - true parallelism and concurrency thanks to the supervision
| trees
|
| - hot code reloading (you recompile the app WHILE it's running)
|
| - fault tolerance (again, thanks for supervision trees)
| ch4s3 wrote:
| Mix is also so much better than anything python has to offer in
| terms of build/dependency tooling.
| mcintyre1994 wrote:
| Jupyter might have fixed this now because it's been a while
| since I used it, but Mix.install inline in Livebook (or any
| CLI script) is so much nicer than how installing Python
| dependencies in notebooks was last time I did that too.
| streblo wrote:
| uv for Python is a game changer, better than anything else
| out there and solves a lot of the core problems with
| pip/venv/poetry/pyenv (the list goes on).
| paradox460 wrote:
| I feel like you can write some variant of this comment
| every few years and just add the previous "best" to the
| front of the stack of things it's better than.
| fire_lake wrote:
| It's true - people were saying that Poetry solves these
| problems for ages. Maybe uv does? I'll wait and see.
| davidw wrote:
| Coming from Erlang, I think macros are one of the things I'm
| ambivalent about in Elixir. There are a bunch of actual
| improvements besides just the syntax itself in Elixir, like
| string handling, but things like macros in Ecto ... not yet a
| fan of that.
| notpublic wrote:
| > ..but things like macros in Ecto....
|
| Well, The whole language itself is built on macros. The
| following series of articles certainly helped me stop
| worrying and love the macros..
|
| https://www.theerlangelist.com/article/macros_1
|
| Some interesting insights from the article: "Elixir itself is
| heavily powered by macros. Many constructs, such as
| defmodule, def, if, unless, and even defmacro[1] are actually
| macros...."
|
| [1] https://github.com/elixir-
| lang/elixir/blob/v1.18.2/lib/elixi...
| shiandow wrote:
| You _can_ abuse the '>>' notation in python for pipes (or you
| could use |, I suppose), but you'll have to deal with
| whitespace shenanigans. I'm also not entirely sure about the
| order of evaluation. And you'll need to do partial function
| application by hand if you want that (though it is possible to
| write a meta function for that).
|
| So one _could_ write class Piped:
| def __init__(self, value): self.value = value
| def __or__(self, func): return
| Piped(func(self.value)) def
| __repr__(self): return f"Piped({self.value!r})"
| Piped('test') | str.upper | (lambda x: x.replace('T', 't')) |
| "prefix_".__add__ # => prefix_tESt
|
| but whether that is a good idea is a whole different matter.
| sbrother wrote:
| Apache Beam in Python does this, with code like
| counts = ( lines | 'Split' >> (
| beam.FlatMap( lambda x:
| re.findall(r'[A-Za-z\']+', x)).with_output_types(str))
| | 'PairWithOne' >> beam.Map(lambda x: (x, 1)) |
| 'GroupAndSum' >> beam.CombinePerKey(sum))
|
| I'm not sure how I feel about it, other than the fact that
| I'd 100x rather write Beam pipelines in basically any other
| language. But that's about more than syntax.
| ejs wrote:
| I love this, I've primarily been working in Elixir for a few
| years now and this is neat to see!
| jarpineh wrote:
| At first read this seems really promising. Getting into
| Elixir/Erlang ecosystem from Python has seemed too hard to take
| the time. And when there I wouldn't be able to leverage all the
| Python stuff I've learned. With Pythonx gradual learning seems
| now much more achievable.
|
| It wasn't mentioned in the article, but there's older blog post
| on fly.io [1] about live book, GPUs, and their FLAME serverless
| pattern [2]. Since there seems to be some common ground between
| these companies I'm now hoping Pythonx support is coming to FLAME
| enabled Erlang VM. I'm just going off from the blog posts, and am
| probably using wrong terminology here.
|
| For Python's GIL problem mentioned in the article I wonder if
| they have experimented with free threading [3].
|
| [1] https://fly.io/blog/ai-gpu-clusters-from-your-laptop-
| liveboo...
|
| [2] https://fly.io/blog/rethinking-serverless-with-flame/
|
| [3] https://docs.python.org/3/howto/free-threading-python.html
| lawik wrote:
| FLAME runs the same code base on another machine. FLAME with
| Pythonx should just work. FLAME is a set of nice abstractions
| on top of a completely regular Erlang VM.
|
| Chris Grainger who pushed for the value of Python in Livebook
| has given at least two talks about the power and value of
| FLAME.
|
| And of course Chris McCord (creator of Phoenix and FLAME) works
| at Fly and collaborates closely with Dashbit who do Livebook
| and all that.
|
| These are some of the benefits of a cohesive ecosystem.
| Something I enjoy a lot in Elixir. All these efforts are
| aligned. There is nothing weird going on, no special work you
| need to do.
| bicx wrote:
| For Livebook, this looks really cool. Love that it calls CPython
| directly via C++ NIFS in Elixir and returns Elixir-native data
| structures. That's a lot cleaner than interacting with Python in
| Elixir via Ports, which is essentially executing a `python`
| command under the hood.
|
| For production servers, Pythonx is a bit more risky (and the
| developers aren't claiming it's the right tool for this use
| case). Because it's running on the same OS process as your Elixir
| app, you bypass the failure recovery that makes an Elixir/BEAM
| application so powerful.
|
| Normally, an Elixir app has a supervision tree that can
| gracefully handle failures of its own BEAM processes (an internal
| concurrency unit -- kind like a synthetic OS process) and keep
| the rest of the app's processes running. That's one of the big
| selling points of languages like Elixir, Erlang, and Gleam that
| build upon the BEAM architecture.
|
| Because it uses NIFs (natively-implemented functions), an
| unhandled exception in Pythonx would take down your whole OS
| process along with all other BEAM processes, making your
| supervision tree a bit worthless in that regard.
|
| There are cases when NIFs are super helpful (for instance,
| Rustler is a popular NIF wrapper for Rust in Elixir), but you
| have to architect around the fact that it could take down the
| whole app. Using Ports (Erlang and Elixir's long-standing
| external execution handler) to run other native code like Python
| or Rust is less risky in this respect because the non-Elixir code
| it's still running in a separate OS process.
| chefandy wrote:
| I hadn't heard of gleam. Looks cool! I like working with elixir
| in a lot of ways but never was a Ruby guy, and I think I'd
| prefer the C-style syntax.
| giancarlostoro wrote:
| I'm more of a Python and C# kind of guy, so Elixir never
| really hit the itch for me, but Gleam definitely does. One of
| these days I'll take a crack to see how I can use Gleam with
| Phoenix.
| chefandy wrote:
| I've been mostly in Python, C# and C++ for the past decade
| or so but got into Elixir as my first functional language.
| Never got comfy with the syntax but dig how everything
| flows. Looking forward to digging into Gleam.
| neonsunset wrote:
| If you liked Elixir but found it too "exotic" you may
| find F# enjoyable instead - it's a bit like Elixir but
| with a very powerful, gradually typed and fully inferred
| type system and has access to the rest of .NET. Excellent
| for scripting, data analysis and modeling of complex
| business domains. It's also very easy to integrate a new
| F# project into existing C# solution, and it ships with
| the SDK and is likely supported by all the tools you're
| already using. F# is also 5 to 10 times more CPU and
| memory-efficient.
|
| (F# is one of the languages Elixir was influenced by and
| it is where Elixir got the pipe operator from)
| ikety wrote:
| My current favorite language, just no time to finish my gleam
| projects.
| thibaut_barrere wrote:
| One possibility for production use (in case there is a big
| value) is to split the nodes into one "front" node which
| requires strong uptime, and a "worker" node which is designed
| to support rare crashes gracefully, in a way that does not
| impact the front.
|
| This is what we use at https://transport.data.gouv.fr/ (the
| French National Access Point for transportation data - more
| background at https://elixir-
| lang.org/blog/2021/11/10/embracing-open-data-...).
|
| Note that we're not using Pythonx, but running some memory
| hungry processes which can sometime take the worker node down.
| giancarlostoro wrote:
| Do any of them communicate with the BEAM? There used to be a Go
| based implementation of the BEAM that allowed you to drop-in
| with Go, I have to wonder if this could be done with Python so
| it doesn't interfere with what the BEAM is good that and lets
| Python code remain as-is.
| lawik wrote:
| There are several libraries that allow a Python program to
| communicate with an Erlang program using Erlang Term Format
| and such.
|
| This approach targets more performance-sensitive cases with
| stuff like passing data frames around and vectors/matrices
| that are costly to serialize/deserialize a lot of the time.
|
| And it seems to make for a tighter integration.
| alienthrowaway wrote:
| > Because it uses NIFs (natively-implemented functions), an
| unhandled exception in Pythonx would take down your whole OS
| process along with all other BEAM processes, making your
| supervision tree a bit worthless in that regard.
|
| What's the Elixir equivalent if "Pythonic"? An architecture
| that allows a NIF to take down your entire supervision tree is
| the opposite of that, as it defeats a the stacks' philosophy.
|
| The best practice for integrating Python into Elixir or Erlang
| would be to have an assigned genserver, or other supervision-
| tree element - responsible for hosting the Python NIF(s), and
| the design should allow for each branch or leaf of that tree to
| be killed/restarted safely, with no loss of state. BEAM message
| passing is cheap
| bicx wrote:
| That's the thing though: a NIF execution isn't confined to
| the the BEAM process by its nature. From the Erlang docs:
|
| > As a NIF library is dynamically linked into the emulator
| process, this is the fastest way of calling C-code from
| Erlang (alongside port drivers). Calling NIFs requires no
| context switches. But it is also the least safe, because a
| crash in a NIF brings the emulator down too.
| (https://www.erlang.org/doc/system/nif.html)
|
| The emulator in this context is the BEAM VM that is running
| the whole application (including the supervisors).
|
| Apparently Rustler has a way of running Rust NIFs but
| capturing Rust panics before they trickle down and crash the
| whole BEAM VM, but that seems like more of a Rust trick that
| Pythonx likely doesn't have.
|
| The tl;dr is that NIFs are risky by default, and not
| really... Elixironic?
| cpursley wrote:
| Really glad to see this, Elixir has languished in the AI wars
| despite being a better fit than JavaScript and Python.
| tombert wrote:
| Forgive some ignorance on this; why is Elixir a better fit for
| AI than Python or JavaScript? I'm not disagreeing, I've just
| never heard that, I didn't think that Elixir had good linear
| algebra libraries like NumPy.
| jyscao wrote:
| It does now with Nx
| cpursley wrote:
| Sorry, I should have been more explicit: better for on the
| user facing implementation side (concurrency, streaming data,
| molding agent state, etc) vs the training side of things. If
| that makes sense.
| tombert wrote:
| Ah, fair enough. I've not done much with Elixir but I have
| done a fair amount with Erlang and you certainly don't need
| to sell me on how great it is for concurrency and
| distributed stuff.
| solid_fuel wrote:
| I've been actively using elixir for ML at work, and I would
| say it's a solid choice.
|
| The downside - unfortunately while bumblebee, Axon, and Nx
| are libraries that seem to have a fantastically engineered
| base most of the latest models don't have native elixir
| implementations yet and making my own is a little beyond my
| skill still. So a lot of the models you can easily run are
| older.
|
| But the advantages - easy long running processes, great
| multiprocessing support, solid error handling and recovery -
| all pair very well with AI systems.
|
| For example, it's very easy to make an application that grabs
| files, caches them locally, and runs ML tasks against them.
| You can use process monitoring and linking to manage the
| locally cached files, and there's no runtime duration limit
| like you might hit in a serverless system like lambda.
| Interprocess messaging means you can easily run ML in a
| background task and stream results asynchronously to a user.
| Additionally, logs are automatically streamed to the parent
| process and it's easy to tag logs with process metadata, so
| tracking what is going on in your application is dead simple.
|
| That's basically a whole stack for a live ML service with all
| the difficult infrastructure bits already taken care of.
| 4b11b4 wrote:
| yes
| lawik wrote:
| As someone very involved in Elixir and who used to do a lot of
| Python this seems very practical for me. I'm actually even more
| interested in that Fine library for making C++ NIFs easy. That
| seems ridiculously valuable for removing hurdles to building
| library bindings.
| crenwick wrote:
| I feel like this project and blog post was made specifically for
| me. Can't wait to use this, thanks!
| djha-skin wrote:
| Elixir is just Lisp with a facelift[1], and lisps can be built on
| Python[2]. It stands to reason that an elixir-like can be built
| on Python too, so you could embed the Python runtime in Elixir
| but Elixir-likes are used to code for both.
|
| 1: https://wiki.alopex.li/ElixirForCynicalCurmudgeons
|
| 2: https://hylang.org/
| pjmlp wrote:
| In a way Python is a bad Lisp, still looking forward that
| catches up in native code compilation and multiline lambdas.
|
| Could be better, but that is what mainstream gets.
| ch4s3 wrote:
| The operating environment of the BEAM is what's great about
| elixir. Hy still has the GIL.
| chantepierre wrote:
| I love to see "well-known" people in the Elixir community
| endorsing and actively developing that kind of approach. Our VM
| and runtime does so much and is so well suited to orchestrating
| other languages and tech that it sometimes feels there's a
| standard track and an off-road track.
|
| The difference between an off-road "sounds dangerous" idea and
| its safe execution is often only the quantity of work but our
| runtime encourages that. Here, it's a NIF so there's still a bit
| of risk, but it's always possible to spawn a separate BEAM
| instance and distribute yourself with it.
|
| Toy example that illustrates it, first crashing with a NIF that
| is made to segfault : my_nif_app iex --name
| my_app@127.0.0.1 --cookie cookie -S mix
| iex(my_app@127.0.0.1)1> MyNifApp.crash [1] 97437
| segmentation fault
|
| In the second example, we have a "SafeNif" module that spawns
| another elixir node, connects to it, and runs the unsafe
| operation on it. my_nif_app iex --name
| my_app@127.0.0.1 --cookie cookie -S mix
| iex(my_app@127.0.0.1)1> MyNifApp.SafeNif.call(MyNifApp, :crash,
| []) Starting temporary node: safe_nif_4998973
| Starting node with: elixir --name safe_nif_4998973@127.0.0.1
| --cookie :cookie --no-halt /tmp/safe_nif_4998973_init.exs
| Successfully connected to temporary node Calling
| MyNifApp.crash() on temporary node :error
| iex(my_app@127.0.0.1)2>
|
| Thankfully Python, Zig and Rust should be good to go without that
| kind of dance :) .
| tommica wrote:
| Its a neat way to do it - spin a temporary one, which can crash
| all it wants without affecting the other nodes. Fits like a
| glove to BEAM.
| incontrol wrote:
| I was super excited until I read:
|
| "...if you are using this library to integrate with Python, make
| sure it happens in a single Elixir process..."
___________________________________________________________________
(page generated 2025-02-25 23:00 UTC)