[HN Gopher] Calling Rust from Python
___________________________________________________________________
Calling Rust from Python
Author : RebootStr
Score : 52 points
Date : 2023-10-08 16:34 UTC (6 hours ago)
(HTM) web link (blog.frankel.ch)
(TXT) w3m dump (blog.frankel.ch)
| woodruffw wrote:
| As others have pointed out: none of these three "generic" methods
| is appropriate when a more precise one (such as dedicated
| bindings between a pair of languages) exists.
|
| PyO3 is practical evidence of Rust's claims around holistic
| security: a user can write safe Rust _and_ safe Python without
| having to directly unsafely traverse the C ABI between them, or
| take a potentially unacceptable performance hit from RPC or IPC.
| pmkelly4444 wrote:
| check out uniffi for a complete workflow for FFI in many
| languages: https://github.com/mozilla/uniffi-rs
| agalunar wrote:
| Small note for the author: I was a bit confused at first by the
| use of "Nix-based" to mean POSIX-compliant [1] since Nix is the
| name of a package manager that NixOS is based on. I've only seen
| "*nix" to refer to Unix-like systems.
|
| [1] Which seems like the right condition here? since I believe
| Unix domain sockets are required by POSIX.
| guitarbill wrote:
| I would not recommend FFI + ctypes. Maintaining the bindings is
| tedious and error-prone. Also, Rust FFI/unsafe can be tricky even
| for experienced Rust devs.
|
| Instead PyO3 [1] lets you "write a native Python module in Rust",
| and it works great. A much better choice IMO.
|
| [1] https://github.com/PyO3/pyo3
| singhrac wrote:
| Absolutely, I was surprised anyone would recommend ctypes for
| Python/Rust today. I've written a few Python libraries using
| Rust to do the slow parts, and PyO3/maturin is a great
| experience.
| anonacct37 wrote:
| Even a decade ago I was seeing projects stop using ctypes and
| instead use things like cffi. Honestly I couldn't tell you
| exactly why ctypes is so bad but I do know that cffi
| integrates a little closer with the c compiler and has better
| header support.
| abdullahkhalids wrote:
| PyO3 is great. I have been using it to rewrite some slow python
| code. My only complaint is that documentation is a bit rough.
| For someone who understands python, but not rust, there was a
| pretty big hill to climb to understand type conversions.
|
| There are also things I am still struggling with. For example,
| there is a rust function I want to repeatedly call from python,
| that takes in a large unchanging python array. How can I pass
| the array from python to rust only once, i.e do the type
| conversion just once, so I don't have to pay that cost
| repeatedly.
| dsheets wrote:
| I recommend creating a pyclass over a struct that stores the
| converted data. Instantiate it once and then repeatedly call
| a function on it in a pymethods impl with the parameters that
| differ.
| forrestthewoods wrote:
| > I would not recommend FFI + ctypes.
|
| Well it depends. C is the lingua franca of Computer Science.
| These days I write a lot of code that needs to be accessible to
| a variety of languages and environments. Python, C#, C++,
| Matlab, Unity, Unreal, etc.
|
| Write a C API and your code can be used _anywhere_. Now if you
| know you only ever need to support Python then sure look into
| PyO3. But if you have mixed use you can't be the flexibility of
| a C API.
| guitarbill wrote:
| Honestly though, I would likely still choose to implement the
| logic in Rust as a crate/library in a workspace, the
| (C)Python extensions as a PyO3 dylib/separate crate, and a C
| FFI dylib as yet another crate.
|
| The benefit is that you only need to ship the CPython
| extension, you can build it with cargo, and everything's in
| one place. I've maintained bindings for Python and C# to
| native libraries, and it's usually a pain for various
| reasons.
|
| The C API might be able to be used anywhere, but that doesn't
| mean it's ergonomic, easy to use, or safe.
| forrestthewoods wrote:
| > The C API might be able to be used anywhere, but that
| doesn't mean it's ergonomic, easy to use, or safe.
|
| Definitely true. That said I think there is an under
| appreciated beautiy to a clean C API.
|
| It's easy to "upgrade" a C API with per-
| language/environment bindings. It can be near impossible to
| "downgrade" a high-level API to C.
| dang wrote:
| Some related past threads:
|
| _How to write Python extensions in Rust with PyO3_ -
| https://news.ycombinator.com/item?id=34968186 - Feb 2023 (9
| comments)
|
| _PyO3 - Python Extensions in Rust_ -
| https://news.ycombinator.com/item?id=32247452 - July 2022 (1
| comment)
|
| _Calling Rust from Python using PyO3_ -
| https://news.ycombinator.com/item?id=29368530 - Nov 2021 (49
| comments)
|
| _PyO3: Rust Bindings for the Python Interpreter_ -
| https://news.ycombinator.com/item?id=25956502 - Jan 2021 (77
| comments)
|
| _Writing Python Extensions in Rust Using PyO3_ -
| https://news.ycombinator.com/item?id=17423013 - June 2018 (1
| comment)
|
| _PyO3: Python Rust binding_ -
| https://news.ycombinator.com/item?id=14859844 - July 2017 (15
| comments)
|
| _(First Release) PyO3: a Python-Rust Binding Library_ -
| https://news.ycombinator.com/item?id=14846606 - July 2017 (1
| comment)
| nhatcher wrote:
| I think you are right. But as a proof of concept is an
| interesting read. I've work extensively with Rust and PyO3 and
| Maturin and even for a medium to large project is great. I
| wrote a bit about that here:
|
| https://www.nhatcher.com/post/rust-in-anger/
| guitarbill wrote:
| Agreed. It's also instructive to look at how C extensions are
| written for the CPython.
|
| I wouldn't necessarily expect a demo; but not mentioning it
| at all seems like it could lead readers down a path where
| they choose the "most performant" option presented (FFI +
| ctypes). The post has a "To go further:" link section where
| even a quick link to PyO3 would go a long way. So I thought
| I'd mention the downsides and an alternative on HN.
| wiktor-k wrote:
| I'd also recommend PyO3 and Maturin. The amount of help these
| crates give is mind boggling (automatic type marshaling and
| even github CI jobs for creating cross platform precompiled
| wheels).
|
| I've created a library using this crate
| (https://github.com/wiktor-k/pysequoia/) and most of the time
| I could just focus on the problem domain instead of technical
| details of the bindings.
|
| There are just a couple of smaller issues (eg. Python to Rust
| async is not built in) but overall it's really nice.
| ynik wrote:
| Right now FFI + ctypes has one big advantage: it supports true
| parallelism with GIL-per-subinterpreter in Python 3.12. AFAIK
| all the higher-level binding libraries/tools (PyO3, pybind11,
| nanobind, Cython) don't support this yet; and in fact can't
| really support it without API-breaking fundamental design
| changes.
| rdedev wrote:
| But wasn't the whole point to drop down to a lower language,
| release the GIL and then do parallel processing? Gil per
| subinterpretee makes things easy python side but I'm not sure
| how much of it's relevant if you are using something like
| pyO3
| Yoric wrote:
| Do you know what kind of design changes would be needed?
| xwowsersx wrote:
| Just looking at the function signatures there:
| fn string_sum(_py: Python<'_>, m: &PyModule)
|
| is the idea that the `Python` type represents the GIL and
| you're actually acquiring it from within your Rust application?
| guitarbill wrote:
| Edit: Yes it does, as ynik rightly points out, and documented
| here https://pyo3.rs/v0.19.2/types#gil-lifetimes-mutability-
| and-p... or here https://docs.rs/pyo3/0.19.2/pyo3/marker/stru
| ct.Python.html#o...
| ynik wrote:
| The Python<'_> type appearing as a parameter means the
| caller has already acquired the GIL for you. The GIL is
| held when Python calls into extension modules, and PyO3
| doesn't automatically release it (but you can do so
| yourself).
|
| It's essentially a dummy parameter that serves as proof
| that the GIL is held. You need to provide it to PyO3 when
| calling back into the Python runtime, that way the compiler
| ensures the GIL is still held where required.
| sonthonax wrote:
| Isn't PyO3 just a set of macros and helpers around FFI and
| ctypes?
| guitarbill wrote:
| "just" undersells it a bit, mainly around type conversions.
|
| Also, there is no ctypes involved, as the result is a CPython
| extension that just loads on the Python side.
| masklinn wrote:
| In the same sense that rust is just a set of macros and
| helpers around assembly.
|
| Also no ctypes.
___________________________________________________________________
(page generated 2023-10-08 23:00 UTC)