[HN Gopher] Type information for faster Python C extensions
___________________________________________________________________
Type information for faster Python C extensions
Author : ingve
Score : 96 points
Date : 2024-01-14 09:48 UTC (13 hours ago)
(HTM) web link (bernsteinbear.com)
(TXT) w3m dump (bernsteinbear.com)
| lifthrasiir wrote:
| Love the idea of abusing the method name to extend an otherwise
| non-extensible interface. :-) That said, you will want a proper,
| future-proof design if you are really going to pursue that. An
| easy way would be to put a single magic number directly before
| the method name. struct PyPyTypedMethodMetadata
| { int arg_type; int ret_type;
| void* underlying_func; int magic; // <--
| const char ml_name[100]; };
|
| Now either future version of PyPy or other implementation can use
| a different magic number to evolve that interface. (If you want
| to be even more careful, put another flag before `magic` and use
| it as a bitmap to the extension pointers... But that should be
| more or less covered by `ml_flags` instead.)
|
| Or even more conservatively, you can always require a method with
| `METH_TYPED` to come with a publicly visible symbol
| `_PyPyTyped_<type name>_<method name>` or similar. (It is also
| possible to disambiguate duplicate names if it becomes a bigger
| concern.) It would be even better if you can do that without
| `METH_TYPED` at all, e.g. by listing all public symbols.
| tekknolagi wrote:
| This is a good idea!
| lifthrasiir wrote:
| After some more pondering, I have another question: how about
| slot methods? I'm sure that you don't really have anything
| comparable to the method name if slots, and more recent
| `PyType_Slot` doesn't help either (unless PyPy starts to
| define its own slot ID). It looks like that there should be
| some centralized extension point instead of per-method
| extensions. Extending either `PyModuleDef.m_methods` or
| `PyModuleDef.m_slots`, by putting metadata before their first
| entry, seems doable. Discovering their presence is another
| problem, though.
| dwattttt wrote:
| Sounds like a job for name mangling
| tekknolagi wrote:
| Oh hey, thanks for posting. Curious what you all think!
| cb321 wrote:
| I think many (most?) CPython extensions using the pure C API
| could be profitably re-done as Cython modules. I once almost
| did this for NumPy when Python3 compatibility was too slow in
| coming. I think a PyPyJIT-CythonAOT system would be a potent
| tonic against Python slowness { personally I prefer just going
| all the way to Nim, though that often entails small
| ecosystem/adoption-entailed work }. I also suspect there is
| prior art on this in the Common Lisp (a compiled, gradually
| typed system) world, possibly as interop between Lisp systems.
| ashvardanian wrote:
| Lower latency native calls in Python would be extremely useful,
| thank you for your work! Is the following GitHub issue the
| right place to monitor progress? https://github.com/faster-
| cpython/ideas/issues/546
|
| I'm open to doing some benchmarking. Several of my libraries
| have pure CPython bindings (StringZilla, UCall, SimSIMD), and
| all perform low-latency SIMD-accelerated ops, so might be a
| good testing ground :)
| tekknolagi wrote:
| It's probably a good "ideas issue" but the real commentary
| will end up happening in a python/cpython issue/PR. They will
| get cross-linked eventually.
___________________________________________________________________
(page generated 2024-01-14 23:01 UTC)