[HN Gopher] Hello, HPy
___________________________________________________________________
Hello, HPy
Author : milliams
Score : 166 points
Date : 2021-03-29 17:38 UTC (5 hours ago)
(HTM) web link (hpyproject.org)
(TXT) w3m dump (hpyproject.org)
| dmw_ng wrote:
| This looks like the holy grail of extension development,
| effectively a Python runtime HAL, it's wildly exciting. Getting
| anything to work optimally on PyPy and CPython more or less means
| writing it twice. CFFI overhead on CPython really sucks for
| anything where the runtime of the wrapped function does not
| eclipse the overhead of the binding. HPy looks like a fantastic
| approach.
|
| Only complaint from a quick scan -- carrying over that horrid
| PyArg_Parse() API. It's huge, needs varargs, isn't type safe,
| repeatedly rebuilds strings to look them up in kwargs, and isn't
| even all that convenient to use. Also, if you haven't dealt with
| a SEGV due to PyArg_Parse() before, it means you probably have
| never tried to write an extension
|
| PyPy will become about 1000x more practical if something like
| this ever makes it into Cython. That would make it easy to bring
| over huge swathes of existing extensions like lxml.
| hexo wrote:
| I've always used Cython to make an extension. Speed improvements
| have always been awesome. I'm not sure when I'd want to use HPy
| over Cython. What are your opinions on this, please?
| ngoldbaum wrote:
| I think the idea is for cython to move away from generating
| CPython C API code to generating hpy code in the long run. From
| the point of view of cython users this won't make much
| difference.
| AlexCoventry wrote:
| Is cython still a thing? If so, what's the benefit of this over
| cython?
| ben509 wrote:
| It still is, and Cython is great for accelerating critical
| Python code.
|
| A C extension is far preferable when you want to code in C,
| either to write a new data type[1], or write a Python frontend
| to a C library[2] that is too complex to be well supported by
| simple FFI.
|
| I think people use Cython more internally when they value the
| maintainability of "mostly Python" over the fact that it's
| slower than what native C would get them.
|
| [1]: https://github.com/tobgu/pyrsistent
|
| [2]: https://github.com/libgit2/pygit2
| throwaway894345 wrote:
| > A C extension is far preferable when you want to code in C
|
| Or any other language that supports C FFI.
| dr_zoidberg wrote:
| > over the fact that it's slower than what native C would get
| them.
|
| You have a quote for that? Cython code, in the context of
| "C-extended-python" probably never is slower than native C.
| There has been a lot of work over the years to make sure of
| that.
|
| Now, if you want to compare it to "a C program/library in the
| wild", then we are not comparing apples to apples. The whole
| point of Cython is inserting C code into a Python module, to
| have (high performance) C code inside a Python environment.
| ben509 wrote:
| My source is simply running `cython -a` to see all the
| highlighted lines, and fixing them and observing how many
| hoops I have to jump through to get code that doesn't
| interact with the interpreter.
|
| Cython makes you do more work to avoid the interpreter. C
| makes you do more work to involve the interpreter.
|
| > There has been a lot of work over the years to make sure
| of that.
|
| Some cython code can compile close to the C code you'd
| write.
|
| But that's not how it's likely to be used, because the
| design pushes you in the opposite direction by default, and
| the value of Cython is that it's far faster than vanilla
| Python, and the painless interop.
| eatonphil wrote:
| I love that the debug mode catches missing refcount
| increments/decrement. I hate programming with manual reference
| counting. Looking forward to trying this out.
| chrisseaton wrote:
| We have this same problem in Ruby - implementing the Ruby C
| extension API if you aren't exactly the same as the reference
| Ruby implementation is extremely challenging. I think Charlie
| Nutter proposed something like this for Ruby in the past
| https://github.com/headius/xni.
| kardos wrote:
| This could be big. A strong porting guide will go a long way. I
| wonder to what extent that could be automated?
| stcredzero wrote:
| _HPy provides a new API for extending Python in C. In other
| words, you use #include <hpy.h> instead of #include <Python.h>._
|
| So, Python finally invented Smalltalk prinitives?
| chrisseaton wrote:
| Not sure what you mean - Smalltalk primitives are methods
| implemented by the language implementation. This statement is
| just about the fact that they provide an alternative API via an
| alternate header file.
| zem wrote:
| this is unnecessarily dismissive - you could have said
| "smalltalk primitives are an interesting parallel to this idea"
| and people would have enjoyed following the pointer, rather
| than getting annoyed.
| dralley wrote:
| Anything that can improve the C extension experience is very
| welcome. The current APIs are kind of... rough.
| pansa2 wrote:
| One obvious improvement here is the use of `HPyContext`. This
| allows the Python interpreter state to be encapsulated rather
| than globally-accessible, which would enable multiple
| interpreters in a single process.
|
| Lua's C API is generally considered to be well-designed, and does
| something similar with `lua_State`. How does the HPy API compare
| to the Lua API more generally?
| leecb wrote:
| > enable multiple interpreters in a single process.
|
| CPython (the default interpreter) has had support for multiple
| interpreters in the same process since 1997, but it has only
| been exposed to the C API, and not in the language itself.
| Python 3.10, coming out later this year, will expose multiple
| interpreters in one process (subinterpreters, see PEP 554 [1]).
|
| I'm excited about what could eventually come out of this. If
| there is one GIL per interpreter, we could have something like
| the `multiprocessing` library for parallel execution, but all
| within one process.
|
| 1 https://www.python.org/dev/peps/pep-0554/
| rkeene2 wrote:
| This is basically how the Threads extension [0] works for
| Tcl. Tcl has long supported creating many interpreters per
| thread. The Thread extension exposes some threading commands
| to the interpreter, which lets you create a new
| thread+interp. You can then send messages to that
| interpreter, or pass it file descriptors, but otherwise they
| are fairly isolated (apartment threading model).
|
| [0] https://www.tcl-lang.org/man/tcl8.6/ThreadCmd/thread.htm
| [deleted]
___________________________________________________________________
(page generated 2021-03-29 23:00 UTC)