[HN Gopher] Superfunctions: A universal solution against sync/as...
___________________________________________________________________
Superfunctions: A universal solution against sync/async
fragmentation in Python
Author : pomponchik
Score : 14 points
Date : 2025-07-21 11:39 UTC (3 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| pomponchik wrote:
| Many old Python libraries got their mirrored async reflections
| after the popularization of asynchrony. As a result, the entire
| Python ecosystem was duplicated. Superfunctions are the first
| solution to this problem, which allows you to partially eliminate
| code duplication by giving the client the opportunity to choose
| (similar to how it is done in Zig) whether to use the regular or
| asynchronous version of the function.
| nine_k wrote:
| But AFAICT in Zig you don't have to have async and sync
| _versions_. Instead, the runtime may choose to interpret `try`
| as an async or as a synchronous call, the latter is equivalent
| to a future / promise that resolves before the next statement
| [1]. This is a sane approach.
|
| Having separate sync / async versions that look like the same
| function is a great way to introduce subtle bugs.
|
| [1]: https://kristoff.it/blog/zig-new-async-io/
| anon291 wrote:
| So much work to implement the monad type class
| zbentley wrote:
| Not really; the problem is that languages with IO monads often
| provide a runtime that can schedule IO-ful things concurrently
| (or, in Haskell's case, lazily) based on the type. Python has
| no such scheduler; users have to run their own in the form of
| an async-capable event loop or a sequential
| (threadpool/processpool) executor for blocking code.
|
| Because of that missing runtime for scheduling and evaluating
| IO-ful things, tools like superfunctions are necessary.
|
| In other words: IO monads are only as useful as the thing that
| evaluates them; Python doesn't have a built-in way to do that,
| so people have to make code that looks "upward" to determine
| what kind of IO behavior
| (blocking/nonblocking/concurrent/lazy/etc.) is needed.
| OutOfHere wrote:
| I advise users to just abandon async in Python for long term
| success because the future of Python is free-threaded, and async
| is inherently single-threaded. Even if you don't need multiple
| threads now, your CPU will thank you later when you do, saving
| you a full rewrite. With Python 3.14, free-threading is an
| established optional feature of Python.
| zbentley wrote:
| The two don't really compete, because async/await is primarily
| about parallelizing IO.
|
| If I want to (say) probe a dozen URLs for liveness in parallel,
| or write data to/from thousands of client sockets from my
| webserver, doing that with threads--especially free-threaded
| Python threads, which are still quite lock-happy inside the
| interpreter, GIL or not--has a very quickly-noticeable
| performance and resource cost.
|
| Async/await's primary utility is that of a capable utility
| interface for making I/O concurrent (and parallel as well, in
| many cases), regardless of whether threads are in use.
|
| Hell, even golang multiplexes concurrent goroutines' threads
| onto concurrent IO schedulers behind the scenes, as does Java's
| NIO, Erlang/BEAM, and many many similar systems.
| PaulHoule wrote:
| People who talk about there being a hard line between
| parallelism and concurrency are always writing code with race
| conditions that they deny exist or writing code with
| performance bottlenecks they can't understand because they
| deny they exist.
|
| I like working in Java because you can use the same threading
| primitives for both and have systems that work well in both
| IO-dominated and CPU-dominated regimes which sometimes happen
| _in the same application_ under different conditions.
|
| Personally there are enough details to work out that we might
| be up to Python 3.24 when you can really count on all your
| dependncies to be thread safe. One of the reasons Java has
| been successful is the extreme xenophobia (not to mention
| painful JNI) which meant we re-implemented stuff to be
| thread-safe in pure Java as opposed to sucking in a lot of
| C/C++ stuff which will never be thread safe.
| rsyring wrote:
| ~my_superfunction() #> so, it's just usual function!
|
| > Yes, the tilde syntax simply means putting the ~ symbol in
| front of the function name when calling it.
|
| There's a way to work around that but...
|
| > The fact is that this mode uses a special trick with a
| reference counter, a special mechanism inside the interpreter
| that cleans up memory. When there is no reference to an object,
| the interpreter deletes it, and you can link your callback to
| this process. It is inside such a callback that the contents of
| your function are actually executed. This imposes some
| restrictions on you:
|
| > - You cannot use the return values from this function in any
| way...
|
| > - Exceptions will not work normally inside this function...
|
| Ummm...I'm maybe not the target audience for this library.
| But...no. Just no.
| PaulHoule wrote:
| It strikes me as a worst of all worlds solution where it is
| awkward to write superfunctions (gotta write all those
| conditionals) and awkward to use them (gotta call fn.something())
|
| For async to be useful you've got to be accessing the network or
| a pipe or sleep so you have some context, you might as well
| encapsulate this context in an object and if you're doing that
| the object is going to look like class
| base_class: @maybe_async def
| api_call(parameters) ... transform the parameters for
| an http call ... response =
| maybe_await(self.http_call(**http_parameters)) ...
| transform the response to a result ... return result
|
| almost every time where I was wishing I could have sync and async
| generated from the same source it was some kind of wrapper for an
| http API where everything had the same structure -- and these
| things can be a huge amount of code because the http API has a
| huge number of calls but the code is all the same structure.
___________________________________________________________________
(page generated 2025-07-24 23:00 UTC)