[HN Gopher] Instrumenting Python GIL with eBPF
       ___________________________________________________________________
        
       Instrumenting Python GIL with eBPF
        
       Author : lukastyrychtr
       Score  : 66 points
       Date   : 2024-07-19 14:31 UTC (8 hours ago)
        
 (HTM) web link (coroot.com)
 (TXT) w3m dump (coroot.com)
        
       | polotics wrote:
       | Wow, 36ms per second is only 3.6% of the time. Python waiting on
       | the GIL is then a pretty overblown problem, as this is not very
       | significant. I wonder if this measure could be run on apps built
       | with various frameworks. I expect that with uvloop and all, the
       | percentage would be even less.
        
         | black_puppydog wrote:
         | It _may_ not be very significant _in this context_.
         | 
         | But the author literally starts by explaining that that's a
         | typical argument _for webservers_ because they 're mostly I/O
         | bound. Anyone working with code that's more CPU bound will have
         | very different numbers, and interpret them differently.
        
           | polotics wrote:
           | are you sure though that being more CPU-bound will imply more
           | waiting on the GIL? CPU-bound python in my experience means
           | libraries, like eg. numpy, that are well-designed and release
           | the GIL.
        
             | fleetfox wrote:
             | If you are interested PEP703 describes the scenarios pretty
             | well: https://peps.python.org/pep-0703/#motivation
        
               | johnjr wrote:
               | I just wrote a post about how the Cpython is much faster
               | without GIL:https://news.ycombinator.com/item?id=40988244
        
               | arp242 wrote:
               | I mean, only the threaded version, which is expected. For
               | tons of cases Python without the GIL is not just slower,
               | but _significantly_ slower;  "somewhere from 30-50%"
               | according to one of the people working on this:
               | https://news.ycombinator.com/item?id=40949628
               | 
               | All of this is why the GIL wasn't removed 20 years ago.
               | There are real trade-offs here.
        
         | memco wrote:
         | That's over 40 minutes of a full day or 5 hours a week spent
         | locking! As the author states it's highly context dependent but
         | in some cases this is a lot of time to do other work if it can
         | be reduced.
        
         | kevingadd wrote:
         | If you're running your application on 100 servers, that's
         | potentially 3-4 entire machines you no longer have to pay
         | EC2/S3/etc fees for, or it's more scaling headroom because
         | you've decreased load across your whole fleet by 3.6%.
         | 
         | It's "small", sure, but in production performance issues are
         | often "death by a thousand cuts" situations, so a 3.6%
         | reduction is a big win compared to optimizations that are often
         | in the 1% range.
        
         | in_a_society wrote:
         | I'm not sure your conclusion is a fair take. In the app I work
         | on, GIL acquisition would easily take 2-3x longer than the
         | postgres queries which would subsequently be issued.
        
       | noident wrote:
       | >Every Python developer has heard about the GIL
       | 
       | Sadly, that is not the world we live in.
       | 
       | I've cleaned up dozens of applications written by people with
       | flawed understandings of threads, multiprocessing, and asyncio. I
       | don't even blame the developers for this; it's a glaring language
       | design problem.
       | 
       | If you need parallelism, Python is not the language you should
       | reach for. Nobody ever takes my word for it until it's release
       | day and the product is a broken pile of spaghetti code.
        
         | simonw wrote:
         | How do you feel about the GIL free option coming in Python
         | 3.13?
        
           | noident wrote:
           | It looks like it will be some time before free-threading can
           | be safely used in production. I don't want to have to worry
           | about whether the underlying C code supports no-GIL mode, or
           | trade off single-threaded performance. [0]
           | 
           | Maybe someday this will make parallelism in Python at least
           | as sane as other languages, but in the meantime, I still want
           | to use a compiled language any time performance matters and
           | wait for the kinks in no-GIL to be ironed out.
           | 
           | [0] https://docs.python.org/3.13/whatsnew/3.13.html#free-
           | threade...
        
         | icedchai wrote:
         | Same. I "inherited" an app doing crazy slow sync IO from their
         | "async" functions... Sure, there was often some async IO mixed
         | in, too, but what's the point?
        
       ___________________________________________________________________
       (page generated 2024-07-19 23:06 UTC)