[HN Gopher] Customize Python dependency resolution with machine ...
       ___________________________________________________________________
        
       Customize Python dependency resolution with machine learning
        
       Author : BerislavLopac
       Score  : 24 points
       Date   : 2021-11-24 09:36 UTC (1 days ago)
        
 (HTM) web link (developers.redhat.com)
 (TXT) w3m dump (developers.redhat.com)
        
       | roganartu wrote:
       | > Keeping the dependency information in a database, which is
       | queried during the resolution process, allows us to choose
       | dependencies using criteria specified by the developer instead of
       | merely importing the latest possible versions, as pip's
       | backtracking algorithm does. You can specify quality criteria
       | depending on the application's traits and environment. For
       | instance, applications deployed to production environments must
       | be secure, so it is important that dependencies do not introduce
       | vulnerabilities. When a data scientist trains a machine learning
       | model in an isolated environment, however, it is acceptable to
       | use dependency versions that are vulnerable but offer a
       | performance gain, thus saving time and resources.
       | 
       | This seems like a really bad idea to me. I could understand and
       | perhaps get behind the idea that you might use something like
       | this to find the optimal version of a package to use in a given
       | project, but unexpected differences between your development
       | environment and production are a common source of outages.
       | 
       | It also requires using a different package manager called Thamos:
       | https://thoth-station.ninja/docs/developers/thamos/. This tool
       | then outputs requirements files compatible with Pipenv, pip, or
       | pip-tools (though notably not Poetry).
       | 
       | That being said, all of the examples and config seems very
       | centered around ML use cases, with the Thamos config accepting
       | settings for OS, cpu, and cuda versions. Is variance in
       | performance between otherwise-compatible versions of ML packages
       | really that big a problem?
        
       | akx wrote:
       | Um...
       | 
       | > The Python Packaging Authority (PyPA), along with the Python
       | community, is working on an endpoint to provide the dependency
       | information.
       | 
       | So what is the `requires_dist` key in e.g.
       | https://pypi.org/pypi/Django/3.2/json ?
       | 
       | (My experimental dependency locking tool Pipimi
       | (https://github.com/akx/pipimi/blob/f055b0c0/pipimi.py#L43-L5...)
       | uses that endpoint.)
        
         | bblommers wrote:
         | That endpoint is not guaranteed to be correct. There's a Github
         | thread [1] with more background info, if you're interested.
         | 
         | [1] https://github.com/pypa/warehouse/issues/474
        
       | dgan wrote:
       | Hm bike shedding, but "thoth" is a horrible name for anyone who
       | isn't a proficient English speaker. I honestly can't pronounce it
        
         | kortex wrote:
         | Thoth, /thoYth/, rhymes with oath. Not sure if /oY/ renders
         | here, it's the near-close back rounded vowel or "horseshoe".
         | 
         | Normally yes, /th/, the dental fricative (commonly "th" in
         | English) is uncommon as far as language sounds goes, and
         | particularly tricky to voice if it's not in phonemic inventory.
         | 
         | But in this case, /toet/ or /toYt/, "tote", like the container,
         | rhymes with goat, is also an acceptable pronunciation.
        
         | marginalia_nu wrote:
         | Thoth is the name of an Egyptian god, not really an English
         | word.
        
         | geofft wrote:
         | It's the English spelling of the Greek name for an Egyptian
         | god: https://en.wikipedia.org/wiki/Thoth
         | 
         | The actual Egyptian pronunciation, as best as we can
         | reconstruct it, uses sounds that didn't exist in either ancient
         | Greek or even Coptic, and don't quite exist in modern English
         | either. I think you'd be entirely justified using another sound
         | if your language doesn't have English "th".
        
       | deycallmeajay wrote:
       | Yeah this sounds like a terrible idea. The current goal is to
       | build reproducible and hermetic builds. By adding more complexity
       | it'll be much more difficult to get the same artifact, build
       | after build as well as give another method for attackers to
       | achieve supply chain injections.
        
       | benjamir wrote:
       | Yeah: Use even more complexity to build software. How about
       | taming software? ( _duckandcover_ )
        
         | sigmonsays wrote:
         | This is honestly a bad joke right? Python packaging could not
         | possibly get worse.... Or could it
        
           | quinnftw wrote:
           | It could: What if a machine learning algorithm picked each of
           | your dependency versions for you ~magically~
        
       ___________________________________________________________________
       (page generated 2021-11-25 23:01 UTC)