[HN Gopher] Generate pip requirements.txt file based on imports ...
___________________________________________________________________
Generate pip requirements.txt file based on imports of any project
Author : mooreds
Score : 112 points
Date : 2024-09-30 16:26 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| notnmeyer wrote:
| looks great. similarly this is why i love 'go mod tidy'. just use
| the thing and let the package manager sort it out after the fact.
| FlipFloopDev wrote:
| Surprised I've never seen this, despite it existing for at least
| 9 years??
| mbank wrote:
| I was looking for this and thought I was doing something wrong
| not finding anything... Great job! I do think though a "clean"
| development mode not needing this would be to work with a virgin
| virtual environment starting a new project and running pip freeze
| on that env.
| robblbobbl wrote:
| Nicely done
| Terretta wrote:
| Please consider consolidating python dependency management
| instead of fragmenting it:
|
| https://docs.astral.sh/uv/pip/compile/
|
| In other words, bring the thinking here. Whether it's new
| thinking, or decade old thinking, it's well time to converge.
| We've had decades of trying it like Heinz catsup varieties. Worth
| trying more wood behind fewer arrows.
| Spivak wrote:
| You're just picking a winner like every other Python dependency
| project. If it's not in a PEP do whatever the hell you want.
| Good ideas will get turned into future PEPs for tooling to
| standardize on. uv itself has two separate locking formats.
| halfcat wrote:
| > _If it 's not in a PEP_
|
| Yes exactly
|
| https://xkcd.com/927/
| imjonse wrote:
| One can start non-distruptively with uv by only using it as a
| pipx replacement at first ( uv tool). Nice and fast.
| Der_Einzige wrote:
| Please consider consolidating python dependency management
| instead of fragmenting it:
|
| https://github.com/mamba-org/mamba
|
| In other words, bring the thinking here. Whether it's new
| thinking, or decade old thinking, it's well time to converge.
| We've had decades of trying it like Heinz catsup varieties.
| Worth trying more wood behind fewer arrows.
| kstrauser wrote:
| While I don't know which (if any) Python package manager will
| "win", I'm confident it won't be Conda or similar. I hope and
| pray not, anyway.
| matrss wrote:
| Yes, the entire workflow of conda/mamba is a liability.
| Having environments detached from directories encourages
| bad practices (reusing environments for unrelated things,
| installing everything into the same environment and getting
| very broken dependency resolutions, that stuff) and it
| doesn't even have a lock file format, so environments can't
| be saved short of dumping its entire directory.
|
| Iff it must be the conda ecosystem, pixi
| (https://pixi.sh/latest/) is a much better pick.
|
| But conda-style packages (or anything with install time
| dependency resolution really) also have the issue of being
| completely untestable. That makes them unsuitable at least
| for user-facing package management, if you care about your
| packages not randomly breaking without warning.
|
| I'd rather see every language converge on a single package
| manager that implements functional package management, i.e.
| guix or nix. One can dream...
| skeledrew wrote:
| I've been using a conda+poetry flow for years and it's
| worked very well. Particularly because envs aren't tied
| to projects. I tried pixi for a few days in a project
| months ago and it was just breakage and untenable
| limitations all around. I prefer the freedom of using per
| project envs when I want, and shared envs other times.
| matrss wrote:
| What are you using each of them for in that workflow?
|
| I've found that there really only are two kinds of
| packages I want to install: those I want globally (e.g.
| CLI tools like git, git-annex, DataLad, whatever shell
| enhancing you want, etc.) and project-specific
| dependencies (anything required to get going with it on a
| freshly installed system with the smallest amount of
| assumptions possible).
|
| The former is sufficiently addressed by `pixi global` and
| other package managers, the latter by pixi projects.
| Notably, conda environments are a bad fit for both (not
| global, not really updatable, not automatically tied to a
| project, not rebuildable due to missing lock files, ...).
| thejevans wrote:
| Conda is nice for those of us stuck in government or
| similar roles with really locked down systems. Without
| conda, I'd be stuck with a global Python 3.9 and without
| the ability to install the non-python dependencies like
| ssl required for various libraries or to run jupyter
| locally.
| kstrauser wrote:
| Do they provide or permit Conda, but not other tooling?
| notatallshaw wrote:
| > Please consider consolidating python dependency management
| instead of fragmenting it: https://github.com/mamba-org/mamba
|
| Mamba doesn't even interact with the official python packing
| ecosystem... It is purely a conda replacement and conda
| packaging is not Python packaging (it's a more fundamental
| fragmentation than choosing a particular tool). So weird
| choice to not fragment Python dependency management.
|
| If you depend on both conda and standard Python packaging
| (e.g. PyPI.org) you can use Pixi: https://github.com/prefix-
| dev/pixi?tab=readme-ov-file#pixi-p.... Pixi attempts to
| bridge the conda world and the Python package world, so it's
| possible to rely more deeply on standards based packaging but
| still use conda tooling when you need it.
| jonathrg wrote:
| How is uv, another flavor-of-the-month Python packaging tool,
| not contributing to fragmentation?
| the_mitsuhiko wrote:
| uv unlike all other options has the best shot at becoming the
| one.
| whalesalad wrote:
| This is not fragmenting it. The requirements.txt file has been
| a steady and well used API for well over a decade. This tool
| just helps you produce one for a project that is missing one.
| ris wrote:
| How about you stop trying to create package managers for every
| single language/ecosystem in existence and instead converge on
| trying to solve the whole problem once and for all with Nix.
| verandaguy wrote:
| Asking honestly since I've seen Nix and NixOS show up on the
| front page here a bunch over the years, but never used it: my
| impression of it is that it fills the same kind of niche as
| the Zig or Nim languages: conceptually pretty cool, but not
| widely adopted outside of a dedicated core user group.
|
| Is this _really_ the case for Nix, or is it actually widely
| adopted, and this adoption is underreported?
|
| If it's not actually widely adopted, what do you think are
| the biggest obstacles in Nix's way?
| jghn wrote:
| > Please consider consolidating python dependency management
| instead of fragmenting it
|
| You realize that this project predates uv by roughly a decade,
| right?
| Terretta wrote:
| > _You realize that this project predates uv by roughly a
| decade, right?_
|
| "Whether it's new thinking, or decade old thinking, it's well
| time to converge."
| chaps wrote:
| Can you expand on what you mean, exactly?
| throwaway314155 wrote:
| They thought they had a better advertising chance for
| `uv` than they actually did. As far as I can tell, `uv`
| is a replacement for `pip`. But the project linked by OP
| doesn't actually replace `pip`, but rather a small subset
| of functionality in `pip` - `pip freeze`. Unless `uv` has
| some sort of import scanning functionality, the
| suggestion to use `uv` instead doesn't really make any
| sense.
| tmslnz wrote:
| How is it different to pip-chill[1]?
|
| [1]: https://github.com/rbanffy/pip-chill
| surye wrote:
| I believe pip-chill still operates on packaged installed into
| the environment. This project seems to derive from the code
| itself, even if no packages are installed in the current
| environment.
| halfcat wrote:
| I don't know if this is how pipreqs works, but I'd be concerned
| about a typo in an import that inadvertently installs a library.
|
| I've found pip-tools [1] to be a nice middle ground between `pip
| freeze` and something heavier like poetry. It's especially
| helpful in showing you where a library came from, when a library
| installs other libraries it depends upon.
|
| One could still have a typo in their requirements.in file with
| pip-tools, but changes there are much less frequent than imports
| in _any Python file in your proejcf_ which could be a daily
| occurrence in some codebases.
|
| [1] https://github.com/jazzband/pip-tools
| mooreds wrote:
| From the GH repo description:
|
| > Looking for maintainers to move this project forward.
|
| So not sure how maintained it is.
| v3ss0n wrote:
| Please consider using PDM instead. https://pdm-
| project.org/en/latest/
| joeyagreco wrote:
| could you explain why?
| 999900000999 wrote:
| Can you have it work with pipfiles too ?
| remram wrote:
| I had forgotten about pipfiles. Like their name implies (/s)
| they are not for pip but pipenv, a separate tool.
|
| https://github.com/pypa/pipfile "This format is still under
| active design and development", last commit "2 years ago". I
| think this is dead.
| 999900000999 wrote:
| https://github.com/pypa/pipenv
|
| Pipenv was last updated 10 hours ago. Looks like it's still
| an active project to me.
| kstrauser wrote:
| That's the repo for the Pipfiles themselves. The (only?)
| thing using them is pipenv, which lives in a separate repo.
| remram wrote:
| Pipfiles is what we are talking about.
| kstrauser wrote:
| True, but Pipfiles are basically whatever pipenv wants
| them to be. I'm not aware of anything but pipenv that
| uses them, although I'm sure there are more than zero
| such things. My point here being that I wouldn't take a
| lack of commits to the Pipfiles repo to mean that they're
| not still under development. More likely it's just being
| developed in the pipenv repo.
|
| But in any case, I'd vastly prefer the standard
| pyproject.toml over some file specific to mostly one
| tool.
| remram wrote:
| The pipfile repo claims the opposite:
|
| > This repository contains the design specification of
| the Pipfile format, as well as a proposed implementation
| of a parser for the specification which can be used by
| Pipenv and, in the future, any other consumer (e.g. pip)
|
| It should probably be updated to reflect whatever the
| current status and goals are.
| kstrauser wrote:
| Alternatively, can the pipenv gang support the same
| pyproject.toml files as everyone else?
| newman314 wrote:
| I am planning to move from Pipfiles to pyproject.toml in
| preparation to use uv and recently found this
| https://github.com/yhino/pipenv-poetry-migrate
|
| I have no idea how well it works.
| sean_pedersen wrote:
| I thought import names and PyPI names are not always equal, thus
| this can not work reliably, right?
| jonathrg wrote:
| It uses this file which maps import names to package names, but
| it's only 1152 packages and I'm not sure how it was generated.
| https://github.com/bndr/pipreqs/blob/master/pipreqs/mapping
| quietbritishjim wrote:
| In fairness that only seems to list those packages where the
| import name doesn't match the package name (e.g., it doesn't
| include numpy) so its overall coverage is a lot larger than
| that.
| qwerty1793 wrote:
| Even worse, there are some different PyPI packages with the
| same import name. For example, `import snappy` is probably
| referring to the compression library
| https://pypi.org/project/python-snappy/ but it could be this
| maths package https://pypi.org/project/snappy/
| danielktdoranie wrote:
| ...it would also be nice if Debian (Ubunutu and other
| derivitives) stop shitting on pip in some ridiculous paranoid
| attempt to stop breaking apt packages (that don't break apt
| packages anyway) or because of "muh security".
| L3viathan wrote:
| Huh? Messing with the system Python install absolutely can
| break your system.
|
| Just use venvs, or install separate Python versions for
| development e.g. via pyenv. You likely don't want to use the
| relatively outdated Python version shipped with Debian anyways.
| idoubtit wrote:
| I used this last year, with a relative success. I was asked to
| fix a Python code written by an intern that was no longer there.
| The code used external libraries, but did not declare any. I had
| only a zip of the source code, without any library. pipreqs was
| able to identify the 22 required libraries. Unfortunately, there
| was a runtime crash because a library was at the wrong version.
| So I had to let a real Python developer handle the next steps.
|
| BTW, this tool is not a dependency manager. Many sibling comments
| seem to misunderstand the case and promote unrelated tools.
| remram wrote:
| Can I have it find the requirements _as of a specific date_?
|
| If I find a script/notebook/project with no requirements.txt, I
| usually know when it was created. Being able to get the versions
| that were selected back then on the author's machine would be
| great for reproducibility.
| 0cf8612b2e1e wrote:
| I have never realized until this moment that I want such a
| tool. Ideally, you would never find yourself in this position,
| but alas.
| matrss wrote:
| For the future, please pick a package manager that can give you
| a lock file alongside your code, so that you have a definitive
| record of the dependencies.
|
| Even if you have all versions as of the time of the last
| modification to the code, you don't know if the dependency
| resolution happened at that point in time, or if the
| environment was set up years prior and never updated.
|
| Nevertheless, this is what you are looking for:
| https://pypi.org/project/pypi-timemachine/
| remram wrote:
| When I said "the author" I was not trying to mean myself.
|
| Agreed that in a perfect world I would never need this,
| but...
| d0mine wrote:
| See whether "exclude-newer" uv option might help if you've
| inherited a script without a dependencies lock file
| https://docs.astral.sh/uv/guides/scripts/#improving-reproduc...
| oprypin wrote:
| uv pip compile --exclude-newer=2019-12-02
|
| https://docs.astral.sh/uv/pip/compile/ *
| https://docs.astral.sh/uv/reference/cli/#uv-pip-compile
| remram wrote:
| This is great!
| TZubiri wrote:
| let's not fuck with supply chain vulnerabilities
| jonathrg wrote:
| Feel free to elaborate.
| skeledrew wrote:
| Anything that takes away from users fully migrating to pyproject
| really needs to die.
| mikepurvis wrote:
| But pyproject itself isn't even taking a stance on the
| underlying question of dependency management, which can still
| be flit, poetry, uv, pipx, pipenv, and who knows what else.
|
| (I'm a poetry user myself, but I can see the writing on the
| wall that poetry is probably not going to be the winner of who
| ultimately ends up in the PEP.)
| joeyagreco wrote:
| and why is that?
| skeledrew wrote:
| Because it's wasted resources for anything that isn't in line
| with said spec[0]. There should ideally be only one explicit
| way[1] to do a thing.
|
| [0] https://peps.python.org/pep-0621/ [1]
| https://peps.python.org/pep-0020/
| hiccuphippo wrote:
| This project looks like a first step to migrate something to
| pyproject.
| bndr wrote:
| Oh wow, my package on the front page again. Glad that it's still
| being used.
|
| This was written 10 years ago when I was struggling with pulling
| and installing projects that didn't have any requirements.txt. It
| was frustrating and time-consuming to get everything up and
| running, so I decided to fix it, apparently many other developers
| had the same issue.
|
| [Update]: Though I do think the package is already at a level
| where it does one thing and it does it good. I'm still looking
| for maintainers to improve it and move it forward.
| joshdavham wrote:
| > This was written 10 years ago when I was struggling with
| pulling and installing projects that didn't have any
| requirements.txt
|
| And 10 years later this is still a common problem!
| spullara wrote:
| Wrap your python program in an LLM that just keeps installing
| things when it fails until it works :)
| andrewmcwatters wrote:
| I don't spend a lot of time in Python, but my current
| understanding having read Python documentation and seeing some
| projects online is that you use pip and requirements.txt with
| --require-hashes, venv, and pyproject.toml to use a standard
| dependency management toolchain.
|
| Is that correct?
| thebigspacefuck wrote:
| Would this be useful for identifying packages in a
| requirements.txt that aren't used? Or is there another tool for
| that?
| manojlds wrote:
| mv and diff?
| timetraveller26 wrote:
| god, please no
| jcarrano wrote:
| When I first started with Python, long ago, I looked into these
| kind of solutions, which didn't work so well, and wondered why
| the concept was not better developed. Later, with experience, I
| realized it is not am great idea, and more hassle than the
| benefits it brings.
|
| I don't think it is good idea to merrily write 10s of import
| statements and end up with loads of dependencies.
| xuraiis wrote:
| Awesome project! Didn't realise this existed when I took a stab
| at implementing a similar thing a few months back:
| https://github.com/nmichlo/pydependence using DFS traversal of
| imports as well as naive support for lazy imports to generate
| optional extras as well as modify the pyproject.toml as part of
| pre-commit hooks.
___________________________________________________________________
(page generated 2024-09-30 23:02 UTC)