Post AXTV46UBEB7cb33I12 by mnot@techpolicy.social
(DIR) More posts by mnot@techpolicy.social
(DIR) Post #AXTQOsYPilczhN7T1s by simon@fedi.simonwillison.net
2023-07-08T04:42:41Z
0 likes, 0 repeats
I finally spent some time figuring out a minimal way of developing and packaging a Python library using a pyproject.toml fileIt turns out you can do it with just a pyproject.toml file - no setup.py, no setup.cfg - and the only tools you need are pip and build, with no need to use Poetry/Hatch/Flit/etc for the simplest casehttps://til.simonwillison.net/python/pyproject
(DIR) Post #AXTR7KPVVs4Qpq9Gm8 by simon@fedi.simonwillison.net
2023-07-08T04:50:51Z
0 likes, 0 repeats
This is my first attempt at packaging without setup.py so I'm nervous I may have missed some important details or got things wrong - eager to hear feedback on this from Python packaging experts
(DIR) Post #AXTRvhvkLAUUGCFc6i by pierrenick@hachyderm.io
2023-07-08T04:59:42Z
0 likes, 0 repeats
@simon I spent some time last year digging into all of that. One thing I still struggle with is omitting certain files from the package, but not the source (e.g.. tests) especially if they're inside the package. So I just include them.But it's way more clean and I don't feel I need Poetry or anything like that anymore. And the 'build' package is magical!
(DIR) Post #AXTS7vt8Gsn31arEGm by simon@fedi.simonwillison.net
2023-07-08T05:01:21Z
0 likes, 0 repeats
In particular I'm not at all confident I understand how pip and build automatically discover Python modules and packages - a note on my confusion about that here: https://til.simonwillison.net/python/pyproject#user-content-it-finds-the-python-files-for-you
(DIR) Post #AXTSLZktMEsQR1Vasa by geraldew@fosstodon.org
2023-07-08T05:01:50Z
0 likes, 0 repeats
@simon my open question is how dependent that method is on all usages - both building and importing - being for having quite current versions of Python running.Every time I read about changes to the packaging approaches, the "depends on Python X.YZ" seems to be left unsaid.Maybe there's no such dependency but with nothing said I don't feel I can trust to that.
(DIR) Post #AXTSYnDEdTBTw64IGO by coreyrayburnyung@mastodon.lawprofs.org
2023-07-08T05:05:01Z
0 likes, 0 repeats
@simon pyproject.toml has changed everything. It also makes it much easier to add other development packages for larger projects. FWIW, I'm in the midst of trying to simplify my workflow when I need more than just a quick pyproject.toml. To add docs, CI, and testing to the mix, I'm finding the best combo to be pdm, mkdocs, and GitHub Actions. Moving away from Sphinx and, to a lesser degree, Poetry, has been similarly revelatory.
(DIR) Post #AXTStL92BaoQxztvai by pandora@chaos.social
2023-07-08T05:10:30Z
0 likes, 0 repeats
@simon use a pyproject.toml. There u can define. Every little bit and also including and excluding paths
(DIR) Post #AXTTz1l3HKPeWzxXOq by simon@fedi.simonwillison.net
2023-07-08T05:22:55Z
0 likes, 0 repeats
@geraldew Yeah I'd like to understand how well this works across different Python versions - since pip and build are separate libraries that are tested against Python 3.8 upwards my assumption is that the patterns in my TIL should work against every modern Python, but I'd like to confirm that
(DIR) Post #AXTUAJG4YLYlzcqgIS by simon@fedi.simonwillison.net
2023-07-08T05:23:59Z
0 likes, 0 repeats
@coreyrayburnyung I've been sticking with Sphinx but using MyST so I can write my documentation in Markdown, and I really like that combination - example here: https://github.com/simonw/shot-scraper/tree/main/docs
(DIR) Post #AXTUMxNzBXjFvCa4Tg by jamwil@fosstodon.org
2023-07-08T05:25:05Z
0 likes, 0 repeats
@simon This is what I’ve been using as a starting point lately: https://github.com/jamwil/bulletproof-python/blob/main/pyproject.tomlNote the [tool.setuptools*] bits for discovering modules.Not an expert, this may be wrong as well but it works well so far for me.
(DIR) Post #AXTV46UBEB7cb33I12 by mnot@techpolicy.social
2023-07-08T05:34:18Z
0 likes, 0 repeats
@simon from what I can tell, that doesn't support setting the version by reading an attribute; setup.cfg supports that.
(DIR) Post #AXTVbP9gpsic3ULaGu by medecau@hachyderm.io
2023-07-08T05:40:08Z
0 likes, 0 repeats
@simon you could test that the installation is proper by running the tests through 'tox''tox' will create virtualenvs for each of the environment configurations and install your project along with the dependencies there – if something is broken with the install you'll know then
(DIR) Post #AXTWsVCPiYlhg3L0Ma by jonathon@masto.ai
2023-07-08T05:55:04Z
0 likes, 0 repeats
@simon I’d suggest being explicit about your build system in pyproject.toml; see https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html (add the [build-system] block and specify setuptools as shown there).As for automatic discovery, setuptools is decently smart about detecting and handling common source layouts, or you can manually configure it similarly to how you would in setup.py (docs on this are in the linked page too).
(DIR) Post #AXTYTyr4WMcAK0n3Qm by simon@fedi.simonwillison.net
2023-07-08T06:13:18Z
0 likes, 0 repeats
@jonathon I was excited to discover that it works without [build-system] - and now I'm resistant to adding that without fully understanding what it changes / makes possible in comparison to omitting that block entirely
(DIR) Post #AXTYfz7RMhmOcEAdEm by leif@tech.lgbt
2023-07-08T06:15:12Z
0 likes, 0 repeats
@simon this is the way, but I recommend the src/ layout
(DIR) Post #AXTYrwNhN04dbETZxI by jonathon@masto.ai
2023-07-08T06:15:20Z
0 likes, 0 repeats
@simon If you leave it out, you’re at the mercy of pip or build to decide which build system gets used for you— and since that defines *all* of your build’s behavior, you probably want to be explicit here if you want reproducibility.
(DIR) Post #AXTeUgXYhRNJXBNlD6 by paulzuradzki@mastodon.social
2023-07-08T07:20:06Z
0 likes, 0 repeats
@simon section 3.3 “Controlling Source Code and File Discovery” in Publishing Python Packages via Dane Hillard may be a useful reference on this point. https://www.manning.com/books/publishing-python-packages
(DIR) Post #AXToo5zxGqGZXCYhnc by pfmoore@mastodon.social
2023-07-08T09:16:00Z
0 likes, 0 repeats
@simon @jonathon if you leave it out, you'll get setuptools by default. That's required by the PEP so you're not at risk of tools having different defaults. But explicit is better than implicit...
(DIR) Post #AXTuxSgKu11CDYEPT6 by nedbat@hachyderm.io
2023-07-08T10:24:49Z
0 likes, 0 repeats
@simon I did a similar exercise: https://github.com/nedbat/pkgsampleAnd how British of you: "TOML suppor twas only added ..." :)
(DIR) Post #AXTw2oXXA7HdPYELWS by tintvrtkovic@mastodon.social
2023-07-08T10:37:05Z
0 likes, 0 repeats
@simon interesting, I've been using non-setuptools build systems for a long time but never knew you could omit the build section (the comments here explain that it defaults to setuptools).I personally don't like using optional dependencies for test, lint, do etc dependencies since it gets exported to pypi as metadata. It's still better than a separate file but pdm, the packaging tool I've switched to using, has a separate section for these so they don't get published
(DIR) Post #AXTzfikmuPAMcbfp5s by vicchi@mapstodon.space
2023-07-08T11:17:34Z
0 likes, 0 repeats
@simon With the caveat that my Python-fu falls into the "know enough to get by and do what I need to do but nowhere near an expert" ... pyproject.toml is a huge step in the right direction *if* you're building and packaging a library but *not* if you're building and installing an application. You'll still need a requirements.txt to install the application dependencies. Which you could generate from pip-tools but that feels like a needless extra step.
(DIR) Post #AXU4lgxD0p8YY49MEC by whynothugo@fosstodon.org
2023-07-08T12:15:00Z
0 likes, 0 repeats
@simon "python -m installer" can also be used to install wheels.It is especially useful when bootstrapping systems or when packaging wheels for other distribution channels (since you can specify a DESTDIR and it keeps dependencies out of scope).
(DIR) Post #AXU66au4nqjHooDQxM by whynothugo@fosstodon.org
2023-07-08T12:30:01Z
0 likes, 0 repeats
@simon Odd, putting 'homepage' in [project] fails for me when I run `pip install -e .`: configuration error: `project` must not contain {'homepage'} propertiesPutting it in [project.urls] works.
(DIR) Post #AXUA2JSw5siAOWjd5M by aryounce@mastodon.social
2023-07-08T13:13:56Z
0 likes, 0 repeats
@simon How does one ‘pip install -e’ in this situation? Will it use the .toml?
(DIR) Post #AXUADWfjKiu0XT4A1A by aryounce@mastodon.social
2023-07-08T13:14:58Z
0 likes, 0 repeats
@simon You addressed that straight away in the post. My apologies.
(DIR) Post #AXUAPjwZobArAvlnsG by nafiul@fosstodon.org
2023-07-08T13:15:01Z
0 likes, 0 repeats
@simon would be interesting to create a video on this. I think I might attempt it next week.
(DIR) Post #AXUNxStX0af5Y3l0me by simon@fedi.simonwillison.net
2023-07-08T15:50:03Z
0 likes, 0 repeats
@whynothugo Thanks, yeah same for me - I'll update my post
(DIR) Post #AXUlZ94vQmeK2V1UWW by simon@fedi.simonwillison.net
2023-07-08T20:14:16Z
0 likes, 0 repeats
@ubernostrum @mnot Unrelated, but I saw you're using sphinx-inline-tabs in webcolors which inspired me to start using that for Datasette just now: https://github.com/simonw/datasette/issues/1153#issuecomment-1627395947
(DIR) Post #AXVCmLQPXGtGrHahxw by simon@fedi.simonwillison.net
2023-07-09T01:19:06Z
0 likes, 0 repeats
OK, I'm getting pretty comfortable with this pattern now - I just shipped pocket-to-sqlite 0.2.3 to PyPI, my first package to work using just a pyproject.toml file with no setup.py https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3
(DIR) Post #AXVCzBGKdBnV6Wztlg by simon@fedi.simonwillison.net
2023-07-09T01:21:09Z
0 likes, 0 repeats
@ubernostrum I did hit a bit of a gnarly problem where "sphinx-build -b xml" doesn't work any more, failing with a "visiting unknown node type: TabContainer" error - I use the XML output to build a searchable index of my documentationFound a workaround fix for that though, thankfully: https://github.com/simonw/datasette/issues/1153#issuecomment-1627450852
(DIR) Post #AXVFNrPblsOMggET0S by pganssle@qoto.org
2023-07-09T01:48:37Z
1 likes, 0 repeats
@simon You really should specify the build system there. Not specifying it isn’t saying, “pip, handle this for me” it’s actually conveying, “I created this file in the period of time after tools other than pip started using it but before pip did.”
(DIR) Post #AXVPVqDR6O9Qpqz14C by ianhopkinson@mastodon.social
2023-07-09T03:42:07Z
0 likes, 0 repeats
@simon I've been playing with this too. I note dropping setup.py means using the relatively new, 3rd party(?) build module which would make me nervous I would prefer to keep a minimal setup.py. My blog post on this is massively popular, clearly a lot of people are trying to understand this: https://ianhopkinson.org.uk/2022/02/understanding-setup-py-setup-cfg-and-pyproject-toml-in-python/
(DIR) Post #AXVPhLbWjy4dN2jp6e by AlSweigart@mastodon.social
2023-07-09T03:43:04Z
0 likes, 0 repeats
@simon THANK YOU. I'm adding all of these links to my to-read list. I am in desperate need of this info. My one attempt to switch to pyproject.toml resulted in weird little errors and I had to switch back to setup.py.
(DIR) Post #AXVTh3lTT1U9rJVcps by webology@mastodon.social
2023-07-09T04:29:00Z
0 likes, 0 repeats
@simon I wonder if someone has built a setup.(cfg|py) to pyproject.toml project that maps everything out accurately.
(DIR) Post #AXVoi0Vh9Phz5mQCES by stuartl@mastodon.longlandclan.id.au
2023-07-09T08:22:05Z
0 likes, 0 repeats
@simon Hey, that's further ahead than I have gotten.I've got several packages I maintain that still use `setup.py` because it's "what I know"… don't know this `pyproject.toml` stuff at all.But… through this project, I'm learning, and hopefully we can build something that at least handles the "basic" cases.If there's some hand-fiddling at the end for the more fancy cases, then so be it…but if we can handle the basics, that may help a lot of people.Your experiences are definitely valuable.
(DIR) Post #AXVotWPgsN9TDv6U08 by diazona@techhub.social
2023-07-09T08:25:07Z
0 likes, 0 repeats
@simon I would hardly call myself an "expert", especially not in a context where you might be getting responses from developers of setuptools and pip and such things, but as someone with a decent amount of experience #packaging #Python projects, the one thing that stuck out to me was the same thing others said, that leaving out the [build-system] section seems like a bad idea. I mean, it's great to do the experiment and show that you *can* omit it, but IMO it falls firmly in the camp of opposite-of-best-practice for "production-quality" projects: even if you trust that all the relevant tools will assume setuptools as a default (which may be the case now but will it always be?), it's still a fairly obscure piece of "magic" for someone reading the build configuration to understand.
(DIR) Post #AXVxBre9T9bZvI7oLQ by mgiacomo@fediscience.org
2023-07-09T09:59:11Z
0 likes, 0 repeats
@simon Thanks for your always inspiring TILs. Quick question: how come do you prefer Pocket over something like a self-hosted wallabag?
(DIR) Post #AXX4LdK4TQCtEPIWAK by simon@fedi.simonwillison.net
2023-07-09T22:54:34Z
0 likes, 0 repeats
@mgiacomo it's one less thing to host!I'm open to alternatives though, I really want the full text article bodies
(DIR) Post #AXXzs1F6lnUk4VYl1c by theaxer@floss.social
2023-07-10T09:38:56Z
0 likes, 0 repeats
@simon It is pretty obvious that a flow like this should exist, but I honestly had no idea and it makes me so happy to see someone posting a pure stdlib #Python workflow like this. This should probably be included in the Python docs. I just wish some very minimal DX improvements to make this easier could be baked into the stdlib.
(DIR) Post #AXZ2ICdHRIafzaRH72 by erdnaxeli@framapiaf.org
2023-07-10T21:40:36Z
0 likes, 0 repeats
@simon Slowly but surely the python packaging system evolves!Still far from perfect but cool to see that.
(DIR) Post #AXa6WgJEUcbj3tBITw by mgiacomo@fediscience.org
2023-07-11T10:02:57Z
0 likes, 0 repeats
@simon I had similar problems trying to automate my pipeline bookmark -> kindle. At the moment I query firefox bookmarks sqlite -> send url to puppeteer -> print-to-pdf (I have a large e-screen) -> send pdf to kindle. Probably with Pocket it's one API call :)