[HN Gopher] PyPI now supports digital attestations
___________________________________________________________________
PyPI now supports digital attestations
Author : miketheman
Score : 112 points
Date : 2024-11-14 14:25 UTC (8 hours ago)
(HTM) web link (blog.pypi.org)
(TXT) w3m dump (blog.pypi.org)
| woodruffw wrote:
| I'm very excited this has come to fruition!
|
| PyPI's user docs[1] have some more details, as does the
| underlying PEP[2].
|
| [1]: https://docs.pypi.org/attestations/
|
| [2]: https://peps.python.org/pep-0740/
| ashvardanian wrote:
| Here is the GitHub issue you can subscribe to for automatically
| generated attestations in the GitHub CI PyPi upload action:
| https://github.com/pypa/gh-action-pypi-publish/issues/288
| woodruffw wrote:
| Just a note: that issue is for connecting the already-present
| attestations to GitHub's attestations feature[1]. Attestations
| are already automatically generated and uploaded _to PyPI_
| regardless of that issue.
|
| (That issue will still be great to resolve, but just to avoid
| any confusion about its scope.)
|
| [1]: https://docs.github.com/en/actions/security-for-github-
| actio...
| dlor wrote:
| This is awesome to see, and the result of many years of hard work
| from awesome people.
| antononcube wrote:
| It looks like another process certain software engineers want to
| program and facilitate. I hope it is and stays optional.
| LtWorf wrote:
| I'm pessimistic about that.
| trishankkarthik wrote:
| Very cool, and congrats!
|
| The corresponding ToB blog post says the following:
|
| > Longer term, we can do even better: doing "one off"
| verifications means that the client has no recollection of which
| identities should be trusted for which distributions. To address
| this, installation tools need a notion of "trust on first use"
| for signing identities, meaning that subsequent installations can
| be halted and inspected by a user if the attesting identity
| changes (or the package becomes unattested between versions).
|
| Agree: signing is only as good as verification. However, trust-
| on-first-use (TOFU) is not the most secure way to map packages to
| attestations because nothing stops attackers who have taken over
| PyPI from tampering with the _unsigned_ mapping of identities to
| attestations in package lockfiles for new clients (certainly in
| containerized environments where everything could look new), and
| even just new versions of packages.
|
| Although [PEP 458](https://peps.python.org/pep-0458/) is about
| signing the Python package index, it sets the foundation for
| being able to securely map packages to signed in-toto _policies_,
| which would in turn securely map identities to attestations. I
| think it is worth noting how these different PEPs can work
| together :)
| cpburns2009 wrote:
| Great, now how do you use attestations with Twine when publishing
| packages on PyPI outside of the Github ecosystem?
| hifromwork wrote:
| You need to rely on one of the four trusted publishers. You
| can't do it yourself: https://docs.pypi.org/trusted-
| publishers/adding-a-publisher/
| guappa wrote:
| You don't. The whole point is that you can no longer sign
| anything. Microsoft signs for you.
|
| And of course the signature means "this user can push to
| github" and nothing more.
| krnavy wrote:
| After 2FA, the previous PyPI buzzword that was forced on
| everyone, JFrog discovered a key leak that compromised
| everything:
|
| https://news.ycombinator.com/item?id=40941809
|
| JFrog also discovered multiple malicious package exploits later.
|
| Now we get a Github centric new buzzword that could be replaced
| by trusted SHA256 sums. Python is also big on business speak like
| SBOM. The above key leak of course occurred after all these new
| security "experts" manifested themselves out of nowhere.
|
| The procedure remains the same. Download a package from the
| original creators, audit it, use a local repo and block PyPI.
| SethMLarson wrote:
| Hello! I believe I'm one of the "manifested" security experts
| you're hinting at :)
|
| Good security doesn't demand perfection, that's why security is
| both prevention and preparedness. The response from our admin
| was in every way beyond what you'd expect from many other orgs:
| prompt response (on the order of minutes), full audit of
| activity for the credential (none found), and full public
| disclosure ahead of the security researcher's report.
|
| > JFrog also discovered multiple malicious package exploits
| later.
|
| If you're referencing malicious packages on PyPI then yes! We
| want to keep PyPI freely open to all to use, and that has
| negative knock-on effects. Turns out that exploiting public
| good code repositories is quite popular, but I maintain that
| the impact is quite low and that our ability to respond to
| these sorts of attacks is also very good due to our network of
| security engineers and volunteers who are triaging their
| reports. Thanks to the work of Mike Fiedler (quarantining
| packages, API for reporting malicious packages, better UI for
| triagers) our ability to respond to malicious packages will
| become even better.
|
| > Now we get a Github centric new buzzword that could be
| replaced by trusted SHA256 sums.
|
| In a way, this feature is what you're describing but is easier
| to automate (therefore: good for you as a user) and is more
| likely to be correct because every attestation is verified by
| PyPI before it's made available to others (which is also good
| for users). The focus on GitHub Actions is because this is
| where many Python projects publish from, there is little
| incentive to create a feature that no one will use.
|
| > Python is also big on business speak like SBOM.
|
| Indeed, there is legislation in many places that will require
| SBOMs for all software placed in their markets so there is
| plenty of interest in these standards. I'm working on this
| myself to try to do the most we can for users while minimizing
| the impact this will have on upstream open source project
| maintainers.
| untitaker_ wrote:
| I'm just here to say that this is a shit reply unworthy of even
| HN. if you want to vent like that, do it in private. anyway,
| green comment moment
| belval wrote:
| I have a bit of uneasiness about how this is heavily pushing
| GitHub actions as the correct way to publish to PyPI. I had to
| check PEP740 to make sure it was not directly supported by
| Microsoft.
|
| > The generation and publication of attestations happens by
| default, and no changes are necessary for projects that meet all
| of these conditions: publish from GitHub Actions; via Trusted
| Publishing; and use the pypa/gh-action-pypi-publish action to
| publish.
|
| If you then click on "The manual way" it adds a big disclaimer:
|
| > STOP! You probably don't need this section; it exists only to
| provide some internal details about how attestation generation
| and uploading work. If you're an ordinary user, it is strongly
| recommended that you use one of the official workflows described
| above.
|
| Where the only official workflow is "Use GitHub Actions".
|
| I guess I am an idealist but as a maintainer this falls short of
| my expectations for the openness of Python and PyPI.
| woodruffw wrote:
| > Where the only official workflow is "Use GitHub Actions".
|
| The standard behind this (PEP 740) supports anything that can
| be used with Trusted Publishing[1]. That includes GitLab,
| Google Cloud, ActiveState, and _can_ include any other OIDC IdP
| if people make a good case for including it.
|
| It's not tied to Microsoft or GitHub in any particular way. The
| only reason it emphasizes GitHub Actions is because that's
| where the overwhelming majority of automatic publishing traffic
| comes from, and because it follows a similar enablement pattern
| as Trusted Publishing did (where we did GitHub first, followed
| by GitLab and other providers).
|
| [1]: https://docs.pypi.org/trusted-publishers/
| silverwind wrote:
| Why does this need to allowlist CI providers in first place?
| Why not publish an open interface any CI provider can
| integrate against?
| SethMLarson wrote:
| Because every CI/ID provider has a different set of claims
| and behaviors that would constitute a "secure" policy for
| verification. If there was one singular way to do that then
| we could, but there isn't yet so PyPI needs to onboard
| providers piecemeal. The work to add a new provider is not
| massive, the reason there are not tons of providers isn't
| because the work is hard but rather because people are
| voting with their feet so Github and Gitlab make sense as
| initial providers to support.
| woodruffw wrote:
| Because the security benefit of Trusted Publishing via OIDC
| versus normal API tokens is marginal at small scales, in
| two senses:
|
| 1. The primary benefit of Trusted Publishing over a manual
| API token is knowing that the underlying OIDC IdP has an
| on-call staff, proper key management and rotation policies,
| etc. These can be guaranteed for GitHub, GitLab, etc., but
| they're harder to prove for one-off self-hosted CI setups.
| For the latter case, the user is no better off than they
| would be with a manual API token, which is still (and will
| always be) supported.
|
| 2. If the overwhelming majority of traffic comes from a
| single CI/CD provider, adding more code to support generic
| OIDC IdPs increases PyPI's attack surface for only marginal
| user benefit.
|
| There also is no "open interface" for PyPI to really use
| here: this is all built on OIDC, but each OIDC provider
| needs to have its unique claims mapped to something
| intelligible by PyPI. That step requires thoughtful,
| manual, per-IdP consideration to avoid security issues.
| antononcube wrote:
| > [...] the user is no better off than they would be with
| a manual API token, which is still (and will always be)
| supported.
|
| This is good to know. I did not see related statements in
| of the documents linked to this discussion, though.
| antononcube wrote:
| I am not sure why my comment above is downvoted -- if you
| know where the perpetual optionality of digital
| attestations is officially stated, please, provide a
| link.
| freeone3000 wrote:
| I think I would be better off with API key + PGP than API
| key alone. And that's being phased out?
| woodruffw wrote:
| You can no longer upload a PGP signature to PyPI, if
| that's what you mean. That was phased out last year (to
| virtually no complaint since nobody was actually
| verifying any of the signatures, much less attempting to
| confirm that their keys were discoverable[1]).
|
| [1]: https://blog.yossarian.net/2023/05/21/PGP-
| signatures-on-PyPI...
| guappa wrote:
| > to virtually no complaint since nobody was actually
| verifying any of the signatures
|
| And this is in no way a consequence of pypi stopping to
| host public keys right? Say the whole story at least...
| Say that there used to be a way to verify the signatures
| but you dropped it years ago and since then the
| signatures have been useless.
| woodruffw wrote:
| If it did, it was well before I ever began to work on
| PyPI. By the time I came around, PGP signature support
| was vestigial twice over _and_ the public key discovery
| network on the Internet was rapidly imploding.
|
| (But also: having PyPI be the keyserver defeats the
| point, since PyPI could then trivially replace my
| package's key. If that's the "whole story," it's not a
| very good one.)
| belval wrote:
| I get that, that's why I didn't go "This is Embrace Extend
| Extinguish", but as constructive feedback I would recommend
| softening the language and to replace:
|
| > STOP! You probably don't need this section;
|
| In https://docs.pypi.org/attestations/producing-
| attestations/#t...
|
| Perhaps also add a few of the providers you listed as well?
|
| > The only reason it emphasizes GitHub Actions is because
| that's where the overwhelming majority of automatic
| publishing traffic comes from
|
| GitHub being popular is a self-reinforcing process, if GitHub
| is your first class citizen for something as crucial as
| trusted publishing then projects on GitHub will see a higher
| adoption and become the de-facto "secure choice".
| woodruffw wrote:
| > but as constructive feedback I would recommend softening
| the language and to replace:
|
| I can soften it, but I think you're reading it excessively
| negatively: that warning is there to make sure people don't
| try to do the fiddly, error-prone cryptographic bits _if_
| they don 't need to. It's a numerical fact that most
| project owners _don 't_ need that section, since most are
| either using manual API tokens or are publishing via GitHub
| Actions.
|
| > Perhaps also add a few of the providers you listed as
| well?
|
| They'll be added when they're enabled. Like I said in the
| original comment, we're using a similar enablement pattern
| as happened with Trusted Publishing: GitHub was enabled
| first because it represents the majority of publishing
| traffic, followed by GitLab and the others.
|
| > GitHub being popular is a self-reinforcing process, if
| GitHub is your first class citizen for something as crucial
| as trusted publishing then projects on GitHub will see a
| higher adoption and become the de-facto "secure choice".
|
| I agree, but I don't think this is PyPI's problem to solve.
| From a security perspective, PyPI should prioritize the
| platforms where the traffic is.
|
| (I'll note that GitLab has been supported by Trusted
| Publishing for a while now, and they _could_ make the
| publishing workflow more of a first class citizen, the way
| it is on GHA.)
| belval wrote:
| > I agree, but I don't think this is PyPI's problem to
| solve. From a security perspective, PyPI should
| prioritize the platforms where the traffic is.
|
| To me that's a bit of a weird statement, PyPI is part of
| the Python foundation, making sure that the project
| remains true to its open-source nature is reasonable?
|
| My concern is that these type of things ultimately play
| out as "we are doing the right thing to limit supply
| chain attacks" which is good an defendable, but in ~5
| years PyPI will have an announcement that they are
| sunsetting PyPI package upload in favor of the trusted
| provider system. pip (or other tooling) will add warnings
| whenever I install a package that is not "trusted". Maybe
| I am simply pessimistic.
|
| That being said we can agree to disagree, I am not part
| of the PSF and I did preface my first comment with "I
| guess I am an idealist".
| woodruffw wrote:
| > making sure that the project remains true to its open-
| source nature is reasonable?
|
| What about this, in your estimation, undermines the open-
| source nature of PyPI? Nothing about this is proprietary,
| and I can't think of any sane definition of OSS in which
| PyPI choosing to verify OIDC tokens from GitHub (among
| other IdPs!) meaningfully subverts PyPI's OSS
| committment.
|
| > PyPI package upload in favor of the trusted provider
| system. pip (or other tooling) will add warnings whenever
| I install a package that is not "trusted". Maybe I am
| simply pessimistic.
|
| Let me put it this way: if PyPI disables API tokens in
| favor of _mandatory_ Trusted Publishing, I will eat my
| shoe on a livestream.
|
| (I was the one of the engineers for both API tokens _and_
| Trusted Publishing on PyPI. They 're complementary, and
| neither can replace the other.)
| mananaysiempre wrote:
| > What about this, in your estimation, undermines the
| open-source nature of PyPI?
|
| Absence of support for self-hosting, in the spirit of
| freedom 0 = OSD 5&6? Or, for that matter, for any
| provider whose code is fully open source?
| woodruffw wrote:
| > Absence of support for self-hosting, or for that matter
| for any non-proprietary service?
|
| This has nothing to do with self-hosting, whatsoever. You
| can upload to PyPI with an API token; that will always
| work and will not do anything related to Trusted
| Publishing, which exists entirely because it makes sense
| for _large_ services.
|
| PyPI isn't required to federate with the server in my
| basement through OpenID Connect to be considered open
| source.
| BiteCode_dev wrote:
| I'm with @belval on this one, it's ok to prioritize
| github, but people that want the standard to implement an
| alternative should not feel like they are doing something
| that may not be supported.
|
| It kinda feels like that right now.
| woodruffw wrote:
| Again, to be clear: the standard does not stipulate
| GitHub or any other specific identity providers. The plan
| is to enable GitLab and the other Trusted Publisher
| providers in short order.
|
| This is _exactly_ the same as Trusted Publishing, where
| people accused the feature of being a MSFT trojan horse
| because GitHub was enabled first. I think it would
| behoove everybody to assume the best intentions here and
| remember that the goal is to secure the most people _by
| default_.
| int_19h wrote:
| I think the point is that this needs to be made clearer
| in the official docs from the get go.
| woodruffw wrote:
| It's said explicitly in the second sentence in the usage
| docs[1].
|
| > Attestations are currently only supported when
| uploading with Trusted Publishing, and currently only
| with GitHub-based Trusted Publishers. Support for other
| Trusted Publishers is planned. See #17001 for additional
| information.
|
| [1]: https://docs.pypi.org/attestations/producing-
| attestations/
| aseipp wrote:
| Anyone can run an OIDC system if they want. But PyPI is
| not under an obligation to trust an OIDC provider running
| on a random rpi3 in your basement. More than that, GitHub
| is "trusted" because we can be pretty sure they have an
| on-call staff to handle incidents, that they can reliably
| say "This token was provided on behalf of this user at
| this time for this build", etc.
|
| Even if you standardized the more technical parts like
| OIDC claim metadata (which is 100% provider specific), it
| wouldn't really change the thrust of any of this -- PyPI
| is literally _trusting_ the publisher in a _social_
| sense, not in some "They comply with RFC standards and
| therefore I can plug in my random favorite thing" sense.
|
| This whole discussion is basically a non-issue, IMO. If
| you want to publish stuff from your super-duper-secret
| underground airgapped base buried a mile underneath the
| Himalayas, you can use an API token like you have been
| able to. It will be far less hassle than running your own
| OIDC solution for this stuff.
| guappa wrote:
| > error-prone cryptographic bits if they don't need to
|
| They can't. Because you wouldn't accept their key anyway.
| woodruffw wrote:
| I think you're missing something. The key in question is
| a short-lived ECDSA key that lives inside a publishing
| workflow and is destroyed after signing; neither GitHub
| nor the Sigstore CA generates a signing key for you.
|
| PyPI will accept any key bound to an identity, provided
| we know how to verify that identity. Right now that means
| we accept Trusted Publishing identities, and GitHub
| identities in particular, since that's where the
| overwhelming majority of Python package publishing
| traffic comes from. Like what happened Trusted
| Publishing, this will be expanded to other identities
| (like GitLab repositories) as we roll it out.
| guappa wrote:
| How does pypi know I'm not github? Because I can sign
| with my keys and not with github's key.
|
| Never mind all the low level details of the temporary
| keys and hashes and all of that. This is an high level
| comment not a university book about security.
| benatkin wrote:
| That's good. IIRC the feature was launched with GitHub as the
| only option, which is the same state of affairs of
| https://jsr.io/ right now.
|
| IMO the launch should have been delayed until there was more
| than one _trusted publisher_.
| freeone3000 wrote:
| It requires an OIDC IdP, though... with PGP, I can verify
| that my identity is constant, but now, I'm reliant on some
| chosen third-party. And it has to keep being the same one, to
| boot! I don't like the lock-in this causes, and I definitely
| don't like the possibility of my right to publish being
| revoked by a third party.
| beng-nl wrote:
| I'm only now learning about what OIDC IdP is (for those
| like me: openid connect identity provider). But from my
| reading, a self hosted gitlab can function as an oidc idp.
|
| That would be enough control, right?
| guappa wrote:
| Except that also for trusted publishing, they only allowed
| github in the beginning and eventually added a couple of
| other providers. But if you're not google or microsoft you
| won't be added.
| didntcheck wrote:
| I might be reading this wrong but doesn't it include four
| workflows, including Gitlab? The "manual way" section is
| grouped under the Github tab, which indicates to me that it's
| only explaining how the Github-specific workflow works?
|
| https://docs.pypi.org/trusted-publishers/creating-a-project-...
| belval wrote:
| I was referring to the "Get Started" section that links to a
| producing attestation page that can be found here:
| https://docs.pypi.org/attestations/producing-
| attestations/#t....
|
| > Support for automatic attestation generation and
| publication from other Trusted Publisher environments is
| planned. While not recommended, maintainers can also manually
| generate and publish attestations.
| woodruffw wrote:
| You're on a slightly different page: Trusted Publishing
| supports four different providers, but this initial
| enablement of attestations (which are built _on top_ of
| Trusted Publishing) is for GitHub only. The plan is to enable
| the others in short order.
| hifromwork wrote:
| Even more, the previous way was to use GPG signatures, which
| were recently deprecated and removed. So you don't really have
| a choice.
|
| >Where the only official workflow is "Use GitHub Actions".
|
| Well you can do it manually with other solutions... as long as
| they are one of the four trusted publishers (see "Producing
| attestations manually does not bypass (...) restrictions on
| (...) Trusted Publishers":
|
| https://docs.pypi.org/trusted-publishers/adding-a-publisher/...
|
| This means that you literally can't do it manually, you have to
| rely on one of:
|
| * Github
|
| * Google Cloud
|
| * ActiveState (I'm not familiar with it)
|
| * Github.com (not just github, only that one instance)
|
| Really surprising development, IMO.
| ryanisnan wrote:
| I think you're being overly critical. When it says that it adds
| support for Trusted Publishers, it links directly to this page:
| https://docs.pypi.org/trusted-publishers/.
|
| This page clearly explains how this uses OIDC, and uses GitHub
| Actions as an example. At no point in my read did I feel like
| this was shilling me a microsoft product.
| guappa wrote:
| Try to run your own gitlab instance and use this to publish.
| See what happens...
|
| At this moment this only supports github.
| doctorpangloss wrote:
| On the one hand, you are totally right, GitHub Actions are the
| VS Code of automation. People choose them because they are
| broke, they work well enough, and the average person chooses
| things based on how it looks. GitHub Actions looks easy, it's
| cozy, it's comfy, VS Code looks like a code editor, everything
| has to be cozy and comfy.
|
| On the other hand, considering all of that, you can see why
| Python would arrive at this design. They are the only other
| people besides NPM who regularly have supply chain attack
| problems. They are seemingly completely unopinionated about how
| to fix their supply chain problems, while being opinionated
| about the lack of opinions about packaging in general. What is
| the end goal? Presumably one of the giant media companies,
| Meta, Google or Microsoft maybe, have to take over the
| development of runtime and PEP process, but does that even
| sound good?
| LtWorf wrote:
| I think the end goal is to only allow github/google whatever
| accounts to publish code on Pypi, so importing whatever will
| be USA-sanctions safe.
|
| The burden to ban russians/koreans/iranians will be on those
| big companies and pypi will be able to claim they respect the
| rules without having the resources themselves to ban accounts
| from sanctioned countries.
|
| So in my opinion, in the future tooling will have some option
| to check if the software was uploaded via microsoft et al and
| can be considered unsanctioned in USA.
|
| Or they might just say that's how you upload now and that's
| it.
| doctorpangloss wrote:
| > so importing whatever will be USA-sanctions safe...
|
| You are talking about guys who can barely figure out how to
| structure a Python package repository. They are not doing
| this kind of 4D chess.
|
| And anyway, none of that makes any sense.
| LtWorf wrote:
| The people paying for this, not them.
| woodruffw wrote:
| None of this is true. There is no plan to disable API token
| uploads to PyPI or to mandate Trusted Publishing, much less
| attestations. It's entirely intended to be a misuse-
| resistant way to upload to the index in the "happy case"
| where the package uses a CI/CD workflow that supports
| OpenID Connect.
|
| (I'm disappointed by the degree to which people seem to
| gleefully conspiracize about Python packaging, and
| consequently uncritically accept the worse possible
| interpretation of anything done to try and improve the
| ecosystem.)
| LtWorf wrote:
| I don't think you can predict the future any more than
| me.
| doctorpangloss wrote:
| I understand the desire to raise awareness about your
| fringe concern. But it makes no sense.
| woodruffw wrote:
| I don't baselessly speculate about it. But yes, I do
| think I can predict with some confidence that PyPI is not
| going to remove API token access.
| LtWorf wrote:
| Funny how I have a talk about the evolution of pypi's security
| at the upcoming minidebconf this weekend.
|
| I guess I'll have to update my slides :D
|
| But well as a debian developer my advice is to just use debian
| and completely ignore pypi, so I might be slightly biased.
| SethMLarson wrote:
| Congratulations to everyone involved in this work! This is an
| incredible building block for not just supply-chain security both
| inside and downstream of the PyPI ecosystem but also for data
| analysis of open source projects (through strong linkage back to
| source repositories). Thank you all <3
| jonnycomputer wrote:
| Supply chain security is very important, and this seems like an
| important step. Seems absolutely essential that something like
| the Mozilla foundation, or EFF, or some other open-source
| friendly entity help provide such a service, instead of
| corralling users into companies with exploitative business
| models.
|
| I am in no hurry to be pushed into using Github, Gitlab or
| whatever else. Programmer's open source code has been monetized
| by these companies to feed AI LLM beasties, and it's
| fundamentally objectionable to me. I self-host my code using
| Gitea for that reason.
| Comma2976 wrote:
| Great, now all that is missing is a decent packaging system for
| Python.
| ris wrote:
| I'm not really convinced of the value of such attestations until
| a second party can reproduce the build themselves on their own
| hardware.
|
| Putting aside the fact that the mechanisms underpinning Github
| Actions are a mystery black box, the vast vast vast majority of
| github workflows are not built in a reproducible way - it's not
| even something that's encouraged by Github Actions' architecture,
| which emphasises Actions' container images that are little more
| than packaged installer scripts that go and download dependencies
| from random parts of the internet at runtime. An "attestation"
| makes no guarantee that one of these randomly fetched
| dependencies hasn't been usurped.
|
| This is not to go into the poor security record of Github
| Actions' permissions model, which has brought us all a number of
| "oh shit" moments.
___________________________________________________________________
(page generated 2024-11-14 23:00 UTC)