https://www.chainguard.dev/unchained/working-as-unexpected Unchained [63d340f327] Products Chainguard Images Reduce your attack surface with hardened container images. Solutions Container Image Security Run hardened container images. Vulnerability Remediation Eliminate CVEs daily. Open Source Software Security Consume OSS safely. Compliance & Risk Mitigation Meet and maintain compliance. Software Supply Chain Security Build secure software by default. Developer Docs [63d33b83a6] Open source Resources Unchained blog Chainguard labs Customer stories Events Security Education [63d33b83a6] Company About Newsroom Careers Chainguard love Sign inContact Unchained [63d340f327] Products Chainguard Images Reduce your attack surface with hardened container images. Solutions Developer Resources Company Sign InContact Back [63d340f327] Container Image Security Run hardened container images. Vulnerability Remediation Eliminate CVEs daily. Open Source Software Security Consume OSS safely. Compliance & Risk Mitigation Meet and maintain compliance. Software Supply Chain Security Build secure software by default. Back [63d340f327] Docs [63d33b83a6] Open source Back [63d340f327] Unchained blog Chainguard labs Customer stories Events Security Education [63d33b83a6] Back [63d340f327] About Newsroom Careers Chainguard love Back [63d340f327] About Newsroom Careers Engineering Working as unexpected Matt Moore, CTO Chainguard May 31, 2024 copied TLDR: This is a tale of a "working as intended" branch protection bypass that allows for protected credential exfiltration. This is a security vulnerability that was reported to GitHub via HackerOne on February 2nd, 2024, and fairly quickly closed as "working as expected." While GitHub may expect this behavior, it violates the principle of least surprise, and so I wanted to outline this vulnerable behavior so that folks don't fall into the trap it creates. As part of hardening Chainguard's security posture, I am continuously in pursuit of ways to leverage controls to treat GitHub like a production environment. On this particular excursion, I was exploring whether I could eliminate the ability to create new branches directly on our upstream repository with a wildcard branch protection *: Image showing GitHub branch name pattern creation with wildcard branch protection. My hypothesis was that the above might prevent folks from creating new branches on our upstream repository (effectively that this would protect non-existing branches in addition to existing branches). It was simple enough to test with an experiment, and I was (partially) wrong. However, now that the branch exists, it (somewhat obviously) shows up as protected. Hmm... I found this intriguing because I had also recently been exploring the use of GitHub's environments feature, which allows you to restrict the visibility of certain secrets to specific branches or to protected branches. Bear in mind, this feature is the most secure level of secret storage GitHub offers. You can configure it like so: Image showing mattmoor-testing GitHub environment configuration. So my very next question was: If my new branch immediately becomes protected, could its workflows immediately be eligible to access these secrets? To test that theory, I crafted the following workflow to exfiltrate a secret I created in the mattmoor-testing environment above called NOT_A_SECRET: -- CODE language-bash -- on: push: branches: name: example secret exfiltration jobs: build: runs-on: ubuntu-latest environment: mattmoor-testing steps: - shell: bash run: | echo ${{ secrets.NOT_A_SECRET }} | base64 Somewhat unsurprisingly this worked: Image showing Run echo * | base64. Image showing melange line indicating "this should be a secret." The full reproduction steps I gave to GitHub are: [66587b44ef] At first, I thought (likely similar to GitHub) that this was a vanishingly small niche. After all, how widely used are wildcard branch protection rules that folks expose secrets to? However, the more it marinated in my head, the more it concerned me, and a very plausible scenario emerged. Digging deeper On previous open source projects I have worked on, it was pretty typical to create long-lived release branches from which we could cut patch releases (e.g. release-1.2). We protected these branches with a protection rule that applied to release-*. Now suppose that we wanted our release workflows to have access to sensitive secrets that only the release workflow should have, say for instance signing keys (e.g. terraform provider GPG keys, deb, rpm, apk signing keys). Gulp. Initially, I had also questioned the value of such an attack with excuses like: my branch was pretty obvious (it is protected, so I couldn't just delete it to cover my tracks), the extra workflow was pretty obvious, and the way I exfiltrated the secret made it available to anyone (not just me). This also did not age well as it marinated in my head. For projects with many releases I could very likely hide my branch typo-style with something like release-1.02 or release-.1.2, which would look like an innocent error. To hide the workflow, I could put the exfiltration itself into an existing workflow, so it blended into the other action executions of the project. Lastly, instead of using something as trivial as base64 to avoid GitHub's secret masking I could do something that made it only accessible to me: I could post it to a service I own. Or if the execution were somehow network jailed (not something Actions supports), then I could encrypt it with an asymmetric key included in the branch and log the encrypted value instead. But I'd have to be a writer on the repo. True, but this is also more common than you'd think. After all, GitHub locks up all kinds of useful permissions behind having this level of access, including things like being able to label issues, move them around project boards, or interact with milestones. In fact, the main feature that previously allowed me as a maintainer to sleep at night with a non-trivial number of repo editors was ironically ... (drumroll) ... branch protections. Now here's a piece I missed in my initial branch protection configuration, which helps to mitigate this, but does not completely close the hole. I'd missed this because of a subtle text difference between what is displayed for new branch protections (above) vs. existing branch protections (below): Image showing subtle text differences between what is displayed for new branch protections (this image) and existing branch protections (next image). New branch protections Image showing subtle text differences between what is displayed for existing branch protections (this image) and new branch protections (previous image). Existing branch protections I was looking at an existing branch protection when exploring these knobs, which indicates that anyone with the ability to write to the repo can still create branches. However, the left hand side indicates that only administrators can create new branches (progress, but ideally this would be configurable with): Image showing that only administrators can create new branches. Best practices for branch protections If you are concerned about vulnerable behaviors in GitHub branch protections, here are some takeaways and best practices to consider: 1. Favor the use of repository rulesets (new) over branch protections (old), which can actually block administrators if they are not explicitly put onto the bypass list. 2. Only use wildcards in branch protections when absolutely necessary. 3. When using wildcard branch protections always restrict who can create matching branches (e.g. so that only admins can create release branches). 4. Only trust branch protections in environments (vs. concrete branches) when absolutely necessary. 5. If you do find yourself doing the above, consider requiring administrators to approve environment access. 6. Prefer the use of a cloud service provider's secret store accessed via OIDC federation over GitHub's built-in secret storage, and ensure that the federation rules are as restrictive as possible. As I mentioned earlier, GitHub marked this issue as working as intended, but (silver linings) it freed me to at least help educate you all to be on the lookout for vulnerable behaviors like this. [66587d1e9b] If you are interested in learning more about Chainguard's approach to our own product security and internal practices with defense-in-depth principles, visit our Trust Center or learn more about our Octo STS project here. Share on Twitter Share on LinkedIn Share on Email Related articles [6657739ef2] Engineering Wolfi at work: Minimal developer workstations in the cloud May 29, 2024 [66439bb2b9] Engineering Reducing vulnerabilities in Backstage with Chainguard's Wolfi May 16, 2024 [6632674dac] Engineering Open sourcing Octo STS May 2, 2024 Ready to lock down your supply chain? Talk to our customer obsessed, community-driven team. Get started Products Chainguard Images Solutions Container Image Security Vulnerability Remediation Open Source Software Security Compliance & Risk Mitigation Software Supply Chain Security Developer Open source Docs Resources Unchained blog Chainguard Labs Customer stories Scanners Security Education Company About Newsroom Careers Chainguard love Legal Contact Follow Newsletter Twitter GitHub LinkedIn TikTok (c) 2023 Chainguard, Inc. Contact Products Chainguard Images Solutions Container Image SecurityVulnerability RemediationOpen Source Software SecurityCompliance & Risk MitigationSoftware Supply Chain Security Developer Open Source Docs Resources Unchained blogChainguard labsCustomer storiesScannersSecurity Education Company AboutNewsroomLegalCareersChainguard love Follow Newsletter Twitter GitHub LinkedIn TikTok Contact Be the first to know Sign up for Chainguard emails to be the first to see product updates, news, and events. [643774c345]