[HN Gopher] GitHub waited 3 months to notify about potential com...
___________________________________________________________________
GitHub waited 3 months to notify about potential compromise
It seems GitHub became aware of the issue around March 2 (or
before, since the fix was released on March 2), and waited until
June 16 to disclose the problem. See full text below: Hi zwass,
We're writing to let you know that between 2022-02-25 18:28 UTC and
2022-03-02 20:47 UTC, due to a bug, GitHub Apps were able to
generate new scoped installation tokens with elevated permissions.
You are an owner of an organization on GitHub with GitHub Apps
installed that generated at least one new token during this time
period. While we do not have evidence that this bug was maliciously
exploited, with our available data, we are not able to determine if
a token was generated with elevated permissions. User privacy and
security are essential for maintaining trust, and we want to remain
as transparent as possible about events like these. GitHub itself
did not experience a compromise or data breach that either created
or resulted from this event. Read on for more information. * What
happened? * GitHub learned via a customer support ticket that
GitHub Apps were able to generate scoped installation tokens with
elevated permissions. Each of these tokens are valid for up to 1
hour. GitHub quickly fixed the issue and established that this bug
was recently introduced, existing for approximately 5 days between
2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC. GitHub Apps
generate scoped installation tokens based on the scopes and
permissions granted when the GitHub App is installed into a user
account or organization. For example, if a GitHub App requests and
is granted read permission to issues during installation, the
scoped installation token the App generates to function would have
`issues:read` permission. This bug would potentially allow the
above installation to generate a token with `issues:write`, an
elevation of permission from the granted `issues:read` permission.
The bug did not allow a GitHub App to generate a token with
additional scopes that were not already granted, such as
`discussions:read` in the above example. The bug did not allow a
GitHub App to access any repositories that the App did not already
have access to. In order to exploit this bug, the GitHub App
author would need to modify their app's code to request elevated
permissions on generated tokens. * What information was involved?
* The following GitHub Apps generated scoped installation tokens
during the bug window for your organization(s). We are not able to
determine if a token was generated with elevated permissions.
Organization: GitHub Apps <redacted> * What GitHub is doing *
GitHub immediately began working to fix the bug and started an
investigation into the potential impact. However due to the scale
and complexity of GitHub Apps and their short-lived tokens, we were
unable to determine whether this bug was ever exploited. We are
notifying all organization and user account owners that had GitHub
Apps installed and had a scoped installation token generated during
the bug window so that they can stay informed and perform their own
audit of their installed GitHub Apps. As a followup to this
investigation, GitHub is looking at ways to improve our logging to
enable more in-depth analysis on scoped token generation and GitHub
App permissions in the future. * What was the potential impact? *
Due to the variety of GitHub Apps, their possible scopes, and the
repositories they may have been given access to, we are unable to
advise on any potential impacts as each customer's situation will
be unique. * What you can do * While we have updated our systems
to resolve this bug and no action is required on your end, we do
recommend you review your installed GitHub Apps. You can use the
following guidance for assessing GitHub Apps, their permissions,
and their access to your private organization repositories:
https://docs.github.com/en/organizations/keeping-your-organization-
secure
Author : zwass
Score : 118 points
Date : 2022-06-16 18:47 UTC (4 hours ago)
| smarx007 wrote:
| I just got it as well and don't understand what I can do. Can I
| somehow force all generated tokens to be revoked and get apps to
| generate new tokens to be on the safe side? Or, rather, is there
| a way to do this without uninstalling the apps and installing
| them again?
| onphonenow wrote:
| Aren't these very time limited anyways (hours vs days)? So once
| they fix the bug and wait an hour the old tokens are gone
| anyways?
| mynameisvlad wrote:
| Per the email:
|
| > Each of these tokens are valid for up to 1 hour.
|
| So these tokens would've been expired for 3 months now,
| according to when the fix was deployed.
| nrmitchi wrote:
| >Each of these tokens are valid for up to 1 hour.
|
| > GitHub quickly fixed the issue and established that this bug
| was recently introduced, existing for approximately 5 days
| between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.
|
| It doesn't sound like there is anything you can or need to do
| with respect to these tokens (whether they were used to take
| action with elevated permissions is a different thing, but it
| doesn't sound like that was the case either)
| sascha_sl wrote:
| My recent experience with GitHub regarding a security issue was
| not very positive either.[1] It turned out, unlike two vendors I
| notified that were affected, they just didn't care. And they
| didn't bother to even tell me that they didn't care.
|
| It's a very edge-case issue in Enterprise SSO, so I wasn't really
| able to generate any blowback with disclosure either. But if you
| find an org with just the right setup it blows a huge hole into
| the SSO product, to the point of making it useless.
|
| There also seems to be an asymmetry between the core product and
| everything else. GitHub Enterprise has issues that aren't even
| considered UX issues (i.e. notifications showing "3 of 0"
| notifications if no SAML session exists) that'd warrant bounties
| if they were in the core product.
|
| [1]: https://notes.acuteaura.net/posts/github-enterprise-
| security...
| throwaway78246 wrote:
| My guess is that GitHub ignores issues so they don't have to
| pay out bug bounties.
| onphonenow wrote:
| I find these takes so silly. Bug bunties are a rounding error
| in the companies budgets, even if they paid out much more
| freely. There are many I think much more obvious reasons orgs
| are slow on issues - everything from figuring what is an
| issue, trying to chase down impacts and more.
| bearjaws wrote:
| I am really curious how SecOps works at GitHub.
|
| Why not after remediation inform users of the flaw and potential
| impact. Then follow up with detailed impact.
|
| Instead we get this 3 months later all they can say is "Some of
| your apps refreshed their tokens during a 5 day period" which is
| not news...
|
| This is also the second time this year there has been significant
| delay in communication. Granted those involved other third
| parties so who knows where the delay lived.
| mistrial9 wrote:
| they might assume that the account holder are a likely
| perpetrator, which might be true but also enables their
| intermediation of the context and control of the sequence of
| communication.
| s09dfhks wrote:
| I would assume the lawyers have to get involved first to write
| up some document proving that github is not liable for what
| users do with their app tokens. CYA, then tell the public
| cypherg wrote:
| They explained it themselves - there's no evidence of
| abuse/exploitation. They literally have no legal requirement to
| even tell you as much as they did. You should be commending them
| for filling you in at all.
| chrisseaton wrote:
| > They literally have no legal requirement to even tell you as
| much as they did.
|
| Is 'fulfilling legal requirements' all you look for in a
| business relationship?
|
| A restaurant has no legal requirement to make this food tasty
| but it's what I'm looking for when choosing where to go.
| akagusu wrote:
| Since almost every popular tech company is a quasi monopoly,
| they use this "fulfilling legal requirements" strategy to
| abuse the market providing overpriced services with bad
| quality.
|
| Unfortunately, people got used to this practice and gladly
| accept when such companies fulfill all their legal
| obligations, even when this hurt them or their business.
| gtirloni wrote:
| How much is GitHub overpricing their bad quality services?
| zwass wrote:
| I would be commending them if this notice went out March 3
| after they had remediated the problem and were aware that they
| had no logs to determine whether there was abuse.
| dogecoinbase wrote:
| > ... we were unable to determine whether this bug was ever
| exploited.
|
| > ...
|
| > Due to the variety of GitHub Apps, their possible scopes, and
| the repositories they may have been given access to, we are
| unable to advise on any potential impacts as each customer's
| situation will be unique.
|
| Absence of evidence is not evidence of absence.
| [deleted]
| hanble wrote:
| That's true, but feels like these are always judgment calls.
| We can always armchair quarterback their judgment calls, but
| none of us have the full info. At least GH is sharing this
| info, which is a good call for trust building IMO.
| averysmallbird wrote:
| That's not fully Github's choice to make. They made a
| judgement call based on seemingly incomplete evidence, and
| have different incentives that everyone else.
|
| Repository owners may well have a different level of
| acceptable risk or legal obligations over the integrity of
| their source code. For example, if I was maintaining
| security software or a popular package, it would be
| entirely appropriate to stop everything and look for abuse.
| Waiting three months makes that harder.
|
| I'm not sure that's trust building.
| throwaway78246 wrote:
| It was also my experience that it takes GitHub several months to
| fix issues and inform compromised users.
| lambada wrote:
| Given it existed for 5 days and you're only now finding out about
| it, it sounds to me like it was perhaps a bug that was fixed
| without realising the full impact of it, or perhaps without
| realising it made it to production; and only an audit that
| happened later caught it.
|
| Not ideal by any means. I'd be curious to know if my theory is
| correct or not.
| zwass wrote:
| Their statements indicate they were aware and investigating. My
| frustration is that they didn't give users the opportunity to
| do their own timely investigation.
|
| > GitHub learned via a customer support ticket that GitHub Apps
| were able to generate scoped installation tokens with elevated
| permissions. Each of these tokens are valid for up to 1 hour.
|
| > GitHub quickly fixed the issue and established that this bug
| was recently introduced, existing for approximately 5 days
| between 2022-02-25 18:28 UTC and 2022-03-02 20:47 UTC.
|
| > GitHub immediately began working to fix the bug and started
| an investigation into the potential impact. However due to the
| scale and complexity of GitHub Apps and their short-lived
| tokens, we were unable to determine whether this bug was ever
| exploited.
| njibhu wrote:
| Can somebody tell me if I'm wrong on my take but this bug/issue
| means:
|
| - a github app which had read permission on issues could elevate
| its permission to write
|
| - a github app which had read permissions to discussions could
| elevate its permissions to write.
|
| So far if the org/user would have been compromise they would have
| seen with issues or conversations containing content from the
| app.
|
| Since these are only examples, I can imagine the case with major
| impact would be a contents:read elevate to content write. But
| again with commit signing, this would also be caught by the user.
| What did I miss where the impact would have not been visible to
| the end user/org ?
| zwass wrote:
| contents:read to contents:write is a big deal! Just to pick out
| a random widely used project, nodejs [1] has a number of
| unsigned commits to the main branch. Their commits could have
| been tampered with during this timeframe.
|
| What about release artifacts?
|
| [1]: https://github.com/nodejs/node/commits/main
| njibhu wrote:
| I guess I can see it, but branch protection rules and pull
| requests reviews would also prevent that to happen in my
| opinion
|
| (also ability to do it with content:write is just speculation
| from my side, they don't make it clear if it is possible,
| that would need to be confirmed by github)
___________________________________________________________________
(page generated 2022-06-16 23:01 UTC)