[HN Gopher] Heroku April 2022 Incident Review
___________________________________________________________________
Heroku April 2022 Incident Review
Author : glennericksen
Score : 98 points
Date : 2022-06-14 17:43 UTC (5 hours ago)
(HTM) web link (blog.heroku.com)
(TXT) w3m dump (blog.heroku.com)
| drusepth wrote:
| Side note on Heroku: they've been very aggressively upgrading old
| app dependencies this week (I've had ~9 apps go down for
| mandatory Postgres maintenance in the last 24h, with almost no
| notice for some of them). With how little information they've
| given regarding this incident, I can't help but wonder if it's
| related.
| btown wrote:
| > Additionally, according to GitHub, the threat actor accessed
| and cloned private repositories stored in GitHub owned by a small
| number of our customers. When this was detected, we notified
| customers on April 15, 2022, revoked all existing tokens from the
| Heroku Dashboard GitHub integration, and prevented new OAuth
| tokens from being created.
|
| Various customers received an email from Heroku on April 15
| saying "We value transparency and wanted to notify you of an
| incident we're actively investigating that may lead to
| unauthorized access to your GitHub repositories connected to
| Heroku."
|
| The way this incident review (to call it a post-mortem would be
| an insult to those who write good post-mortems) phrases things,
| customers have no way of knowing if that email meant they were
| one of those "small number of customers" or not. And what is a
| small number, anyways? Is 49% of customers a small number of
| customers? It's an absurd situation.
| ezekg wrote:
| > We began investigating how the threat actor gained initial
| access to the environment and determined it was obtained by
| leveraging a compromised token for a Heroku machine account. We
| determined that the unidentified threat actor gained access to
| the machine account from an archived private GitHub repository
| containing Heroku source code. We assessed that the threat actor
| accessed the repository via a third-party integration with that
| repository. We continue to work closely with our partners, but
| have been unable to definitively confirm the third-party
| integration that was the source of the attack
|
| So they still don't know how it happened.
| jrochkind1 wrote:
| That's basically the only paragraph of new information in the
| whole... well, let's call it what it is, a press release
| basically.
|
| It is a thing to know. That they had a token checked into
| source code that probably shouldn't have been, and that the
| attacker somehow got access to the source code in a private
| github repo.
|
| * yeah, we still don't know how. Which is kind of important.
| But figuring out how they got access to a private github repo
| is to some extent back on github, at least potentially...
|
| * it to some extent points back to github (fairly or unfairly);
| so how did they get access to the source code you were supposed
| to be protecting?
|
| * it certainly reminds us the readers why we don't put secrets
| in source code repos. (Of course, they ultimately need to be
| stored _somewhere_ , and that somewhere can always be breached.
| But having them in as few places as possible and places
| designated specifically for secrets, we can make sure that
| things like a Github OAuth token for a Github integration
| doesn't give access to your deploy secrets...)
|
| * Which again makes me wonder... "via a third-party integration
| with that repository"... WAS it an integration that didn't even
| need source code read access, but had it due to Github's
| terribly non-specific integration auth permissions? I would
| love some more attention to that _github_ problem as a result
| of this hullaballo. I kind of can 't believe fixing integration
| permissions granularity has been such a low priority for a
| fairly well-resourced github. I guess it doesn't sell more
| accounts... it just loses you some once it results in a
| vulnerability, if that becomes known.
| nightpool wrote:
| > is to some extent back on github
|
| How is it on Github? We know that it was a third-party
| integration that compromised. Almost every single third-party
| integration needs the ability to read source code. The fault
| here lies on Heroku for storing secrets that allowed access
| to their main customer database in a source code repo that
| was accessible to a third-party provider, as well as with the
| third party provider (whatever it was) for allowing their
| implementation to be compromised. At that point it's game
| over--your main database should never be accessible from your
| source code alone.
|
| It is strange to me that they're certain it was accessed
| through a third-party Github integration, but they don't know
| _which_ integration it was specifically. That feels like a
| failure of logging on Github 's part, without any additional
| information.
| jrochkind1 wrote:
| > How is it on Github?...
|
| The whole sentence I wrote was: "But figuring out how they
| got access to a private github repo is to some extent back
| on github, at least potentially..."
|
| > ...That feels like a failure of logging on Github's part,
| without any additional information.
|
| There you go, you answered your own question too.
| jarcoal wrote:
| It's a little hard to parse parts of that paragraph, but it
| sounds like the repo (presumably hosted on GitHub) had access
| tokens granted to third party integrations (similar to Heroku
| being granted access to GitHub on behalf of their mutual
| users).
|
| Assuming that's true, it should be trivial for GitHub to tell
| them which third party integration the token was associated
| with.
| [deleted]
| ectopod wrote:
| AIUI, the repo contained a single token that gave access to
| Heroku. Additionally, a bunch of third party tools had
| legitimate access to the repo. Any one of them could have
| been used to steal the token.
| [deleted]
| dentarg wrote:
| Maybe it was via Travis CI? https://blog.travis-
| ci.com/2022-04-17-securitybulletin
| jansan wrote:
| Is it known who at Github can access private customer repos?
| vorpalhex wrote:
| I'd assume everyone who works there. They aren't encrypted or
| anything. If you have hardware access you have full github
| access.
| [deleted]
| jeremymcanally wrote:
| But I can definitively say that any repo access is heavily
| logged. And this info is from like almost 10 years ago when
| I had first hand experience. I know they've upped their
| game in terms of repository access gating since then.
___________________________________________________________________
(page generated 2022-06-14 23:01 UTC)