[HN Gopher] What happens when you leak AWS credentials and how A...
___________________________________________________________________
What happens when you leak AWS credentials and how AWS minimizes
the damage
Author : tiborhercz
Score : 168 points
Date : 2023-04-06 10:55 UTC (12 hours ago)
(HTM) web link (xebia.com)
(TXT) w3m dump (xebia.com)
| nikanj wrote:
| It's interesting how well cryptos have adapted to monetizing
| stolen aws credentials.
| pmx wrote:
| I can understand how AWS can react quickly using the secrets
| scanning service but how do malicious actors do it without having
| access to that service?
| Gordonjcp wrote:
| Luck, I guess?
|
| Maybe they have a bot watching some "latest thing on github"
| API, that scans everything coming in for credentials.
| blibble wrote:
| there's a public firehose feed of everything on github
|
| https://api.github.com/events
|
| (not sure why really)
| andy_ppp wrote:
| It says in the docs this API is delayed 5 minutes though. So
| no idea how they did things this quickly unless you can add a
| webhook to every repository somehow as it is created?
|
| It's probably just polling and luck to be honest.
| chpatrick wrote:
| Maybe the 5 minutes delay is so partners like AWS get a
| chance to scan first.
|
| https://docs.github.com/en/code-security/secret-
| scanning/abo...
| lordnacho wrote:
| Has anyone got a script or git hook that will suggest you not to
| push anything that looks like a credential? Could be really
| useful.
|
| It would need to identify strings that look like a key/password,
| and I guess also it would need an override, perhaps a comment
| like "// Demo password".
| Dunedan wrote:
| The excellent pre-commit framework (https://pre-commit.com/)
| has a hook for that in its official hook collection:
| https://github.com/pre-commit/pre-commit-hooks#detect-aws-cr...
| Jeremy1026 wrote:
| GitHub has some protections that you can enable to help prevent
| this.
|
| https://github.blog/2022-04-04-push-protection-github-advanc...
| Xylakant wrote:
| Note: paid add-on, and I believe it requires an enterprise
| subscription to start with.
|
| There exists some open source tooling in the same space.
| jonfw wrote:
| I've used sync-secrets before, which has worked well-
| https://github.com/Yelp/detect-secrets
| jrochkind1 wrote:
| I use one a git pre-commit hook that according to my notes i
| copied from here (at some point in the past, it may have
| changed since then):
|
| https://gist.github.com/czardoz/b8bb58ad10f4063209bd
|
| Then adapted with a better regex for secret keys I got from
| here: https://github.com/awslabs/git-
| secrets/blob/80230afa8c8bdeac...
|
| (I think maybe I was getting false positives before)
| aeyes wrote:
| There are several tools which can be used with pre-commit, for
| example trufflehog.
| javajosh wrote:
| 12:33:12 - Pushed the credentials to GitHub 12:34:19
| - The AWSCompromisedKeyQuarantineV2 policy is attached to the IAM
| user test-user by AWS
|
| Ah, so my exploit must detect and execute within about a minute.
| Seems doable.
| prepend wrote:
| This is the equivalent of the crime movie where they start the
| heist and then start the stopwatch.
|
| "Ok, secret is pushed, we have 77 seconds until Amazon shuts us
| off. Let's go!!!!"
| dylan604 wrote:
| Where are you getting the additional 10 seconds to pull off
| this heist?
| prepend wrote:
| Ha, the magic of my bad arithmetic. It should be 67
| seconds.
| rzzzt wrote:
| Harold Faltermeyer's tune kicks into high gear:
| https://www.youtube.com/watch?v=XfYgW3Uj9g4&t=107s
| sebzim4500 wrote:
| Doable for an automated system. Anything that requires human
| intervention would be way harder.
| janikvonrotz wrote:
| OpenAi does the same. Can this somehow used to quarantine other
| credentials? F.e. is it checking the key and the secret? Or is
| exposing the key sufficient?
| belter wrote:
| "The Anatomy of an AWS Key Leak to a Public Code Repository"
| (2019) - https://blog.technodrone.cloud/2019/03/the-anatomy-of-
| aws-ke...
| cloudginger wrote:
| Interesting blog, thanks for sharing!
| bradreaves2 wrote:
| Really interesting experiment!
|
| To answer "how do the bad guys find them so quickly" the answer
| is "judicious queries to the GitHub API."
|
| My research group describes how this works, and what we could see
| with it, in this paper: https://bradreaves.net/mmr19.pdf
|
| We're currently working on follow up work to make secret
| management better (with NSF funding[1]), and we're always looking
| for research collaborators[2].
|
| [1]
| https://nsf.gov/awardsearch/showAward?AWD_ID=2055554&Histori...
|
| [2] https://bradreaves.net/collaborate/
| BillinghamJ wrote:
| Tailing this presumably: https://api.github.com/events
| macintux wrote:
| Apparently not; there's reportedly a 5 minute delay on that
| feed.
| cloudginger wrote:
| I really liked this blog, thanks for sharing!
| orf wrote:
| I set up a project[1] to automatically leak AWS secrets published
| to the Python package index, which then triggers the secret
| scanning process and quarantines the keys[2]
|
| 1. https://github.com/pypi-data/pypi-aws-secrets
|
| 2. https://github.com/pypi-data/pypi-aws-
| secrets/blob/main/keys...
| djbusby wrote:
| Chaotic Good?
| remram wrote:
| It is really silly that the only easy way to report a leaked
| secret from most services is to push it to GitHub. I don't
| understand how the industry failed to standardize any secret-
| reporting scheme.
| orf wrote:
| How would this work in practice? In order to effectively
| filter for tokens you need a set of pretty specific regular
| expressions.
|
| These specific expressions may contain non-public information
| about the structure of the tokens.
|
| But imagine we live in a world where companies are OK with
| exposing this information publicly and investing time in
| keeping the expressions up to date.
|
| Now what? How do you report it to services whilst stopping
| abuse? Like, if the code isn't public then all this is an
| endpoint to let anyone to disable another users (or
| companies) key without any explanation or recourse.
|
| Pushing to GitHub is actually quite an elegant solution to
| these issues I think, even if I do think it is a bit too
| centralized.
| mcspiff wrote:
| I mean yes -- that's the point. If untrusted parties have
| access to your keys, it's already game over. You've lost.
| Disabling them is the nicest thing an attacker can do for
| you.
| orf wrote:
| Not really: it depends on the permissions assigned to the
| keys.
|
| I wouldn't like to wake up to an email that says "your
| key has been disabled because someone anonymously
| reported is as leaked, sorry if this has broken your
| entire system".
|
| What do you do with this, outside of obviously
| quarantining and/or disabling the key? How was it leaked?
| What's the context?
| blackoil wrote:
| If rogue party has access to the key, they can do a lot
| more damage.
| orf wrote:
| Indeed, so you're building all this tooling and
| complexity and introducing more issues for the very small
| intersection of people that:
|
| 1. Are not malicious
|
| 2. Have access to a key
|
| 3. Are unable or unwilling to commit it to GitHub
|
| It would be great if this stuff was public and available
| without a central authority. But after working on it for
| a while it seems like a fairly good compromise.
| Kwpolska wrote:
| There's already an entire pipeline that handles a key
| being compromised when it is found on GitHub. All the
| "tooling and complexity" you need is a simple HTML form
| to ask for a key and where you found it, and some server-
| side code to trigger the same pipeline when somebody
| submits the form.
|
| There are three issues with the use of GitHub here:
|
| 1. Not everybody knows that AWS will invalidate tokens
| committed into a public GitHub repository.
|
| 2. There is a window (67 seconds according to OP) in
| which the compromised token is public but working. For
| the "small intersection of people", you could bring it
| down to 0.
|
| 3. GitHub protects GitHub keys, and apparently AWS keys,
| but does it protect Azure keys? Or GCP keys?
| fiddlerwoaroof wrote:
| GitHub secret scanning is a product they allow partners
| to take advantage of: https://docs.github.com/en/code-
| security/secret-scanning/abo...
| renewiltord wrote:
| This is a misunderstanding of how standards come about.
| Standards don't come about because someone theorizes pushing
| the envelope and then everyone gets together to form a
| convention to decide the best way to push the envelope.
|
| Standards describe existing behaviour of value that benefits
| from a single interface. For that, the behaviour has to be
| first demonstrated. That's how the envelope is pushed. You
| make an implementation, then you show how that solves
| something worthwhile, and then if there are multiple
| implementations that could benefit from a single interface,
| you do that.
| LelouBil wrote:
| Discord does a similar thing, if a bot toke is discovered online
| it automatically gets disabled and you get an email.
| jimt1234 wrote:
| Please be aware of leaking AWS credentials in any log output,
| especially Jenkins console output. That's _usually_ protected by
| some layer of auth, but if I get some chump 's account info, and
| something is leaking AWS creds to Jenkins console output, it's
| probably gonna end up with full account take-over.
| belter wrote:
| "What to Do If You Inadvertently Expose an AWS Access Key" -
| https://aws.amazon.com/blogs/security/what-to-do-if-you-inad...
|
| "What do I do if I notice unauthorized activity in my AWS
| account?" - https://repost.aws/knowledge-center/potential-
| account-compro...
| tiborhercz wrote:
| Ever wondered what would happen when you leak AWS credentials to
| GitHub? I was, so I leaked AWS credentials to GitHub to see what
| would happen. Both AWS and malicious actors respond very quickly
| and start taking actions on these leaked credentials.
| pritambarhate wrote:
| I think it will be great if AWS allowed us to use the AWS keys to
| be used from certain IPs only. It will add additional layer of
| security which can help to prevent a lot of misuse. Just like AWS
| best practices of putting DB instances in a private VPC can
| prevent a lot of attack vectors, ability to use AWS Keys only
| from certain IPs or IP ranges, can also prevent from misuse, even
| if one accidentally leaks certain keys.
| bpodgursky wrote:
| I don't think AWS wants to imply in any way that you need to be
| on a VPN to interact with AWS resources. It's not the direction
| they are going (Zero trust / RBA
| https://aws.amazon.com/security/zero-trust/).
| Godel_unicode wrote:
| Zero trust doesn't mean no VPNs, it means not just VPNs. In
| other words you can put your control plane behind ACLs (and
| you should!) but you should still require strong auth from
| the private network.
| bpicolo wrote:
| Haven't tried it, but should be able to use service control
| policies to do this broadly.
|
| https://docs.aws.amazon.com/organizations/latest/userguide/o...
|
| Can definitely do it at the permission policy level as shown in
| the links others provided.
| Hikikomori wrote:
| It's already possible, but on a iam action level.
| wmfiv wrote:
| Source IP is an IAM global condition so it's available for all
| requests.
|
| https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...
| circular_logic wrote:
| This looks to be possible via this guide
| https://repost.aws/knowledge-center/iam-restrict-calls-ip-ad...
| aeyes wrote:
| This example is for a role but for IAM users it works the
| same way.
| [deleted]
| mcdwayne wrote:
| Basically, rotate as soon as you can* and start looking through
| your AWS logs and setting to see if any services you don't
| recognize have been spun up. Is you think you have been attacked
| or see stuff you did not spin up, contact AWS support ASAP!
|
| *Do NOT just revoke keys if it is in a production system where
| other people are working or are depending on. Talk to your team
| and figure out what the remediation process is internally and
| follow that! If you are working by yourself and no one is relying
| on services this key is associated with, then yeah, just revoke
| and replace ASAP.
|
| Here are a couple blogs that cover what you are looking for as
| well https://blog.gitguardian.com/leaking-secrets-on-github-
| what-... https://blog.gitguardian.com/thinking-like-a-hacker-aws-
| keys...
| jrochkind1 wrote:
| > In the email they also give guidance on how to secure the
| account. They do this in four steps: 1. Rotate and delete the
| exposed AWS Access Key...
|
| I work at a very small shop (at a non-profit institution).
|
| In the past, when we've accidentally leaked a credential, AWS has
| required us to rotate _all_ of our AWS Access Keys in the
| account, not just the exposed one.
|
| Which I know ideally rotation would be easy/automated and this
| wouldn't matter, but at the time it was disruptive for us. And a
| bit frustrating when we had intentionally created separate AWS
| credentials with least privs needed for the service, etc. But
| thinking on it, I guess any given credential might give
| permissions to _create_ other credentials or other devious
| things, rotating all of them might be necessary to be safe.
|
| It's been a couple years since we had a leaked credential now
| (hooray). I wonder if the policy has changed, or if I
| misunderstood, or if in fact you still need to rotate all
| credentials in the account, maybe OP just only had one credential
| in the account they were intentionally doing insecure things so
| the compromised key was in fact all keys. (I don't specifically
| remember the `AWSCompromisedKeyQuarantineV2` policy, but probably
| am just not remembering it, I had other things to worry about at
| the times!)
| Hikikomori wrote:
| Can also avoid having any static credentials at all if
| possible.
| scarface74 wrote:
| OB: There is never a reason to have access keys anywhere in your
| source code or a config file in your source control directory.
|
| All of the AWS SDKs will get your credentials from
|
| - hardcoded credentials in your code -> don't do this
|
| - environment variables - you get this when running locally from
| your control tower interface that most companies access via SSO
| via AD/Okta etc
|
| - your global configuration file.
|
| When running on AWS, your credentials are retrieved via the
| attached IAM Role for Lambda, ECS, EC2, etc.
|
| I have done one third party integration where I used Azure DevOps
| to integrate with AWS using the AWS tools for Azure DevOps. Even
| then you store your credentials in an Azure vault separate from
| your source code.
| nerdjon wrote:
| This is one of those things that I was somewhat aware existed
| since we had seen various reports of it, but I was never fully
| aware of the speed (and obviously automated) of it.
|
| Is this actually documented by aws anywhere? I assume other cloud
| providers must be doing something similar.
| flerchin wrote:
| It's not clear to me that we have proof that it was a malicious
| actor. Was it perhaps AWS automation testing out the validity of
| the credentials before applying the leaked credential policy?
| ceejayoz wrote:
| The AWS process is hopefully "does this match a secret we know?
| invalidate!" and nothing more. Relying on "can this key do x?"
| would be a problem, because it might be able to do x _in the
| future_ , and it doesn't appear they're testing _all_ possible
| capabilities either.
| bdcravens wrote:
| A few years ago this wasn't in place. I goofed and leaked a
| credential via Github, and woke up to $50k and climbing in AWS
| fees (Monero miners). AWS worked with us, and once we proved we
| had adequately locked our account down and added additional fail-
| safes, they took it off of our bill.
___________________________________________________________________
(page generated 2023-04-06 23:01 UTC)