[HN Gopher] The Cloud Conundrum: S3 Encryption
___________________________________________________________________
The Cloud Conundrum: S3 Encryption
Author : daitya
Score : 42 points
Date : 2023-01-16 18:29 UTC (4 hours ago)
(HTM) web link (www.secwale.com)
(TXT) w3m dump (www.secwale.com)
| 0xbadcafebee wrote:
| > Next, at best SSE-S3 adds a defense in depth protection against
| a physical loss, theft or confiscation of an AWS hard drive
| storing your data. Think crazy scenarios like a tornado or fire,
| followed by more chaos and somehow the AWS hard drive landing at
| Goodwill. If the data on it is unencrypted, game over. As you can
| imagine, the likelihood of this happening is about the same as
| that of the United States winning a cricket world cup.
|
| It's incredibly easy to dumpster dive old hardware from
| datacenters (or waste management facilities). Even if their
| policy is to demagnetize and shred every hard-drive out of every
| dead machine, that's assuming they always follow their policy
| correctly. Humans are flawed (and subject to bribes).
|
| I have friends who run K8s clusters on hardware they took from
| the dumpsters behind datacenters. (Not Amazon datacenters, but
| ones where the government has an entire floor dedicated to them)
| daitya wrote:
| AWS will now encrypt all new data in its Amazon S3 storage
| service by default. Huge announcement, secure default for the
| win, sure, but it gives a false sense of security.
| tgv wrote:
| AFAIK, it only helps against hackers/insiders who steal the raw
| file. If you provide access to your bucket, it isn't doing
| anything. As always with Amazon (and other services, really):
| you really should understand the service. Just using one
| because you read "encryption" is unwise.
| limitedsupply wrote:
| Wouldn't your example apply to any encrypted hard drive as
| well? If you provide access to it encryption is not doing
| anything.
| NathanKP wrote:
| It's all about layers of protection.
|
| This layer of server side encryption is designed to protect
| against an attack where someone breaks into an AWS data center,
| pulls a disk drive out of a storage system that is hosting S3
| objects, and then tries to read the data off of that disk
| drive. Without the key (which is obviously stored separately,
| on a separate system) the disk will be useless to them.
| ilyt wrote:
| Which is like _the_ least likely attack to happen.
| zxcvbn4038 wrote:
| I believe all of the AWS S3 SDKs except php support client side
| encryption, you could also roll your own. In my mind that is more
| secure then S3 SSE or KMS because you are the only one with the
| key.
| lokar wrote:
| I surprised they did not always encrypt data at rest, google
| does.
| mmoskal wrote:
| Legacy? Google public cloud operation is relatively recent.
| boulos wrote:
| GCS launched publicly in 2010 [1]. That's still 4 years after
| S3, but it's not "recent".
|
| [1] https://en.m.wikipedia.org/wiki/Google_Cloud_Storage
| foota wrote:
| I was amazed when I read the other week it wasn't already
| encrypted at rest by default.
| icedchai wrote:
| You could enable it on the bucket level for many years now.
| This would apply to all new objects in the bucket. So if you
| enabled it at bucket creation time, it was effectively "by
| default."
| frenchman99 wrote:
| People here say that this doesn't secure anything. But in
| addition to encryption by default, AWS announced they will as of
| April 2023 enforce secure S3 configuration (no public access) by
| default. So things are definitely getting better when you put
| these announcements together.
|
| https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security...
| jbverschoor wrote:
| Well it's kind of insane that it has been open by default,
| resulting in SOOOO many leaks.
|
| They might as well have open SSH access on all EC2 instances
| using root + "password"
| beaviskhan wrote:
| That's not entirely fair - the default has been to ALLOW
| people to set buckets to public and/or to set public ACLs on
| individual objects. Both of these require the customer to Do
| Something in order to grant public access. The default has
| NOT been "public access for all" ever, that I am aware of.
| fdr wrote:
| This definitely dates back to when offloading images and
| other static assets to S3 from your (era-appropriate) Apache
| or Varnish server was an optimization of the day...now,
| obsolete, as, at the time, there were not low-friction,
| lower-cost CDNs available, such as Cloudfront or Cloudflare.
| davewritescode wrote:
| What's always been unclear to me is how much I actually gain from
| providing my own KMS key use with S3 if at the end of the day my
| private key is available to the s3 service.
|
| I would only use client side encryption/decryption if I had
| really sensitive data and I truly couldn't trust AWS with it.
| ldoughty wrote:
| As you said, using a key they generated (but not the default
| key) doesn't protect you from AWS-side attacks.
|
| The main advantage is more control over the key & permissions
| on your side of the shared-responsibility model.
|
| AWS default key has a fixed policy and is wide open to your
| account, so it's not suitable for:
|
| - Using different keys for different situations/data within the
| account; this can help with defense-in-depth & errors in least
| privilege from escalating. Example: Lambda accidentally has
| overly-permissive S3 permissions, and you're using the default
| key. It could potentially read more S3 buckets/files than
| intended. With different keys, it couldn't decrypt the data in
| other buckets.
|
| - Using different accounts. Default keys are only usable within
| your account, you can't grant access to use it to other
| accounts (which might sound odd, but there's use cases where
| you'd consider doing this)
|
| - Default keys can't be used by some AWS services, e.g. the
| default key policy can't be used by CloudWatch Events to
| enqueue to an encrypted SQS queue, for instance. (workaround is
| adding a resource-policy to the queue, but I personally do not
| like resource-based policies -- I like all my permissions in
| IAM roles & policies so there's one place to audit and control)
|
| All this said, I often leverage the default keys heavily, as
| using your own KMS key is more expensive.
| paranoidrobot wrote:
| Assuming for a moment that AWS's security at all matches the
| public API.
|
| There are two ways to grant access to a KMS key: as a policy
| grant on a user/role, and as a policy on the KMS key itself.
|
| If you don't have access to the key, then reading the S3 object
| doesn't work.
|
| So I have to trust that AWS's security here is that the S3 API
| itself can't unless granted that access.
| theteapot wrote:
| > If you don't have access to the key, then reading the S3
| object doesn't work.
|
| I'm still not convinced. Just seem like theater. S3 has its
| own access control. If you don't have access to the S3 object
| then reading the S3 object doesn't work ... Requiring
| additional access to the key seems redundant - theater.
|
| P.S. I'm not arguing against encryption at rest. I'll take
| it, but using my own key doesn't seem to offer too much over
| using one of AWS's rel low level access.
| colmmacc wrote:
| AWS isn't monolithic internally and we use permissions
| boundaries between systems like S3 and KMS (in fact they
| are mediated by IAM). There's some more detail in my talk
| from re:Invent -
| https://www.youtube.com/watch?v=kNbNWxVQP4w .
|
| For a case like this, the upshot of it is that with
| permissions on the KMS key, S3 is effectively locked out
| and can't fetch it. KMS is a hermetic system with no
| operator access with HSMs at its root, so that provides a
| pretty meaningful difference. It also means that controls
| can be handled by different customer teams, and that a
| single change on a shared CMK can render an unbounded
| number of S3 objects inaccessible quickly.
| theteapot wrote:
| Useful. Thanks.
|
| > KMS is a hermetic system with no operator access with
| HSMs at its root, so that provides a pretty meaningful
| difference.
|
| So, if I don't use a CMK (Customer Managed Key), what am
| I getting instead? Assumed (perhaps naively) KMS would
| still be used under the hood.
|
| > a single change on a shared CMK can render an unbounded
| number of S3 objects inaccessible quickly.
|
| I feel like you could achieve that with an IAM policy. I
| guess the key can apply to a wide range of unrelated AWS
| services so act like a ~meta policy. Seems useful.
| paranoidrobot wrote:
| It's protection against mistakes where you've accidentally
| granted more access than was intended.
|
| For instance, if someone does a wildcard resource "acme-
| prod-widget-*" for a new policy, but forgets that widget
| also has (say) a customer-pii bucket, too.
|
| Well, if acme-prod-widget-customer-pii is encrypted with a
| customer-pii specific KMS key, then there's no accidental
| leak of that data.
| davewritescode wrote:
| I understand that but I guess my question is other than
| another layer of access control to prevent mistakes, what am
| I actually getting.
|
| I know AWS has fixed this now, but in years past we paid a
| ton of money in KMS requests from s3 for these types of
| configurations and we asked ourselves what is this really
| buying us?
|
| At the end of the day I have to assume some AWS employees
| have access to some or all keys in KMS.
| colmmacc wrote:
| No AWS employee has access to the keys directly from KMS.
| It's a hermetic system with no operator access like that.
| KMS Keys are released to AWS services for use based on IAM
| permissions and grants, and a time-bounded cryptographic
| pattern we call Forward Access Sessions ... where we end-
| to-end verify that the requesting service has a recent and
| legitimate signed request from the customer.
|
| KMS also has the capability to support an external trust
| store (https://aws.amazon.com/about-aws/whats-
| new/2022/11/aws-kms-e...) ... where AWS holds no key
| material at all.
| aborsy wrote:
| There is huge amount of data in S3 and EC2's. Other cloud
| providers such as GCP hold similarly large amounts of data. The
| governments are no doubt very interested in these gold mines.
|
| I am curious if cloud providers such as AWS provide default
| access to data stored in their data centers to the governments
| (to all data, and by default, in a successor to the PRISM
| program, not on a case by case basis and under subpoenas which is
| a different program)?
|
| If so, KMS doesn't help, and client-side encryption is the only
| way to protect the data.
| specialp wrote:
| There is a danger too to encrypting your data with non built in
| KMS key: The key can be deleted and your data can become
| irrecoverable. KMS has a waiting period of 7-30 days when it is
| deleted but it is up to you to alert yourself of the fact that
| someone deleted it. We have AWS config alerts on this that send
| out via various means.
| advisedwang wrote:
| SSE-S3/SSE-KMS does provide defense against a few things:
|
| * Lost/stolen hard disks from AWS datacenter
|
| * Attacker or rogue AWS employee who has access to low level
| storage but not keys.
|
| The article is correct that this misses many important situations
| but it is still better than nothing.
| ilyt wrote:
| That's kinda like securing a door while not having walls
| tomschlick wrote:
| Welcome to security certifications
| wrldos wrote:
| These are far less likely outcomes than your root account being
| compromised, an IAM fuck up, dumb assume role configuration or
| something using an assumed IAM role being compromised and
| having verbatim access to the bucket.
|
| The feature is there for paperwork reasons.
| klodolph wrote:
| Less likely? Yes. However, these are real, legitimate risks
| that need to be mitigated! This is not just some faff that
| needs to get put on paper.
|
| Think about your risk model: there are a lot of employees at
| your cloud provider, and a lot of high-profile customers.
| There's a defense in depth approach here. Metal detectors,
| security guards, and searches to enter & exit data centers.
| Data encrypted when on-disk. Keep the keys in some KMS
| system. Physically isolate the KMS system, use HSMs, restrict
| permissions to the KMS.
|
| Without these measures, you might end up with a system where
| ordinary customer support representatives, software
| engineers, data center technicians, or system administrators
| / SREs can exfiltrate customer data on a whim. At a large
| cloud provider like Amazon, Google, or Microsoft, that's not
| a small attack surface. That's a lot of people. The idea is
| to make it extremely hard for one of these employees to
| exfiltrate customer data without somehow leaving a record of
| it which ultimately gets the employee fired and prosecuted.
|
| Customers will make mistakes too--screw up their permissions,
| fail to encrypt their data, or whatever. The idea behind
| defense in depth is that any one security measure can
| probably be circumvented, but multiple independent security
| measures are harder to circumvent.
|
| It's really easy to be cynical about security but I have
| worked at cloud providers and the threats are real.
| wrldos wrote:
| I'm fully aware of those risks. The issue for me is a
| matter of realism.
|
| A combination of the default policy being poor on all AWS
| services, the shared responsibility model of AWS delegating
| this risk to clients and the default security posture in
| the industry being terrible makes the real problem of
| solving this like traipsing across a field covered in poo.
|
| An analogy; you have to tick the box to put your front door
| key under the doormat. The burglar knows where the hole in
| your fence is and knows you might keep your keys under the
| door mat. Plus your kids left their keys on the github bus.
|
| Of course the SOC2 audit says you ticked the box and you
| tell everyone your house is a castle.
| oxfordmale wrote:
| If your root account is comprised, it is game over anyway.
| maximilianroos wrote:
| "less likely" risks are still worth protecting against!
|
| I still put my seat-belt on, even though there's an airbag in
| the car.
| TavsiE9s wrote:
| It's there so folks can check their list and go "Yep, it's
| encrypted at rest. Yay us!"
| daitya wrote:
| Author here. Exactly, that's the biggest risk!
| thallium205 wrote:
| SOC2 here we come!
| ejb999 wrote:
| >>The article is correct that this misses many important
| situations but it is still better than nothing.
|
| I am not really sure its better than nothing - what it does do
| imo, is muddy the waters a bit for casual users, and perhaps
| make it seem like the customer doesn't really need to do as
| much securing themselves - because it is now 'encrypted by
| default' - giving a false sense of security - and as others
| have pointed out, really just protects against extremely rare
| edge case problems, i.e. the lost hard disk.
| colmmacc wrote:
| The biggest benefit is compliance. Many customers have to
| satisfy external regulators or internal compliance teams who
| mandate encryption at-rest. Those requirements are typically
| designed for relatively low-security on-premises data-centers
| and basic systems, rather than the kind of secure facilities
| and erasure coding schemes that AWS maintains, but always-on
| encryption makes the checkbox easy.
|
| Another often overlooked benefit of at-rest encryption is that
| it can enable crypto-shredding; delete the object key and the
| object can no longer be recovered. That's faster, more
| efficient, and better for the environment compared to other
| kinds of "secure erasure". Of course if you want robust crypto-
| shredding you do have to take some care about where the keys
| might be cached and for how long, but you can't do it without
| encryption at-rest.
| daitya wrote:
| Author here. Agree, I cover these in the article. Crypto-
| shredding might be the AWS' motivation to enable the feature
| as a 0-click default. But for the customers, they don't get
| much beyond the compliance checkbox, and that too is
| questionable as more and more auditors become aware of cloud
| security controls.
| neilv wrote:
| Earlier in AWS, when I was migrating a complex sensitive system
| from self-hosted servers, there was a requirement not to leave
| data at rest in S3 unencrypted, and there was no suitable S3
| feature for this.
|
| Since I had to write the AWS API from scratch anyway (fringe
| language, and we tried to avoid C memory problems), I also
| implemented transparent encryption at the same time. (Using an
| off-the-shelf vetted implementation of appropriate encryption
| algorithms. And in such a way that, when handling large data, it
| could run on other cores if no accelerator.)
|
| Of course, as the article points out, the EC2 instances (or
| whatever needs to access the data within the objects in S3) by
| definition need access, but at least that access is restricted
| more closely to what actually needs it.
| DethNinja wrote:
| "Users" in this case are supposed to be software engineers. Thus,
| I don't see how Amazon's claim can be misleading. Server side
| encryption is very valuable, especially against breaches on the
| cloud provider.
|
| I would expect any mid to senior level engineer to understand the
| implications and add further security (local encryption before
| upload etc.) measures as neeeded.
| ilyt wrote:
| Breaches in cloud provider so far tend to be entirely software,
| at least for the big ones. Encryption at rest doesn't help
| prevent software bugs
| advisedwang wrote:
| I wonder if the real reason for encrypting is that it allows
| cheaper _deletion_. You can save the IO of deleting an entire
| file and just delete the encryption key. (aka cryptographic
| erasure, see [1])
|
| [1]
| https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S...
| severino wrote:
| This also implies that if the encryption key is that easy to
| dispose of, then it's also easy to somehow lose or screw it
| turning your valuable client data into a random stream of
| bytes, doesn't it?
| advisedwang wrote:
| They can dispose of it in the same way that they dispose of
| the actual core data (ie write over it X times, and ensure
| when disk is EOL that it's shredded). With cryptographic
| erasure you can do exactly the same erasure process on fewer
| bytes and get the same effective results.
| gregw2 wrote:
| My speculation is that they now have nitro encryption on all or
| enough of the servers supporting s3 so it's now no marginal
| cost for them anymore since it's inline ASIC encryption and not
| taking incremental CPU cycles anymore.
| hdjjhhvvhga wrote:
| One problem with utilizing unencrypted drives is that if
| there's a hardware failure, it's more expensive to dispose of
| them. Normally Amazon overwrites the data, degausses the drive
| and then destroys it, but if, say, the electronics is faulty,
| the first step becomes more difficult.
| throwaway320498 wrote:
| Not this. A customer's bytes are likely spread across millions
| of hard drives. At the filesystem layer, the region of the disk
| those bytes occupied still need to be reclaimed.
|
| What you're describing makes sense for a service like EBS where
| a single customer's bytes are limited to small number of disks
| in well-defined locations.
| wiredfool wrote:
| Encryption is easy. Key management is hard.
| [deleted]
| aborsy wrote:
| AWS has access to the AWS-managed SSE-KMS keys, since AWS manages
| those keys on behalf of customers.
|
| I am wondering if AWS has access to data encrypted with the SSE-
| CMK keys as well?
|
| They state that the SSE-CMK keys reside in tamper-proof FIPS-
| validated HSMs and not even Amazon has access to them. But it's a
| bit misleading because they use envelope encryption. Furthermore,
| the encryption and decryption are still server side.
|
| They could have better phrased it that, the only advantage of the
| SSE-CMK keys over SSE-KMS keys is that, with former, the users
| can define key policies (providing another access control layer
| similar to IAM) and monitor usage.
| daitya wrote:
| Author here. Technically, they can. HOWEVER if this threat is
| part of a customer's risk profile, i.e., they cannot risk even
| a possibility of AWS having access to their data, then use
| client side encryption.
___________________________________________________________________
(page generated 2023-01-16 23:01 UTC)