[HN Gopher] Encrypting Postgres Data at Rest in Kubernetes
___________________________________________________________________
Encrypting Postgres Data at Rest in Kubernetes
Author : plaur782
Score : 87 points
Date : 2021-10-31 15:08 UTC (7 hours ago)
(HTM) web link (blog.crunchydata.com)
(TXT) w3m dump (blog.crunchydata.com)
| CamouflagedKiwi wrote:
| Was hoping for something a little more profound than "use an
| encrypted storageclass for your volumes".
| Fiahil wrote:
| Yes, what they explained is entirely based on encrypted EBS
| volume. Nothing related with Kubernetes apart from the
| \encrypted: true` in the StorageClass.
| speedgoose wrote:
| These automatically encrypted disks with managed keys in the
| clouds are nice to check the "encrypted at rest" checkbox for
| security audits but I think that they add little security. In
| most scenarios I can think of, both the data and the keys will be
| accessible to the attacker. And the ones where the attacker would
| have access to only the encrypted data seems very unlikely, like
| physical access to the data center with the knowledge of where is
| physically stored the data. But I would be gladly proven wrong.
| ATsch wrote:
| I agree in the general case but I think there are uses for it
| that only go slightly beyond the fully automatic version. For
| example selectively encrypting less frequently needed sensitive
| fields can give you more granular access control and logging,
| especially if you have to check in with another service to
| perform the decryption instead of having the key available
| locally.
| exitheone wrote:
| But that is exactly the point. You want to ensure that disks
| later sold by your cloud provider do not contain your data in
| plain. That's the whole point with encryption at rest. It's not
| supposed to protect you against an attacker with root access to
| your server.
| [deleted]
| [deleted]
| jeffbee wrote:
| But encryption at rest _does_ protect you from insider or
| attacker access with full privileges on the servers where the
| data is stored. The data would necessarily be visible to a
| root user on the system where the data is being decrypted by
| an application, but that 's not necessarily the same node.
| yashap wrote:
| A good, fairly recent example from the city I live in, of
| hard drives being sold that were (accidentally) absolutely
| chock full of unencrypted customer PII: https://www.google.co
| m/amp/s/globalnews.ca/news/4476625/ncix...
| itisit wrote:
| The big cloud providers destroy old drives.
| CamouflagedKiwi wrote:
| Agreed, although it does mean that if you release the volume
| you aren't relying on the cloud provider to wipe it before
| reuse (AFAIK they generally do that, AWS definitely does but
| it's possible other providers might not, especially if you're
| on some more custom OpenShift style setup).
| jeffbee wrote:
| GCE persistent disks are always encrypted anyway, with a
| customer-provided key if the customer chooses, or an
| automatic ephemeral key otherwise. I think the only storage
| option in GCE that doesn't feature customer encryption keys
| is locally-attached SSD, where you'd need to layer your own
| crypto on top. Definitely don't rely on cloud providers to
| wipe your data.
| zorked wrote:
| Not everything is physical access though. Suppose someone comes
| up with a way to force S3 to read random storage blocks, or to
| make a virtualized storage device read past its boundaries in
| the underlying storage, or to intercept another VM's ring
| buffer in the hypervisor. It's an entirely different scenario
| to read plaintext vs read something encrypted.
|
| Even if all it protects is against the scenario of the cloud
| provider forgetting to wipe a disk, that's worth it.
| speedgoose wrote:
| That's a good point. I prefer to encrypt my files before they
| go to S3, it's easy to do in applications or using Minio as a
| gateway.
| dividedbyzero wrote:
| Minio can do that transparently?
| throwaway984393 wrote:
| > And the ones where the attacker would have access to only the
| encrypted data seems very unlikely, like physical access to the
| data center
|
| This is the primary purpose of encryption at rest. It is not
| unlikely. Have you ever worked in an average datacenter? Some
| of them have very little physical security and don't monitor
| employees' access to cages. And if you have a cage on the same
| floor as another customer, all it takes is a fake name badge
| and a clipboard to walk up to someone with their cage open and
| say you're doing a routine inspection. Walk into the cage, pop
| out a drive, put it under your clipboard, walk away. Ask
| physical security pentesters how "difficult" it is to steal
| from a datacenter. And let's not even get into dumpster diving,
| where clients regularly toss entire servers without erasing
| disks into dumpsters.
|
| Like someone else said, it also is a good practice in multi-
| tenant configurations where a misconfigured storage or compute
| backplane could expose data to the wrong tenant.
| hsbauauvhabzb wrote:
| Any case studies related to AWS or azure?
| latch wrote:
| In most cases, you're trusting other organizations to dispose
| of the disk, ensure they can be securely re-used (1) and that
| their employees will generally behave correctly and ethically.
|
| But still, in general, I agree. I fill in these security
| questionnaires maybe once every couple months, and I'm starting
| to see a clear shift from encryption of data-at-rest to
| encryption of data-in-use. Which mostly just feel like an even
| more tedious form of security theater.
|
| (1) https://news.ycombinator.com/item?id=6983097
| sylr wrote:
| Am I the only one enforcing a strict no database in kubernetes
| policy ?
| ahachete wrote:
| I wrote recently a piece [0] on why I believe you _should_ run
| your databases in Kubernetes. FYI.
|
| [0]: https://thenewstack.io/kubernetes-will-revolutionize-
| enterpr...
| SuspiciousSwan wrote:
| No, other people also need consistent performance and a simpler
| system that works in practice.
| Sebb767 wrote:
| Honest question: Why is that? What is wrong with your DB
| running in Kubernetes and what do you suggest as alternative?
| sylr wrote:
| I consider anything in kubernetes disposable. I want to be
| able to lift and shift anything in a new cluster without any
| migration in a matter of minutes.
|
| Anything "stateful" like a database breaks this paradigm.
|
| I have nothing against databases used as cache that can be
| "re-filled" upon re-creation, but I believe anything holding
| business critical data shall be held outside of a kubernetes
| cluster. Why, because being one command away deleting your
| StatefulSet, Helm Release ... etc scares the shit out of me.
|
| You can of course minimize the risk with correct RBAC, ensure
| proper backup/restore migrations but that require lots of
| staff and efforts I can't spare.
|
| So until I can be reassured that I have all the tooling that
| can recover rapidly any catastrophic failure/mishap, and that
| all this tooling is tested monthly, I enforce using managed
| databases services.
| dilyevsky wrote:
| There are databases where individual nodes are disposable.
| A lot of managed dbs aren't exactly zero ops and hard to
| service without downtime too
| kobalsky wrote:
| if you set a retain policy on persistentvolumes it will
| prevent your volumes from being deleted even when you
| delete the owning objets. cloud providers will keep the
| virtual drive on those cases even if you delete everything
|
| regardless, it's the wrong thing to fear. this is at the
| level of logging in every user as root on your servers and
| databases because proper user management would require
| extra staff and efforts you can't spare.
| jandrewrogers wrote:
| Performance is a common reason. There are architectural
| incompatibilities between high-performance database engines
| and the way Kubernetes is designed to work. Kubernetes does
| not respect the traditional contract between the OS and the
| database engine, transparently interfering with that
| interface in adversarial ways that degrade important
| optimizations. Ironically, the performance loss can be
| substantially worse than running databases in a properly
| configured virtual machine -- virtual machines have some
| overhead but they otherwise don't interfere with the proper
| functioning of this software. Kubernetes wasn't designed to
| efficiently support the syscall and hardware utilization
| patterns of I/O intensive applications and this is evident
| throughout, even requiring non-standard hacks to set things
| up in Kubernetes that really shouldn't be necessary.
|
| Deploying databases in Kubernetes is fine for many
| applications, I've done both. Not every application that uses
| a database is data intensive.
| mindwok wrote:
| Do you have any specific examples of Kubernetes interfering
| with this contract? I've not heard of this kind of
| behaviour before.
| jeffbee wrote:
| I have a strict policy of not running my own database at all.
| Does that fit within your policy, or does it violate your
| policy if my database-as-a-service vendor uses a container
| orchestration platform?
| sylr wrote:
| No it does not. I don't run database in kubernetes because I
| don't trust myself to be able to recover from disasters.
|
| How database-as-a-service vendor run their services is none
| of my business as long as they deliver the performances I
| need and working backup/recovery procedures.
| rozularen wrote:
| why is that?
| Spivak wrote:
| Because resource intensive stateful workloads with persistent
| data is basically k8s's Achilles heel. It's not that k8s
| can't handle it, it's just that you get pretty much no
| benefits from k8s and so the extra configuration overhead is
| rarely worth it compared to running an external db cluster.
| plaur782 wrote:
| No. You are not alone. There are still a number of
| organizations that are cautious about deploying databases in
| Kubernetes. That said, various third party surveys as well as
| anecdotal evidence of what we are seeing at Crunchy Data
| suggests that deploying databases on Kubernetes is increasingly
| common. The degree to which it makes sense often depends on
| whether or not the organization is standardizing around
| Kubernetes for their deployment model more generally.
| zinclozenge wrote:
| You're not the only one, obviously. But that stance is not
| really relevant anymore. Just about every database as a service
| company will run their databases in kubernetes now.
| handrous wrote:
| Tangentially related: what's state-of-the-art for data protection
| & access control for small organizations? One runs into the
| "someone's gotta be trusted with the master keys" problem there
| so early & often that all the "big" solutions feel silly. Do
| small shops just farm this out via SaaS and hope their provider's
| doing the right thing?
|
| (the answer back in the day, and perhaps still, was just "they
| don't really worry about it at all, and hope nothing goes wrong")
___________________________________________________________________
(page generated 2021-10-31 23:00 UTC)