[HN Gopher] Why the hell is your Kubernetes API public?
___________________________________________________________________
Why the hell is your Kubernetes API public?
Author : todsacerdoti
Score : 28 points
Date : 2024-03-23 16:10 UTC (6 hours ago)
(HTM) web link (leebriggs.co.uk)
(TXT) w3m dump (leebriggs.co.uk)
| belter wrote:
| "...For reasons I can't quite understand, we've sort of
| collectively decided that it's okay to put our Kubernetes control
| planes on the public internet. At the very least, we've sort of
| decided it's okay to give them a public IP address..."
|
| Is that what Google does with Borg? ;-)
|
| "...Many of the developers at Google working on Kubernetes were
| formerly developers on the Borg project. We've incorporated the
| best ideas from Borg in Kubernetes, and have tried to address
| some pain points that users identified with Borg over the
| years..." - https://kubernetes.io/blog/2015/04/borg-predecessor-
| to-kuber...
| jauntywundrkind wrote:
| Kind of great that this is true! Speaks well that kube is
| generally pretty secure out of the box.
|
| Being subject to denial of service is a risk though.
| 1oooqooq wrote:
| the solution they suggest is worse. they think a hipster vpn is
| authentication.
| smw wrote:
| The tailscale k8s operator actually does auth as well as vpn.
| kbolino wrote:
| Software-defined mesh networks are quite a bit more powerful
| than conventional VPNs, even if they often get used for the
| same purposes.
| packetslave wrote:
| What the hell is a "hipster VPN?"
| j-krieger wrote:
| There is a very good paper [1] on attackers using open container
| and kubelet registries from colleagues of my uni advisor. Funnily
| enough, this was supposed to be my thesis' focus but they were
| faster. Happens to the best of us I guess :)
|
| [1]: https://dl.acm.org/doi/abs/10.1145/3607199.3607205
| vbezhenar wrote:
| It is public, so what? Obviously cloud operators consider it
| secure enough to offer that option by default. What are the
| reasons to consider it not secure?
| YZF wrote:
| It's a good idea to minimize the attack surface you expose on
| the public Internet. Basically the default assumption should be
| to consider everything insecure. Cloud vendors by nature take
| ownership of protecting all their public facing APIs and
| they're in the business of offering and securing those. That
| doesn't mean you should open everything to the Internet.
| dangus wrote:
| Not only that but cloud providers put their managed Kubernetes
| behind IAM.
|
| If you've managed to breach that and you've got Amazon or
| Microsoft convinced that you are a privileged account owner, it
| would be trivial to modify the infrastructure to make a private
| API endpoint public or create a bastion host with access.
| 1letterunixname wrote:
| Why people expose their core infrastructure on the internet
| directly and unfiltered always baffles me. I guess they must be
| SWEs who have never touched a server.
| empath-nirvana wrote:
| There is no such thing as a "private" network that is magically
| more secure than the internet, and if you don't trust it on the
| internet, you shouldn't trust it on your private network. If you
| have anything of value, there are going to be bad actors poking
| around in your private network eventually.
| dangus wrote:
| This article is working backwards to solving a problem that most
| development teams shouldn't have. I will first walk along with
| the article and try out their backwards solution before getting
| to my point:
|
| If you're using Kubernetes managed by a cloud provider, e.g.,
| Amazon EKS, authentication is already handled through IAM. This
| is a service that's designed to be exposed to Internet traffic
| and that AWS is essentially managing for you. If you're able to
| get past AWS IAM then the fact that your cluster API is public is
| moot because gaining access to an AWS account that has
| significant kubectl privileges is likely going to also expose a
| number of other privileges to make changes in AWS.
|
| Okay, perhaps that's not enough for your liking. We want more
| defense in depth. AWS already tells us how to use a private API
| endpoint via their documentation:
| https://docs.aws.amazon.com/eks/latest/userguide/cluster-end...
|
| But of course, those suggestions bring us here to this article:
| Amazon suggests using a bastion or a cloud IDE, but we might find
| that too clunky.
|
| This is where the article offers "a better alternative: Tailscale
| is basically one of those fancy easy-to-use VPN replacements that
| promises to get rid of VPNs and bastion hosts in favor of
| protocol-aware proxying. And, that's fine, but it's not cheap
| compared to a traditional open source VPN if you're using it for
| a company: https://tailscale.com/pricing
|
| Besides not being very cost-effective, we're still working
| backwards to the actual issue here. Who needs kubectl access and
| why are they using it so much in production that a bastion or VPN
| is annoying? Why don't we just make it so that developers don't
| have to use kubectl in production anymore? Isn't their code being
| deployed with an automated pipeline?
| xg15 wrote:
| I think something doesn't quite add up in OP's claim. Having full
| access to the control plane effectively let's you do anything you
| want with the cluster. If there were really millions of
| completely unsecured Kubernetes instances lying around, we should
| see all kinds of worms and botnets already taking advantage of
| them - and we should also see lots of attempted probes/attacks to
| infect new instances as soon as they come online (dark forest,
| etc). Yet I haven't heard about anything like this so far. This
| leads me to believe that the "public" instances actually _are_
| protected in some way, just not by a private network.
|
| Not a Kubernetes expert, but doesn't the Kubeconfig contain some
| sort of secret that the client needs for authentication? You
| might be able to get some basic health check without that, but
| I'd be very surprised if you could do much more without
| authentication.
|
| At least if that's how it works, this would be an example of
| "zero-trust" networking, where you just assume anything can be
| reached from the internet anyway and secure your services based
| on that premise. If done well, it shouldn't matter whether or not
| your services are actually exposed on the internet or not.
|
| Of course in the interest of defense-in-depth, I'd still try to
| make sure my services are _not_ actually exposed - bugs happen
| all the time and additional security layer can 't do harm.
| maxcoder4 wrote:
| Of course that's how it works, there are various authentication
| methods that must be completed [2] to do anything. The author's
| claim is that similarly like you shouldn't expose RDP to the
| public internet, you shouldn't expose your k8s API.
|
| In practice the difference is that while RDP (or database or
| WordPress) credentials are vulnerable to bruteforce, in almost
| all cases k8s clusters are secured by either mtls or jwt tokens
| - both utterly non bruteforcable. You're not completely right
| though, unsecured k8s clusters do happen[1]. And there is some
| defense in depth component, as you've mentioned.
|
| And yeah, we're one RCE away from the cloud melting disaster.
| But then, the same is true for nginx or openssh.
|
| [1] http://redhuntlabs.com/blog/unsecured-kubernetes-clusters-
| ex...
|
| [2] Fun but meaningless fact: kube-api defaults for both
| authentication and authorization are to allow anonymous users
| and allow everything to everyone (AlwaysAllow)[3]. This is
| meaningless because every k8s distribution in existence changes
| authorization default with appropriate flags.
|
| [3] https://kubernetes.io/docs/reference/access-authn-
| authz/kube...
| ufmace wrote:
| Thought about it for a few seconds, and yeah, I'm sticking with
| meh on this one.
|
| I do have a managed K8s cluster running on one of the major
| hosting services. It seems I can indeed hit the /healthz endpoint
| without auth. But I don't see any reason to care. First off, the
| control plane URL is a UUID tacked onto the hosting service's
| DNS. Good luck figuring out what it is starting from the URLs of
| the sites hosted on it. Even if you're my ISP or somehow
| otherwise in a position to observe traffic to the CP, good luck
| correlating it to what's actually hosted there. Yeah you can see
| that the cluster is healthy, but not much of anything else
| without auth AFAIK.
|
| Second, there's a tension between "network restrictions don't
| help that much, if your control system is actually secure, it
| should be secure enough to expose to the clearnet" and "defense
| in depth always, another layer never hurts anything". I'm leaning
| towards the first. You can always come up with an excuse to add
| another layer, but how much does it actually protect you from any
| reasonably plausible attacker? Lots of stuff far more sensitive
| and valuable than anything I've got has control interfaces
| exposed to the net like that, relying solely on the security of
| OpenSSH, K8s TLS cert system, etc for their protection. If
| somebody figured out how to bypass that, and chose to use the
| capability against my stuff, who's to say that any extra layers
| of security I could dream up would actually slow them down? I
| think it's not likely, and those extra layers are much more
| likely to trip me up and let me accidentally lock myself out than
| to stop any plausible attacker.
___________________________________________________________________
(page generated 2024-03-23 23:01 UTC)