Migrating from multiple docker-compose VPS to Kubernetes reddit / selfhosted ---------------------------------------------------------------------- hardyrekshin · 2023-10-25 22:58 Almost 3 years ago, I paid for a few VPSs on which I host a variety of services. (Vaultwarden, gitea, drone, meshcentral, metabase, gptresearcher, etc) Interspersed among the VPSs are a series of data processing containers to handle crypto data. With the contract coming up for renewal, I'm exploring how to separate the hardware from the software so I'd only need to deploy the container to a pool of servers, and the infrastructure decides on which server to run the container, correctly route incoming requests, and update cloudflare dns for containers which are meant to be oublicly facing. I went through the kubernetes the hard way tutorial and have a cursory understanding of kubernetes but with some substantial gaps which I couldn't Google away. For the replacement platform, I'm thinking to: \- Combine multiple VPSs as a baseline cluster to run internet-facing loads \- Use some home servers for backend/non-internet facing processes and make the data available on the Internet facing hosts. \- Add the ability to dynamically add more VPSs or preemptible instances from GCP/AWS I'm still stuck on the first part. Standing up a kubernetes cluster using multiple VPS with different public IPV4 addresses. Googling around heavily suggests this is not a common use case. Or at least I'm not using the correct terms. Is there a better solution for me to pursue? ---------------------------------------------------------------------- rumblpak · 2023-10-25 23:08 Personally, id look into serverless installs if you want to stay in the cloud based on your requirements. You dont necessarily need to run your own cluster, it might be significantly cheaper to go that route. That said, if you want to create your own cluster, do it local in vms to get an understanding first, then scale out your prod. ---------------------------------------------------------------------- Vynro · 2023-10-25 23:14 Realistically for the cloud based stuff I'd look into running with a "managed kubernetes" provider, which would take a lot of the more intricate details of it out of your hands, and let them manage them for you - of course, this is self hosted so I 100% understand the want for doing it yourself. For my homelab I use K3s to run a cluster, with longhorn for storage, metallb to expose IPs of services and kube vip. My cluster at home doesn't really get public facing access so I don't worry too much about the security aspect per say ( I still have security, just didn't have to focus too much about 4 VM's having appropriate permissions to talk to each other) but you'd like have a bit more on your hands if you rolled your own in the cloud vs a managed cluster. Just my 2 cents ---------------------------------------------------------------------- hardyrekshin · 2023-10-25 23:55 Serverless works best when utilization is low. The data processing containers are always on streaming in data so likely a no-go for me. Also uncertain to what extent something like vaultwarden could run serverless. And noted on standing up a local cluster. That seems to be what most others have been doing in the self- hosted/homelab community. ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 00:15 Notes on the managed kubernetes option. Main blocker for me is cost. Pricing out the same specs as my current fleet of VPS comes out to 3x the cost. Granted, there's a very good chance my VPS provider is oversubscribing the servers. However, it's unlikely the other tenants on the server are consistently at 85-90% utilization so at least in theory I come out ahead. Your setup is what I'm finding more tutorials explaining so I guess I'll just stand up something like that on my esxi server. ---------------------------------------------------------------------- Vynro · 2023-10-26 00:27 If you'd like some good tutorials for K3s, I'd check out TechnoTim on YouTube, learned all my kubernetes knowledge from him. He even has a GitHub repo that will standup a high availability K3s cluster within minutes on specified nodes. Yeah cloud compute can get pricey for sure unfortunately. I do expose some of my clusters apps with Traefik v2, and using that reverse proxy I point to my MetalLB IP addresses from my services - I'll point out that I run my Traefik proxy on a separate VM that runs all the containers I don't want to maintain within kubernetes in a docker compose stack, and not within the cluster - but when you mix Flux CD with K3s it's been the easiest management of kubernetes I've ever had, and if something catastrophic happens in my cluster I can be back up and running relatively soon since it's all in GitHub. (Used to take me a weekend to fix or reprovision a cluster, but now with Tim's scripts and Flux, I can have a fresh cluster running in about an hour with all my backups etc. ). My storage takes the longest to get back up and running from backups for the cluster from my NAS but longhorn has taken a lot of the pain out of it. ---------------------------------------------------------------------- rumblpak · 2023-10-26 00:28 Serverless is just another word for container, you just dont control the kubernetes side of things. I have several serverless instances serving production workloads at high utilization. While I haven’t run vault warden in a serverless manner, i run it in a container in my kubernetes cluster at home right now. ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 00:32 When I hear serverless I think of Google cloud functions or AWS lambda. Were you referring to something else? ---------------------------------------------------------------------- rumblpak · 2023-10-26 00:35 I mostly use google so yeah cloud functions is the primary solution but google does have a couple more serverless offerings though I don’t know how many of them are available outside of enterprise. ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 00:35 I tried to follow the techno Tim tutorial but it appears the solution powered by rancher does not work on ESXi hosts. For some reason I haven't been able to diagnose, after creating the node and running the command rancher provides, the node cannot detect or connect to rancher. But good to know what's possible. ---------------------------------------------------------------------- Vynro · 2023-10-26 00:38 I would look into his newer stuff regarding his ansible playbook for provisioning a cluster , and then I use k8s lens to monitor / get a"GUI" for my cluster personally. I find it way better than rancher, and I have one less cluster to manage (when I was deploying rancher into it's own highly available cluster ) ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 00:43 I just priced out an e2 instance 8gb ram 4 threads. Comes out to 51/mo 30 invocations of a cloud function with 8gb ram and 86400000ms runtime comes out to 173/mo I must be misunderstanding something. How long are your cloud functions scripts running? ---------------------------------------------------------------------- clintkev251 · 2023-10-26 00:43 I would challenge that. Often serverless architectures use containers, but the main qualification for something being “serverless” is that you don’t manage the underlying infrastructure. When I think of serverless I think of AWS Lambda (can utilize containers, but doesn’t have to), DynamoDB (nothing to do with containers) RDS Aurora (nothing to do with containers) Fargate (this one is container centric) ---------------------------------------------------------------------- rumblpak · 2023-10-26 00:48 Can’t speak to work specifics but we use a variety of cloud functions, cloud run, and compute. It’s a balance to get the best pricing for your use case. Functions are more for short-lived applications, cloud run is more for microservices. ---------------------------------------------------------------------- rumblpak · 2023-10-26 00:51 Fair, but I’m not trying to hit every definition here, more to hit the ELI5 crowd. There’s functionality no difference between an application scratch container and above examples you’ve listed. ---------------------------------------------------------------------- clintkev251 · 2023-10-26 01:03 I would say the ELI5 is just “serverless is when someone manages the infrastructure for you”. Saying containers are serverless is confusing because containers can be run in a serverless fashion, but they can also be run in a serverfull fashion ---------------------------------------------------------------------- AlfredoOf98 · 2023-10-26 04:09 Just a little note regarding complicated (or hard-to-express) search queries: Try Bing's AI Chat. You can explain your issue and it's generally very good at finding the proper search terms. It'll do the searching for you as well, but some times it would be enough to just know the terms that you need to use. ---------------------------------------------------------------------- seidler2547 · 2023-10-26 04:28 For 173/mo you can get almost 10 VPS with 8 cores and 30GB RAM and 800GB SSD each at Contabo. ---------------------------------------------------------------------- vixfew · 2023-10-26 05:39 IMO, k8s isn't worth it for hobbyists users unless you already have the knowledge i.e. for a job. It's too complicated. There's [Nomad](https://www.nomadproject.io/), which gets the same job done and is much easier to manage. 1) if your vps are from the same provider - look for private network option. If not - install a vpn (wireguard). Might want to install a vpn anyway, just in case providers virtual network isn't really private. Nomad: add IPs in the list in config, with proper ports open it just works. 2) The hard way - multi-cluster. The easy way - use vpn and same subnet. 3) Should be possible, as I mentioned I used Nomad for my orchestration needs, with N it's as simple as deploying a few binaries, config and starting it as system service. You might want to look into Ansible. Great for automation, unless you want to go full managed route, which isn't fun (imo) nor cheap If you want to try setting up Nomad cluster, feel free to dm, I've got some Ansible scripts that can convert a bunch of vps into a cluster ---------------------------------------------------------------------- WiseCookie69 · 2023-10-26 07:09 Look at K3s. Since a while it has built-in support for Tailscale (can also use Headscale). Alternatively, it doesn't really matter how or where your nodes are located, if you add a VPN to allow them to talk to each other. Your main issue would be storage. But that's easily fixed with a topology aware CSI and then keeping your stateful workloads either wherever they got their volumes provisioned, or forcing them to be provisioned on your home servers. ---------------------------------------------------------------------- Qxt78 · 2023-10-26 07:37 That is what rook-ceph or longhorn is for. Longhorn is good for beginners ---------------------------------------------------------------------- Qxt78 · 2023-10-26 07:41 If you want a production grade cluster without the hassle that works on any vm. Looking into native rke(docker based) or better rke2 (containerd based). Using it for years without rancher. Just works. Also easy to setup following the "getting started" tutorial. Note on rke2. Comes with some components already hardened which is better for security reasons. Overall a good enterprise ready solution (rke2) ---------------------------------------------------------------------- adamshand · 2023-10-26 09:26 As another option why not docker swarm? I think it will do what you need and is much simpler than k8s. I’ve been using caprover and really liking how simple and reliable it’s been. In the process of going from a single node to a swarm, still researching / experimenting but liking it so far! ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 11:53 I read into Docker Swarm, the community surrounding it is much smaller and there appear to be far fewer tutorials compared with k8s. In addition to hosting the usual self-hosted applications that are often mentioned, I also do a good amount of processing of crypto and other financial data. Being able to more dynamically add/remove servers and have the containers automatically assigned / routed should at least in theory save me a lot of time administering the infrastructure. ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 11:57 Good point about Tailscale. It hadn't occurred to me. How does DNS work for a Tailscale-networked cluster of servers? My understanding is at least one of the nodes would need to be designated as the ingress. I could potentially also have all the master nodes hold the ingress, but then I believe that means I'd need to use round robin DNS in cloudflare to ensure the domains are always pointing at the cluster. Storage might be a problem, but being more cloud aware potentially means I can run DuckDB against minio to scan S3 objects when doing data-intensive tasks. ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 12:08 You're right about the job angle. A quick look at linkedin and I'm struggling to find jobs which require nomad. So while it is easier to manage, the skills may not have as much value as k8s. Good point about using a vpn to connect everything. Probably going to just use tailscale as another commenter suggested. And thank you for pointing out Ansible. I don't expect to need to bring up and tear down servers that often, but at least after a quick google, it looks like it also can automate bringing containers up and down too--which should be very useful. ---------------------------------------------------------------------- AnonymusChief · 2023-10-26 12:22 Hello OP. The major advantage of running containers in a cluster is replication. So it would be expected that if you have a 3-node Kubernetes cluster, you would replicate your deployments (which would create a replica of your deployment/container in each of your nodes). Although you may need to learn kubectl commands, you will be smooth running in no time since you do not need to configure the servers, no manually setup a load balancer. The service provider that I have in mind while reading your post is Digital Ocean, or GCP. They are equally fluid to use. Compare the pricing for both service offerings. With DO, the control node is free and is managed. ---------------------------------------------------------------------- ionfury · 2023-10-26 15:03 Anything cloud kubernetes at a small scale and still worth doing is going to be much more expensive than doing VPS. You're either paying for the overhead of the managed control plane or paying for extra VPS to run your own control plane. You _can_ try and do a cluster on spot instances to save $$$ but that is a pain in the ass and most selfhosted services can't run HA. You _could_ run combined worker/control nodes but then you're always at risk of shooting your foot off if you actually start kicking the tires on your compute. I _guess_ you could run a two node worker/control cluster but then what's really the point? I'm 100% in on k8s at home, but I just don't think it makes sense from an /r/selfhosted perspective in the cloud if you don't want to pay for the extra overhead. ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 15:09 I'm of the view self-hosted is not exclusive to on-prem. I suppose the privacy-focused self-hosted camp will materially disagree. And that being said, the more I've been googling, the more I'm realizing cloud k8s is not yet practical at my scale. So I'm currently exploring a pseudo hybrid infrastructure where I have a single VPS to hold critical internet facing workloads where uptime matters as a single docker-compose server and then combine cloudflare tunnels and an on prem k8s cluster for all other I-can-afford-some- downtime. ---------------------------------------------------------------------- ionfury · 2023-10-26 16:22 IMO the key sellers of k8s is scaling and availability. You can't really scale metal k8s (unless you're actually scaling your physical nodes by load, if so kudos to you I'll shut up and take notes), and if you're not leaning on the availability, then what are you trying to achieve? It sounds like you have actual cost and uptime concerns and aren't just doing this for lark. I've selfhosted plenty in the cloud but I went down this same road a few years ago and decided against hybrid/cloud k8s for the above reasons. If you're just coming out of a k8s 101 tutorial I'm gonna go out on a limb and predict you're gonna have a lot of downtime until you get your sea legs either way. I'd suggest not making any big plans or changes yet, and self hosting a cluster for a few months. At that point you should understand more of the pitfalls and issues you will encounter, and you can migrate your services over ad-hoc as you feel comfortable with where you're at. You should also check out the [kubernetes @ home](https://discord.com/invite/k8s-at-home) discord! ---------------------------------------------------------------------- adamshand · 2023-10-26 17:28 The community is much smaller. I've found Swarm pretty straight forward, can get in up and running very quickly and only have to tweak existing compose files. Unless I'm misunderstanding what you are wanting, you can also dynamically add/remove servers and there's no need to do any special routing, Swarm does all that automatically. Anyway, not trying to convince you! :-) ---------------------------------------------------------------------- natermer · 2023-10-26 19:07 The problem with hosting kubernetes on VPSes is that exposing the Kubernetes API to the public is pretty sketchy. I know a lot of people do it, but I don't like the idea. I also like having multiple smaller Kubernetes clusters then a single big one. Easier to manage and breakage is more isolated. You can incorporate external services into kubernetes pretty easily using kubes services and endpoints. I suggest using K3s as it is very lightweight, easy to deploy, and is k8s compliant. There are default set of services k3s deploys by default and are designed for more 'IOT' applications. Things like service-lb. These things can be disabled if you want during install time. For managing it I like to ArgoCD on a 'administrative' kubes cluster local to you. It has no problem connecting to multiple clusters and has a nice declarative yaml files for configuring things that work well with a git-based workflow. The web UI is nice and is used widely. ---------------------------------------------------------------------- hardyrekshin · 2023-10-26 19:20 What's your take on inter-cluster communication? E.g. I could hypothetically have 3 clusters: 1. Administrative 2. Web-Facing 3. Backend Potential use cases: - Backend might produce updated parquet files which needs to be transferred and made accessible on metabase in the Web-Facing cluster. - Web-Facing might need to send batched inputs (from webhooks for example) to Backend for processing. ---------------------------------------------------------------------- ionfury · 2023-10-26 21:20 Service mesh is what you're looking for here. Istio is a front runner in this space. Without knowing more of your use case though multi cluster is really adding a lot of complexity here and I'm not sure what you're getting over, say, namespaces and network policy. ---------------------------------------------------------------------- s_busso · 2023-10-26 22:44 The cheapest and most straightforward solution would be to use Docker Swarm. You will benefit from the cluster without the cost or complexity of Kubernetes. Much less to learn, reuse the docker compose. Here is my go-to stack for similar needs: Swarm, Portainer, Caddy. ---------------------------------------------------------------------- s_busso · 2023-10-26 22:51 Swarm community may be smaller, but there is also less to learn or do specifically on Swarm. This is because Swarm uses most of the same components as Docker, such aumes, and networks. ---------------------------------------------------------------------- hardyrekshin · 2023-10-28 11:48 One feature that's very appealing to me is being able to deploy an image, and not having to worry about the server the image is deployed to being overloaded. Metabase is a good example. When I'm not using it, background data processing pods can run. When I'm actively running queries, the background pods get deprioritized. After developing a couple manifests from scratch there are clearly many more things I need to learn. As the VPS renewal date is coming up, I'm for now migrating low priority services into an on prem server and connecting them with cloudflare tunnel. What I need to figure out is how to transfer the output of the data back to the cloud VPS.