Homelab Apps on Kubernetes Cluster reddit / selfhosted ---------------------------------------------------------------------- fabio_teixei · 2023-11-20 19:36 So, i got my home lab up and running just fine with the usual suspect apps on Docker: * Radarr * Sonarr * Bazarr * QBittorrent * NZBGet * Jacket * Tautulli * Overseerr * SearxNG * PiHole * Bitwarden * etc. Now i have deployed an Kubernetes cluster. I'm already running Portainer, Rancher and Traefik on this cluster (MetalLB and Cert- Manager as well). I want to move most if not all my Docker containers to Kubernetes. I know there is many ways of doing that, but using helm will be the easiest way. Is any of you doing that? Using Kubernetes for your containers? How are you deploying the containers? I think that using docker-compose much easier than Kubernetes even with helm and if it (kubernetes) was not de defacto standard for conainer orchestration on the enterprise world (i'm IT professional and learning Kubernetes is kinda a necessity), i would prefeer to use docker-compose with docker-swarm-mode. ---------------------------------------------------------------------- Bagelsarenakeddonuts · 2023-11-20 20:33 Truenas scale runs on kubernetes if you want an easy mode ---------------------------------------------------------------------- fabio_teixei · 2023-11-20 21:16 It's already deployed my Kubernetes, It's not the Kubernetes base the issue, is the apps. Most, if not all, guides on how to deploy those apps thei mention only docker or docker-compose methods. ---------------------------------------------------------------------- itzeric02 · 2023-11-20 21:34 I think you are looking for this https://artifacthub.io/ ---------------------------------------------------------------------- Bagelsarenakeddonuts · 2023-11-20 22:01 The containers themselves are containers, they will work in either. Your env variables and settings etc will be the same. The differences between kubernetes and docker are entirely independent of the applications themselves. Its the networking, configuration and nuances of kubernetes itself that will differ from docker compose, and nobody can tell you explicitly how to do that. ---------------------------------------------------------------------- deadlock_ie · 2023-11-21 00:37 I’ve just moved my docker stuff over to kubernetes so I’m probably a little further along than you are. I’ve been creating deployments from scratch - they’re really not much more complicated than docker-compose configs. Once I cracked translating the first docker-compose.yml to its k8s equivalent it was reasonably easy to do the rest. There’s a learning curve but it’s not too steep (yet). I haven’t looked at Helm much yet - it seems to be more powerful than I need at the moment. ---------------------------------------------------------------------- fabio_teixei · 2023-11-21 01:52 Do you have any kind of exemple to share? With the docker-compose and the deployment yaml side by side to comparison? ---------------------------------------------------------------------- fabio_teixei · 2023-11-21 01:54 I didn't mean on that sense. I mean that I can deploy on docker using an docker-compose file, I don't know how to do the same kind of thing with Kubernetes. ---------------------------------------------------------------------- suphomies · 2023-11-21 04:31 I'm using terraform to manage my kubernetes applications. If there's a helm chart available I'll pick that. Else I'll use the standard kubernetes objects via terraform. But there's more overhead this way. But i find it easy to move os when i had to. ---------------------------------------------------------------------- WiseCookie69 · 2023-11-21 12:48 ArgoCD. If there's something that doesn't come with a Helm chart, i just wrap it into bjw-s' common chart (https://github.com/bjw-s/helm- charts/tree/main/charts) and call it a day. ---------------------------------------------------------------------- deadlock_ie · 2023-11-21 16:42 Yeah, sure - here's a fairly simple one: Plex. I'm using k3s instead of full-blown k8s but it doesn't make a huge difference. I'm going to keep using k8s as an abbreviation for simplicity's sake. ​ Original docker-compose.yml - [https://pastebin.com/7hiLvSmT](https://pastebin.com/7hiLvSmT) Kubernetes: [https://pastebin.com/Cjkn6FH7](https://pastebin.com/Cjkn6FH7) ​ The Kubernetes one is obviously a little more complex but not hugely so. There's also definitely room for improvement and I'm not too bothered with resilience since at the moment I have 2x NUCs - the one that I'm migrating from docker off, and the one that I'm migrating to Kubernetes on. I'll circle back once I've moved everything, wipe the old NUC, and add it as a second node eventually. Note that the Kubernetes YAML has multiple definitions - from the top you have: * app * persistent volume claim * service Each of these could (arguably should!) be in a separate file. Note that I'm using 'usenet' as my namespace. The app section should be pretty straightforward - the number of replicas (pods) is 1, the app name is plex, environment is production etc. Some of that stuff only matters when you start getting into multi-node clusters, where you decide what gets deployed to a given node based on the labels that are applied etc. Further down in the app section we have the template, and within that we have the container definition. This is pretty similar to the docker-compose.yaml version and should be mostly self-explanatory. You should be able to figure the volume stuff out yourself - the mount points/paths are defined within the container spec, then the actual volumes themselves are in the next section. In the next section we have persistent volume claim (pvc for short) that's referenced in the app's volume list (plex-pvc), which is using longhorn as a backend ([longhorn.io](https://longhorn.io), you'll have to deploy that in your k8s cluster before you can use it). This references a persistent volume (pv) that doesn't exist. When I apply this yaml for the first time - using `kubectl -n usenet apply -f plex.yaml` - a pv will be automatically created with the reclaim policy set to delete. This means that if the pvc is deleted, the pv will also be deleted shortly thereafter. I'm editing the auto-created pv (`kubectl edit pv xxxxxxxx`) and setting the policy to retain; this is a silly way to do it and I should really be defining the pv myself, with the reclaimPolicy already set to retain (unless you don't care but since this is where your Plex config/database/cache etc. is going to live, you probably do in this case!). Finally we have the service definition, which maps port 32400 on the host (192.168.1.254) to port 32400 on the pod(s, if we had replicas set to >1). I'm pretty sure that the containerPort parameter in the container spec /also/ does this, but setting up a service allows me to explicitly set the external IP and also creates a DNS entry for plex in my usenet namespace, meaning that other pods/apps in that namespace can just connect to 'plex' to force a library refresh (for example). Pods/apps in other namespaces can connect to plex.usenet.svc.cluster.local if needed. If we had >1 replicas defined, the service would round-robin between the plex pods in the usenet namespace when someone connects using plex(.usenet.svc.cluster.local). I've deliberately left off things like ingresses and endpoints, which are provided by Traefik or nginx-ingress (basically to let you expose ports 80 and 443 on your host node(s) and then route traffic to the correct pod(s) based on the hostname in the GET request/SNI, amongst other things). Configmaps are another useful thing to wrap your head around - they're the equivalent of mounting a file - for example traefik.yaml - from your local filesystem at a location on your pods except they get 'copied' into the k8s cluster and are available on all nodes for use by all pods that need them. Basically. As I said, I'm just a little bit further down the road than you are in that I've started migrating so there's definitely some constructive criticism to be levelled at my work here! It's interesting stuff though and with each migration I'm learning something new and doing something better/more efficiently. Some day I might even set up my persistent volumes in a way that isn't completely arse-backwards. ---------------------------------------------------------------------- fabio_teixei · 2023-11-21 19:13 Thanks. like jimmy Cliff said, i can see clearly now :p. But realy, that was realy helpfull. I was able to create a deployment for raddar using your exemple. I'm use radarr as my guinea pig app for testing, my plex is on an powerfull machine with powerfull GPU for transcoding. Rigth now i was able to create the deployment but i'm geting an error about my pvc mout: "Unable to attach or mount volumes: unmounted volumes=\[media\], unattached volumes=\[data kube-api-access-68jt8 media dados\]: timed out waiting for the condition" I'm not sure what i did wrong since i have already used the same PVC before for another radarr test deployment before with no problem. But i will eventualy figure that out: `apiVersion: apps/v1` `kind: Deployment` `metadata:` `name: radarr` `namespace: media` `spec:` `replicas: 1` `selector:` `matchLabels:` `app: radarr` `template:` `metadata:` `labels:` `app: radarr` `spec:` `containers:` `- name: radarr` `image: cr.hotio.dev/hotio/radarr:latest` `env:` `- name: PUID` `value: "1000"` `- name: PGID` `value: "1000"` `- name: TZ` `value: "EST"` `ports:` `- containerPort: 7878` `volumeMounts:` `- name: media` `mountPath: /config` `subPath: configs/radarr` `- name: dados` `mountPath: /dados` `- name: data` `mountPath: /data` `volumes:` `- name: media` `persistentVolumeClaim:` `claimName: pvc-media` `- name: dados` `persistentVolumeClaim:` `claimName: pvc-dados` `- name: data` `persistentVolumeClaim:` `claimName: pvc-media` Thanks for the help ---------------------------------------------------------------------- fabio_teixei · 2023-11-21 21:43 Nevermind, i found the problem. SQLlite does not play ball on my SMB share. I will use longhorn