Orchestration software for very small server setup, like very very small one? (k8s, nomad, etc) reddit / selfhosted ---------------------------------------------------------------------- HosonZes · 2023-11-24 18:44 Hi, I've written some microservices and am looking to deploy them. Since I am not very confident in cloud pricing (= too expensive right now), I am looking into ways of operating a very small server setup. Lets say I have 5 services, one is the database, 3 should be run as jobs every minute, one service should be scaled based on load. I am aware that basically described tasks that k8s or nomad would be very good at it. The issue with them is: While I am going to update the services, etc, I do not need a large cluster. I am very sure that I can start with one pod/node and maybe get a second one, if needed. For this setup, k8s (or other flavors) is just overkill (learning and maintaining it). Nomad from hashicorp looks totally cool for that, but it is recommended to have 3 servers with crazy specs for each of them (doing quorum, leader follower, replication, etc.) Which is overkill when I plan to have 1 worker node in total :D Nomad has \`-dev\` option running server and agent on the same node, but in production? I don't know. Nomads server also uses his ip and other things for identity. When they change, the server instance is basically dead and loses its data. That's why a quorum of 3 servers is recommended as a minimal prod setup. Docker compose is not ideal, because I would like to update single containers without tearing everything down. Also, cron for my periodic tasks is not part of docker or docker swarm except plugins, workarounds, or configuring a container running \`cron\` but then meddling with \`flock\`, etc. ​ I am aware that it actually does not sound like I need an orchestrator, but monitoring all the jobs and restarting a container manually sounds not optimal for me and maybe there is something out there, that helps me. Since the tech community knows more than me, I would love to get some other opinions or point of views. Thanks! ​ ​ ---------------------------------------------------------------------- ANotSoSlyLawnTurtle · 2023-11-24 19:05 I've never used them but I've heard good things about [k3s](https://k3s.io/) and [MicroK8s](https://microk8s.io/). Both seem lightweight and suitable to use on a single node. With regards to docker compose though, you can update containers without tearing everything down. See this [StackOverflow question](https://stackoverflow.com/questions/39168239/update-only- one-container-via-docker-compose) for some details if you want to do so manually. You can alao set up something like [Watchtower](https://containrrr.dev/watchtower/) that handles automatically updating for you. ---------------------------------------------------------------------- HosonZes · 2023-11-24 19:12 Maybe the micro k8s versions will do it. Thanks for pointing it out. The good thing is, that k3s and mk8s can run pods on the server. That solves my one machine issue :). The downside is, that it has still the complexity of k8s. But mk8s basically keeps telling me: You like ubuntu, than go grab mk8s :D Watchtower helps updating containers but when docker is not the best fit for cron and scaling I will still have to workaround that. That gave me more options. Thanks u/ANotSoSlyLawnTurtle! ---------------------------------------------------------------------- Arioch5 · 2023-11-24 19:20 Check out k0s https://k0sproject.io/ The k0sctl tool can manage and upgrade the environment very easily with one command. You still have to learn about kubernetes but this with argocd is pretty low maintenance. ---------------------------------------------------------------------- HosonZes · 2023-11-24 19:30 Do you know this project? From the first look I do not see a difference to microk8s. What does it do differently? ---------------------------------------------------------------------- from-nibly · 2023-11-24 20:27 If you are just starting and messing around you can go a long way with a single node k3s cluster (I prefer nixos since it makes managing and replicating things REALLY repeatable, but it is it's own rabbit hole) BUT if you need several 9s your going to need even more than just 1 server with k3s on it. Your gonna want redundancy, monitoring, and processes. 1. 3 nodes while only using capacity of 2 2. Shared volume infra like ceph or a nas 3. Load balancing firewall like opnsense 4. Multiplexed internet 5. UPS for power issues 6. Onsite backups + cloud backups 7. Kube-prometheus-stack (or the contents of) 8. KEDA (for auto scaling) (Not a day 0 recommendation) The reason kubernetes is complex (and hard to learn) is cause it kinda forces you to consider all kinds of reliability, and scaling issues, that you may not need for a while. If you only have one machine, it does feel like a bit much to NEED an autoscaler. You can create a vanilla cron job that runs a docker container command so you don't have to "install" anything on your node. L You can use multiple docker compose files to manage stuff independently so you can upgrade stuff without affecting other things. I know you say you want auto scaling, but what are you autoscaling against? Like is something else scaling up at different intervals? I think a thing to question is if your extra instances ever need to scale down. Auto scaling is a cost saving measure and if you have static infrastructure with no other load then why ever scale down? Do your cron jobs take too many resources and you have to scale down your micro services? If so you've got way more to consider that just plain autoscaling, and maybe you need to scale your infrastructure in which case your back to questioning whether or not you need to scale down. I'm questioning your requirements only because if you are trying to just "get something done" k8s and nomad are going to be a distraction since you aren't already familiar with them. If learning k8s or nomad is also part of your goal, then awesome, I would definitely suggest k3s. ---------------------------------------------------------------------- NiftyLogic · 2023-11-24 20:42 Nomad is totally fine to run on low-spec machines. On my homelab, I have the following running Nomad + Consul: - VM with 1GB as arbiter - 2 MFF PCs with 16GB and i5-6500T Totally fine to run client and server on the same machine in a non- enterprise setup. ---------------------------------------------------------------------- Arioch5 · 2023-11-25 00:22 Yea I know both Microsoft microk8s and k0s. Today I use k0s, I used to use microk8s. My knowledge of it was very good and I'm about 2 years out of date FYI. Microk8s was designed for development and arguably works ok for production. K0s is designed for production but built to be zero dependencies and easy to operate. This shows up in: * Microk8s defaults to dqlite as it's data store, k0s uses the much more performant etcd. * K0s does scale and load testing and advertise what to expect, microk8s doesn't. Microk8s uses snaps by canonical, k0s is statically complied go it can run almost anywhere. Both are backed by reputable open source companies. Canonical (Ubuntu) for microk8s and Mirantis (very active in OpenStack and k8s ecosystem) for k0s. Both are fine, I find k0s more appropriate outside of Dev environments but choose what you like. ---------------------------------------------------------------------- subven1 · 2023-11-25 04:19 How about some software for server management and app hosting like [cloudron.io](https://www.cloudron.io/) ? It is a complete and easy solution to host your own (docker based) apps or you can just install free apps from the build in app store. You can use Cloudrons base image to make use of addons (services) that are already build into Cloudron like: graphite, mailserver, mongodb, mysql, nginx, postgresql, sftp, turn, redis, ldap, oidc, recvmail, scheduler (cron), sendmail and tls or build an app on top of the [LAMP app](https://docs.cloudron.io/apps/lamp/). Everything is automated from OS updates, plattform + app based backups (with persistence if needed) to proxy setup and certificates. Besides the webUI, Cloudron also [provides a RESTful API](https://docs.cloudron.io/api.html) to manage apps, users, groups, domains and other resources. It also has its own [Build Service](https://docs.cloudron.io/apps/build-service/) and [Image Registry](https://docs.cloudron.io/apps/docker-registry/) or you could host your own Gitlab/Gitea with just one click. Instead of real orchestration you maybe could use automation tools like n8n or Ctfreak to archive what you need. Cloudron is free for up to 2 apps so keep that in mind but it runs well on a VPS with as low as 2GB RAM and 25GB of disk space. ---------------------------------------------------------------------- HosonZes · 2023-11-25 07:47 >sure and if you have static infrastructure with no other load then why ever scale down? Do your cron jobs take too many resources an Need to think about this. I am sure, that several docker files or compose projects will do some of the jobs. Main reason I am hesitant: I am pretty sure, services will fail, and need to be restarted. Docker can do this. I also know I have to meddle with cron jobs and I can built the cron inside the docker container that should run the service. But this is some sort of work-arounding a lack of a "job" feature in docker and it also feels wrong to mix the service with the infrastructure together (self-imposed pain :D) One of the continuous services is actually one, I do not know in advance, how it will perform. It processes many concurrent jobs, which is a good thing, but I also suspect, that I still will have to replicate this service on the same machine to utilize resources better, adding a small load balancer for it, etc. One service instance is concurrent but still limited to a certain degree. This part will need some monitoring but also tailoring. And for this I would prefer a proper solution, albeit one with reduced learning curve. Monitoring is a thing, too: I would love to have a dashboard to see my tiny services in action and vanilla docker does not provide it. There will be error streams I am going to log into an observability platform but I still feel, that I need more, than docker provides out of the box. Also: I love new tech, but I do not expect, that my tiny one server vps SaaS will instantly blow up, so I will have to autoscale it onto multiple federated cloud providers or a fleet of VPS machines. While I do not want to waste time learning a thing, I cannot properly handle alone, I also would like to learn a technology that helps me getting started right now but also provides a path for later growth. That's why I hesitate to spend time with single docker containers and low level fiddling (the cron part) inside those, because I did not find evidence, that it provides the potential (even with docker swarm) ---------------------------------------------------------------------- HosonZes · 2023-11-25 07:55 >k space. You could rent 2 small servers, one for development and one to This is a new option. Thanks for this. Did you used this in production? I know I will have custom and tailored images running, because each of my containers (besides the database) will be my own services, and cloudron looks like it was rather designed to pick a ready solution, am I understanding it correctly? It also says, it keeps the systems up to date, which again is very high level, usually it is some sort of terraform to provision or ansible to configure the machine and abstracting those details makes it hard for me as a tech guy to understand, what they actually are doing. ---------------------------------------------------------------------- subven1 · 2023-11-25 08:24 >Did you used this in production? I run 3 Cloudron servers for many years and administer another 4 with some just beeing used inside a LAN. ​ >cloudron looks like it was rather designed to pick a ready solution, am I understanding it correctly? Most users will just pick apps from the store but others like myself use Cloudron to host their own services and custom app packages. It is [actually pretty easy](https://docs.cloudron.io/packaging/tutorial/) and there is a lot of help and templates at the [Cloudron app packaging forum](https://forum.cloudron.io/category/96/app-packaging- development) if you just start. ​ >It also says, it keeps the systems up to date, which again is very high level Cloudron uses neither Ansible nor Terraform and relies on scripts and crons. It uses automatic Ubuntu security updates, firewall and a bit of OS hardening to secure the plattform. You can [take a look at the s ources](https://git.cloudron.io/cloudron/box/- /tree/master/src?ref_type=heads) if you are curious. ---------------------------------------------------------------------- from-nibly · 2023-11-25 14:45 To clarify I was talking about using the hosts native cron not the containers cron. And simply execute the container at the given time. But if you want to learn something then go with k3s. It's incredibly simple to get going. ---------------------------------------------------------------------- HosonZes · 2023-11-27 06:45 After spending a couple of days with k8s I really want to get back to nomad. k8s complexity is way too high. The one thing about nomad is: There is much less documentation and community out there for nomad. Especially how to secure a nomad cluster in production. When you get stuck in internal details where you can't get out, you might be in trouble. Have you had small production workloads running in public? ---------------------------------------------------------------------- NiftyLogic · 2023-11-27 11:54 Same here, had a deeper look at MicroK8s and decided to go the Nomad route... Unfortunately, I'm just running a homelab setup. With two publicly exposed services, but noting enterprise like. Does that count as "in producion"? If yes, what are your questions? ---------------------------------------------------------------------- HosonZes · 2023-11-27 13:13 When I think about containers in production I think about security and isolation. Like in a dev environment let the containers talk or discover what ever there is in their network (whether it is docker, k8s or nomad). But in production I'd rather have isolated and secure containers as much as possible, because being exposed to the internet may always result in comprised services, code injection, hijacked containers and so on. This would be the wrong time to learn that container security is a thing and there may be security best practises on cluster level as well. But if such information is not available, then using the simple approach might also mean "easier to hack or break". ---------------------------------------------------------------------- NiftyLogic · 2023-11-27 14:08 Agree, the community for Nomad is much smaller than for K8S. Making it harder to follow established best practices. OTOH, a K8S which is not well-understood is IMHO an even higher security risk. Simplicity is security. ---------------------------------------------------------------------- NiftyLogic · 2023-11-27 22:04 Actually, hardening the Hashistack is not too hard (pun intended). IMHO this is the better documented part of their tools. For Nomad, Consul and Vault, just make sure that you define proper roles and assign tokens to these roles. Minimal permissions are pretty easy to set up. Besides that, Consul Connect helps to encrypt the connections between services via mTLS. In addition, you can declare intentions in Consul which services are allowed to talk to each other.