Infrastructure as a Code reddit / selfhosted ---------------------------------------------------------------------- driversti · 2023-12-18 17:35 Hello everyone. I have about 15 or 20 self-hosted Docker containers on my Raspberry Pi 4 and now I want to move them to a mini PC. I use Portainer to spin up the containers, but manual management is boring and not easy to restore when something goes wrong or when moving to different hardware. So I decided to declare in code what I want to achieve. I'm a Java developer with some Docker experience, and have recently become familiar with Ansible. I'm currently considering two options: \- Docker Compose So I decided to declare in code what I want to achieve. I'm a Java developer with some Docker experience and have recently become familiar with Ansible. I'm currently considering two options: The first option is the easiest and most intuitive. In one (or more) Docker Compose files I can declare the configuration of each of my services and their dependencies. At the same time, I can also use Ansible to do this. However, it is not clear to me how to do this. The questions are \- What tools do you use to maintain the configuration of your homelab? \- Which do you prefer: docker-compose or Ansible? \- What else would you suggest to consider in my case? ​ Thank you all for your answers, suggestions, attention and time! ---------------------------------------------------------------------- clintkev251 · 2023-12-18 17:48 Ansible and Docker compose are both great tools that have different strengths. Ansible is great for stuff like managing and provisioning machines, like if you wanted to ensure that certain dependencies and configurations are applied. Docker compose is focused on just managing the state of your containers. It does this very well. So basically both are great tools, but you should always use the right tool for the job, and between docker compose and ansible for managing containers, docker compose is the right tool ---------------------------------------------------------------------- Eirikr70 · 2023-12-18 17:48 I use Docker compose. I have no knowledge of Ansible, but Docker compose makes it very easy. ---------------------------------------------------------------------- kongu123 · 2023-12-18 17:54 Keep in mind that with Docker-Compose, you can continue to use portainer if that is an environment you are comfortable in. ---------------------------------------------------------------------- etnicor · 2023-12-18 18:14 Use systemd to start/stop containers. E.g. manage the containers as a systemd service like any other Linux service. Manage systemd files with ansible. docker-compose and ansible has nothing todo with each other. ---------------------------------------------------------------------- themonstersarecoming · 2023-12-18 18:28 Here was my experience: I took a couple of weeks and set everything up with Ansible and Traefik and it was AMAZING! It was so cool I could set up multiple machines and get my network exactly how I wanted in a declarative way. I could nuke the whole thing and have 4 machines all set up and talking to each other in a couple of mins. Fast forward a couple of months and I needed to change or fix something, and now I can't remember exactly how everything works. My takeaway was that these are amazing tools, but unless you're going to write yourself extensive documentation then it is more trouble than it's worth. If I was a system administrator using Ansible every day it would be my favorite thing ever invented. Or, if I was working with a bunch of people we could use it for our infa along with the magic of git. However, as a normal person who can't remember stuff they don't use regularly, I found it added complexity that I rather not have. I went back to just using Nginx Proxy Manager and just set my machines up manually when I need to, although I do use docker-compose still (really, that's extremely easy to understand). ---------------------------------------------------------------------- thesarthakjain · 2023-12-18 18:34 Hey, can you please share a list of services you're running on your pi? I'm basically you but with just a handful of services. ---------------------------------------------------------------------- thesarthakjain · 2023-12-18 18:36 I personally use docker compose, what I do is have a single big compose file that runs everything on production and rest all the containers that come and go as I test them are on another compose file at some other folder path. This way I can keep testing new services without disrupting my family from enjoying the movie. ---------------------------------------------------------------------- AdventurousSquash · 2023-12-18 18:56 Terraform to create the vms, ansible to config those vms, and compose files for running the services I host ---------------------------------------------------------------------- drakgremlin · 2023-12-18 18:58 I use Kubernetes which scales with machines but is opinionated. You'll get a higher return by running more services on them. I actively stay away from Chef/Puppet/Anisble/Salt family of systems now. There are a lot of people who use them but it feels like everything is a hammer with them. As an orchestration layer they lack feedback mechanisms and are expensive to run. ---------------------------------------------------------------------- crusader-kenned · 2023-12-18 19:00 Why use systemd isn’t just as good to simply use restart-unless- stopped in docker? ---------------------------------------------------------------------- rm968211 · 2023-12-18 19:03 Not super familiar with ansible, but I tend to try to take the simplest possible approach to things like this and avoid unnecessary bloat while also maintaining max portability. My personal choice for doing this is docker compose. My server setup is a git repo containing my IAC which consists of multiple docker compose files. Each compose file is essentially a category of service, networking, streaming, game servers, etc. Also in this repo I have some shell scripts to start specific services, or all services at once (this is optional and just a convenience thing for me. The scripts just use docker compose up commands with various flags to point at the secrets file, etc.) I then have a different git repo containing all my “secrets” that get injected when docker compose up is called. I put any volumes that need to be created for these containers in one folder called config, with subfolders for each container. Once this is all setup, all you need to move your setup to a new machine is a copy of your config folder, clone your two git repos, and run docker compose up. I do use portainer, but I pretty much use it as a more convenient way to view logs, see what’s currently running, or stop start reset problem containers. ---------------------------------------------------------------------- 10031 · 2023-12-18 19:06 I wish Portainer didn’t completely take over the compose management though. Dockge does it well, but i still keep portainer when i need to see any dangling images or whatever. ---------------------------------------------------------------------- YYCwhatyoudidthere · 2023-12-18 19:07 I can so relate! The number of times I have implemented the "perfect" solution that then didn't require ongoing attention and forgot the details. Started the research to figure it out again, got distracted by a newer project, and end up in a bake-off rabbit hole. Damn all of you :-) ---------------------------------------------------------------------- nhermosilla14 · 2023-12-18 19:19 I would say Docker Compose can do pretty much all you described here. Ansible could be useful to make it even more automatic (because you can describe the whole environment), but I'd say it will only introduce unnecessary complexity to your use case. ---------------------------------------------------------------------- osuhickeys · 2023-12-18 19:19 I use both, ansible to configure a new VM they way I want it including installing docker and cocker compose and then docker compose to deploy services. ---------------------------------------------------------------------- Effective-Heat-7903 · 2023-12-18 19:21 >What tools do you use to maintain the configuration of your homelab? >- Which do you prefer: docker-compose or Ansible? I do both. The server (VM) is completely configured via Ansible. That includes creating the VM including cloud init config, installing packages (Docker), configuring the docker service. Next step is the docker deployment via Ansible. In the same Ansible repo, I keep my docker compose structure with compose yamls, and config files. These files are all ready to be copied over to the host and templated. Although I try to keep the templating to mostly .env files, seldom other config files and never the compose yaml itself to be able to up the compose file without Ansible. That works pretty well for me as I've developed a small generic role that can deploy any compose project and can control all my config via inventories. If people here are interested, I would be open to share my project on GitHub (after some anonymozation I would need to do) Throwaway because I probably would publish on my personal GitHub. ---------------------------------------------------------------------- sitram · 2023-12-18 19:24 I have divided all my containers into multiple docker-composer files. This way I can easily spin them up on another docker instance in case I want to migrate them. ---------------------------------------------------------------------- JuiceStyle · 2023-12-18 19:42 K3s is what I use for my homelab. It's simple to install and you treat it like any k8s cluster. Use k9s and helm and your favorite cicd (recommend Gitlab runner kubernetes executor) or fluxcd/argocd to install things. ---------------------------------------------------------------------- Square_Lawfulness_33 · 2023-12-18 19:51 Try dockge, is primarily a stack manager and it stores the compose file in actual text files. ---------------------------------------------------------------------- driversti · 2023-12-18 19:54 I have very little experience with k8s. I'm considering k3s to get started, but currently don't even know what to deploy there. I have 10+ LXC Alpine Linux servers on the second RPI4 and more servers will come soon, but I'm not sure this is what I need right now. But anyway, thank you for the suggestion :) ---------------------------------------------------------------------- driversti · 2023-12-18 19:55 This is the most realistic scenario in my case too. I could create a git repository and store all setups there. ---------------------------------------------------------------------- ArgoPanoptes · 2023-12-18 19:57 I use docker-compose and recently moved all the stacks from Portainer to Dockge because it saves all the stacks as compose files in a defined folder, I can backup them, and version control. I know how to use Ansible but unfortunately didn't write a playbook to reinstall all the environment if something goes wrong because it would take so much time to write but I will one day. ---------------------------------------------------------------------- Simon-RedditAccount · 2023-12-18 20:08 > I would be open to share my project on GitHub Sign me up. Actually I feel this is the right way to do. Currently I'm *more than familiar* with docker-compose, but haven't played with Ansible at all. Both do different jobs, and do them great. ---------------------------------------------------------------------- driversti · 2023-12-18 20:10 I like your approach. Thank you. I have no plans to use VMs on my mini PC but run Docker containers on bare metal. What advantages do you see for yourself in using VMs? ---------------------------------------------------------------------- driversti · 2023-12-18 20:10 What advantages do you see for yourself in using VMs? ---------------------------------------------------------------------- driversti · 2023-12-18 20:11 Yeah, I think so too :) ---------------------------------------------------------------------- driversti · 2023-12-18 20:14 Awesome! Thank you for sharing your experience and approach. ---------------------------------------------------------------------- driversti · 2023-12-18 20:15 Interesting... I fell in love with Ansible. I set up multiple roles, hosts, vaults, and playbooks and now setting up a new server is so much easier! ---------------------------------------------------------------------- driversti · 2023-12-18 20:16 What benefits give you VMs? Why not a bare metal? ---------------------------------------------------------------------- bgatesIT · 2023-12-18 20:18 Wait till you get you’re feet wet with Kubernetes :) ---------------------------------------------------------------------- driversti · 2023-12-18 20:19 Sure! Here are my services: Consul, filebrowser, Homepage, Jellyfin, MeTube, stream, Nextcloud, NPM, overseerr, PgAdmin4, PiHole, Portainer, PostgreSQL, Prowlarr, Radarr, Docker Registry (for my images), Transmission. ---------------------------------------------------------------------- driversti · 2023-12-18 20:21 So true :D ---------------------------------------------------------------------- driversti · 2023-12-18 20:23 Yeah, I know what you're talking about :D That's why I want to document my setup. Document in code, because I love code. It always says truth and is up-to-date, which is not possible with any documentation. ---------------------------------------------------------------------- Old-Assumption4984 · 2023-12-18 20:23 I was in a similar situation (also a Java dev) with my selfhosted projects which I also write. I decided to do what we do in work environments and switch to kubernetes. I chose [https://k3s.io/](https://k3s.io/) and selfhost my own private docker registry. I used [https://kompose.io/](https://kompose.io/) to convert my docker compose yaml's to kube manifest yamls. ---------------------------------------------------------------------- sitram · 2023-12-18 20:25 That is exactly what I do. I set up a name for every stack, like infrastructure, managers, databases, tools in .env files. This way I can easily see what containers belong to each stack in portainer. Most importantly I can bring down entire stacks from the command line without actually taking down portainer or other critical containers. I used to have one big file for all containers but it got quite annoying when I had to make changes. It recreated a lot of containers not just the one I wanted, which wasted a lot of time. ---------------------------------------------------------------------- driversti · 2023-12-18 20:26 >docker-compose and ansible has nothing todo with each other. Totally different software for different usecases. That's true and I've mixed them up here. Unfortunately, I'm not familiar with systemd and after reading others' answers I conclude that docker-compose is the tool I need. Simple and readable. And always up-to-date ---------------------------------------------------------------------- JuiceStyle · 2023-12-18 20:31 So I was going to reply with "learning k8s is easy with k3s!" And started writing a beginners guide, but then I got to 4 paragraphs and said to myself.... Yeah this isn't really as easy as I initially thought 😂 There's def a lot to ingest to learn k8s, the hardest part really is knowing what parts to learn in what order. ---------------------------------------------------------------------- driversti · 2023-12-18 20:32 Agree. Thank you :) ---------------------------------------------------------------------- roytay · 2023-12-18 20:32 I know there is an ansible module you can use that replaces the docker-compose.yml, but you can also keep your dc yamls and just use ansible to copy them to the target(s) and run them. That way if you drop ansible someday, you still have your dc's. And you can run the kinds of containers that *use* the dc. Your playbook could also make sure docker and docker-compose are installed and do any other pre-launch prep you need. You could have playbooks for "up", "down", "upgrade", etc. Honestly, the ansible helps most when you have a bunch of nodes to configure. For one server, running ansible from your laptop isn't that much better than ssh to the server to run docker-compose. So for that case, I would say do it if you want to learn ansible. ---------------------------------------------------------------------- drakgremlin · 2023-12-18 20:32 If it works for you then more power to you! ---------------------------------------------------------------------- New_d_pics · 2023-12-18 20:33 I spun up [gitea](https://github.com/go-gitea/gitea) for mostly the same reasons, was moving from an Ubuntu server to proxmox. I used [autocompose](https://github.com/Red5d/docker-autocompose) to pull my current container compose files, copied them into individual entries on gitea, edit accordingly for new location then spin em up on new server. Also learn to use individual volumes for containers if you've not yet, it makes it super easy to move around. ---------------------------------------------------------------------- driversti · 2023-12-18 20:34 Yeah, I use both, 'cuz Portainer is a very good visualization tool for me. And it allows spin up a new service very quickly. Then I can move the new service to the docker-compose file with all required settings. ---------------------------------------------------------------------- driversti · 2023-12-18 20:37 Completely agree. I guess mentioning Ansible was a mistake. But thanks to the topic discussion, I convinced that docker-compose (potentially multiple them) is exactly what I need. ---------------------------------------------------------------------- driversti · 2023-12-18 20:39 I guess in my scenario, this would be an overkill. I wasn't clear enough in my post. ---------------------------------------------------------------------- driversti · 2023-12-18 20:39 😂 ---------------------------------------------------------------------- driversti · 2023-12-18 20:40 Thanks. I haven't heard about it yet, but will give it a try. ---------------------------------------------------------------------- bgatesIT · 2023-12-18 20:40 It just takes the whole portainer/docker-compose concept and props it up on Steroids. I started playing with it about 6 months ago and implemented it almost everywhere i could. ---------------------------------------------------------------------- patientbold · 2023-12-18 20:40 Funny that your counter example is exactly the reason I love IaC: I don't have to remember how things work, it ~~is~~ can be self documenting. In my case, the only thing I know is that I have a folder with all my IaC defined. I open that folder on vscode and then I glance to remember things like \`run-ansible.sh\`. All the hours of research to setup my home infra is automagically documented in my ansible (or shell) scripts. Being using the same folder for at least 5 years already. To answer OP, specifically: I recommend ansible + docker compose. EASY to learn, easy to maintain and it has being working for years in my case. ---------------------------------------------------------------------- driversti · 2023-12-18 20:42 Thanks. I haven't heard about it yet, but will give it a try. ---------------------------------------------------------------------- AdventurousSquash · 2023-12-18 20:47 At home I only have a dusty old stationary running proxmox which serves what I need in terms of a couple of vms, some running containers in them. For my other stuff I use the cloud, since I work at a provider I have free use of our resources (as long as there’s no shortage). Terraform and ansible let’s me spin those vms up in seconds and configure them based on my needs whether it’s on my own proxmox or in the cloud (just different tf providers). All the IaC is hosted on my own gitea and mirrored to GitHub in case my shit breaks. It’s handy and works for my use case is the best I can say :) ---------------------------------------------------------------------- osuhickeys · 2023-12-18 20:49 I chose to use a VM for a few reasons: 1. I host the VM volumes on a NAS. Moving to a new host is as simple as removing an iSCSI mount from the old host and presenting it to the new host. So migration to new hardware is a few mins. of work. 2. I've got automation in place with Packer and Vagrant so building a new VM from scratch based on the latest Ubuntu downloadable iso is also a few mins of work if I need to go down that path. All of the data is on a separate volume that I can just detach from the old VM and attach to the new VM. Run the ansible playbook and docker compose up and can be on a new VM in less than an hour with all services up and running. 3. Resource containment - I have one VM that hosts a media player and other apps for family and friends. I have another VM that is hosting a container to do media transcoding (and a few other more resource intensive applications) which use a lot of memory and CPU. Keeps transcoding from causing other apps family and friends are using from being impacted. Cold probably accomplish the same thing with one VM, but with the automation in place, it is pretty simple to use a couple of VMs and keep the OS config more simple IMHO. ---------------------------------------------------------------------- Alucard2051 · 2023-12-18 20:54 I currently am in the process of learning Ansible and terraform more out of fun than need. For my actual set up though, I am using docker compose running on a git server. Ansible seems awesome as a back up incase your home lab is hit by lighting, but compose takes much less time to edit and maintain ---------------------------------------------------------------------- Alucard2051 · 2023-12-18 20:57 Not going to hate on anyone use Kubernetes, but I will say that once you start using it, it becomes your life. I feel like I had to trouble shoot more using Kubernetes than any other type of set up ---------------------------------------------------------------------- Bill_Guarnere · 2023-12-18 21:05 Imho there are two paths you can follow: 1. go ahead with plain docker-compose and get rid of portainer or any web interface, and use git to manager versioning of your docker- compose manifests, in this way you will abandon the "clickops" way portainer makes you follow (just like any web interface, think about Rancher on the Kubernetes world) 2. take this opportunity to dig more and more in the infrastructure as code path, it's not necessary for running your services, think about as an educational path towards it: 2.1. move from docker to kubernetes (K3s is a good way, it's easy to install and monolithic as a single node cluster) 2.2. define every single piece of your K8s infrastructure (from deployments, pods, statefulsets, services, persistent volumes, ingress, secret, configmap, etc etc..) as manifest file, not a single shell command, declarative mode is the way 2.3 setup ArgoCD and setup pipelines to create and manage every single piece of your architecture thought it, you should be able to setup everything or change everything with a commit on your git repository. ---------------------------------------------------------------------- SkipTam · 2023-12-18 21:07 Ansible, docker compose and git. My tools for mostly everything I run. I’m staring to look into dockge and maybe gitea runner for automatic deployments of docker compose ---------------------------------------------------------------------- micalm · 2023-12-18 21:38 AFAIK Portainer is well known for no reasonable backup/recovery options. That's bad luck/a choice I also made. If you find a good alternative, let me know. For now, documentation and bind mounts wherever possible is the best I could find. ---------------------------------------------------------------------- sublimesinister · 2023-12-18 21:41 I use a NixOS config that's committed to git. There are various degrees of how much immutability you wanna enforce, but I'm just referencing images from Docker hub using tags. I use Podman as the container backend and manage networks and such using systemd services. One thing I haven't figured out yet is how to do alerting for new versions. ---------------------------------------------------------------------- arkane-linux · 2023-12-18 21:52 Ansible and docker-compose are two very different tools which strive to do very different things. Ansible will be hyper verbose and quirky if used to manage your Docker envirnonment. docker-compose can't (I guess you could if you are presistent enough) be used to configure your bare metal machine meaning you will still have to configure that whole part manually. ​ Use Ansible to configure the bare metal OS, docker-compose for the Docker environment. ---------------------------------------------------------------------- LidgChris · 2023-12-18 21:53 id probably declare it in code. ---------------------------------------------------------------------- Tiwenty · 2023-12-18 21:59 My services are managed by compose, but I also wrote an Ansible to clone my compose repo, copy some stuff, etc. So I guess you can do compose for the services, and Ansible if you wish to setup some stuff around it :) ---------------------------------------------------------------------- Then-Quiet-5011 · 2023-12-18 22:40 K8s + tanka + ansible + git. Whole setup can be deployed with couple of commands in few minutes. ---------------------------------------------------------------------- esturniolo · 2023-12-18 22:49 The story of my life. Now I’m in love with Packer, but I know that I’ll cheat it on the future. Again. ---------------------------------------------------------------------- _k4mpfk3ks_ · 2023-12-18 23:18 You could also use a combination of both. Use Ansible for setting up system level dependencies (e.g. docker itself) and then use the docker compose Ansible module to apply your docker compose file. ---------------------------------------------------------------------- Mean_Einstein · 2023-12-18 23:26 Have a look at ansible-doctor for writing documentation automatically based on the ansible roles variables and stuff. ---------------------------------------------------------------------- ZaxLofful · 2023-12-18 23:35 I use Docker Compose and Gitea, pretty much the exact way you laid out! ---------------------------------------------------------------------- PM_Me_Rulers · 2023-12-18 23:48 How do you deploy docker compose with ansible? I've already got docker compose files with split by type (network containers, services, DB etc) but it's all on a single Ubuntu proxmox VM. I've got ansible running to configure a base debian12 lxc with but still figuring out how to neatly slot docker and docker compose in. Do you just copy in the compose files and set a systemd service to run them on boot? ---------------------------------------------------------------------- TuhanaPF · 2023-12-19 00:18 Yes, I've decided to rebuild my entire setup from scratch and figure everything out again like I did the first time. And this time, I'm documenting everything heavily, including links of where I found specific answers to really niche problems I have. I think it's an essential part of self-hosting if you ever hope to get your setup working again. ---------------------------------------------------------------------- pandupewe · 2023-12-19 00:31 I also do this because vm tend to easier to backup as whole and move all of my service to different system if needed ---------------------------------------------------------------------- Beastmind · 2023-12-19 01:12 Sometime a well written bash script is better than drowning into something big like ansible ---------------------------------------------------------------------- cr1tic · 2023-12-19 02:21 You're half way there, documentation with examples is always necessary. Assume you won't remember a thing when you return to it and just right up a decent README.md ---------------------------------------------------------------------- pogky_thunder · 2023-12-19 02:47 Keep all your projects on separate compose files with a good docker data hierarchy. That way moving to new hardware is a copy paste. You can start a single service with a docker-compose command and all your services with systemd. ---------------------------------------------------------------------- themonstersarecoming · 2023-12-19 02:55 The best part is I don't have to assume. I'm 100% confident that I won't remember a thing. I'm like a reverse Neo, "I knew PHP?" 🤯 ---------------------------------------------------------------------- cr1tic · 2023-12-19 02:57 💯 ---------------------------------------------------------------------- sbbh1 · 2023-12-19 03:18 I went overboard with Ansible and have every service defined in a yaml file which deploys the container, sets up traefik, adds config files and creates an uptimekuma monitor among others. https://github.com/Subbeh/homelab/tree/main/ansible/apps Role here: https://github.com/Subbeh/homelab/tree/main/ansible/roles/apps ---------------------------------------------------------------------- ismaelgokufox · 2023-12-19 03:57 Ahhh, dockge, portainer’s core functionality without making your stacks hostage of it. ---------------------------------------------------------------------- nonlogin · 2023-12-19 04:15 There is docker provider for Terraform. Works really good for me, replacing compose files. ---------------------------------------------------------------------- holzgraeber · 2023-12-19 05:54 Except with bash, you start to add some temporary fix and it gets a bit convoluted. Since you then don't understand it anymore, you just patch something together until you get fed up with it and start all over again. That's where J see the benefit of ansible, because it forces you to keep everything well structured. ---------------------------------------------------------------------- sitram · 2023-12-19 06:30 Whenever possible I try to map volumes to external folders for my containers. I have a location on my NAS server where each contain has its own folder with configuration files. Even if my entire VM running the docker containers crashes, I can quickly spin up the containers in another VM and point them to their configuration folder from NAS. In case I retire a certain service, I just archive their corresponding configuration folder and keep for later in case I change my mind and want to reuse that service. ---------------------------------------------------------------------- mirisbowring · 2023-12-19 07:00 All points can be realized with containers ---------------------------------------------------------------------- JMPJNS · 2023-12-19 07:10 ansible has a docker compose module that you can use for thathttps://d ocs.ansible.com/ansible/latest/collections/community/docker/docker_com pose_module.html ---------------------------------------------------------------------- ebrious · 2023-12-19 07:29 One thing I did while learning was just write a `provision.sh` script that provisioned everything up in one go. Same with `backup.sh` or other thin wrappers around the ansible-playbook command. These aren't really helpful when you're familiar with things, but was super helpful when I wrote something I didn't revisit for months and had entirely forgotten Ansible syntax/commands. ---------------------------------------------------------------------- Varnish6588 · 2023-12-19 08:11 i use terraform and helm charts as I use kubernetes for my home setup ---------------------------------------------------------------------- localhost-127 · 2023-12-19 09:10 Is there a way to connect multiple environments into one centrally managed dockage? ---------------------------------------------------------------------- Square_Lawfulness_33 · 2023-12-19 09:12 As far as I know, No. ---------------------------------------------------------------------- Krax0x · 2023-12-19 09:37 It doesnt have to completely take over management - I use docker- compose files and run it from the system, and just manage images from portainer, I prefer not to add the compose file in the portainer environment. ---------------------------------------------------------------------- px6l_ · 2023-12-19 11:14 r/nixos FTW ---------------------------------------------------------------------- nicdjb · 2023-12-19 11:16 Is this a bad thing? I'm using GitHub to hold my compose files and using the GitOps process to update deployed stacks. Genuinely asking as perhaps what I'm doing isn't the best way :-) ---------------------------------------------------------------------- Not_your_guy_buddy42 · 2023-12-19 11:57 Maybe I am doing it wrong but I setup Ansible to: * create user and dirs to run compose * chown/chmod * create docker networks and whatever other preliminary steps * create the docker compose file - just using blockinfile: - based on variables (also secrets from vault) to run containers with UID/GID of the new user, do bind mounts, setup a subdomain on traefik, add a backup container, add entry to the dashboard, logging, and on and on. Thanks to ansible I don't need to redo all that from scratch with each project. * it is based on templates so I mostly need to adjust a few variables and adjust the blockinfile: statements which write the compose file and config files, * create any other needed config files also using blockinfile: Example: write traefik.yml Example 2: write a custom nginx.conf to map to an obstinate container that wouldn't accept settings * all manner of checks if user, dirs, files, mounts etc exist and the previous compose is properly cleaned up... then run the compose (with -p project name , also a variable, so it doesn't get confused) * do anything else specific projects need (create a user in database, etc) I can't believe I have written so much because I'm more of an ansible noob , well maybe it helps someone or maybe it's nonsense. YMMV ---------------------------------------------------------------------- viggy96 · 2023-12-19 13:45 I've been running docker-compose with traefik as a reverse proxy for a while now, and I barely touch it now. Updates itself, and I never really have any issues. I upload my config to GitHub: https://github.com/viggy96/container\_config ---------------------------------------------------------------------- amarao_san · 2023-12-19 14:25 I'm doing Ansible full time at work and I'm damn good at it. I've tried to do the same for home stuff and realized that it does not worth it. Everything is designed for enterprise-grade: you have CI, multiple runners, test for roles, hundreds of paid hours spend on them, and you call it infrastructure as a code (and you see how you can make it a bit less horrible if only you have fixed requirements and few more months). The same at home just does not qualify: you don't have hundreds of hours of dedicated hard work, you can't just spin 10 copies of your home infra just to test all 10 scenarios with different versions, etc, etc. For home automation xkcd chart is the real answer: https://xkcd.com/1205/ If you do it often, automate to free time. If you do it twice a year (or very two years once), don't. ---------------------------------------------------------------------- Icy_Holiday_1089 · 2023-12-19 15:00 My opinion on this is that the code used to generate infrastructure shouldn’t obfuscate the infrastructure you’re building. Start with a Git repo with docker compose files and as long as it’s organised and documented it’s a good starting point. I love ansible but there is no immediate need to use it especially if it’s simply going to overwhelm you. Docker also has a very good build system which can also be used for building infrastructure. I use it for a few small projects but I haven’t really used it enough to recommend it. Unfortunately I’m a bit more old school and my of my projects are still bare metal + ansible based as I’ve built and maintained them for years this. ---------------------------------------------------------------------- Scavenger53 · 2023-12-19 15:23 you usually dont put containers on containers, you spin up a vm and put them on that. ---------------------------------------------------------------------- Typhon_ragewind · 2023-12-19 16:23 It's odd that you find it difficult to migrate your portainer based system to another machine. I did a major overhaul a couple of months ago and just restored the portainer persistent data folder and got it running again. After a few minutes of downloading images all my previous services (around 100 containers) were back online (minus a couple of minor permissions issues on some of them). As for your questions: 1. Portainer, since it allows me to easily manage all my nodes using docker-compose. Portainer data is backed up using restic 2. I never actually tried ansible, so no opinion here 3. If restoring from Portainer data doesn't work for some reason, you can try setting up the mini pc alongside your pi and use portainer stack migration (or do it manually, you don't have that many services from what i've seen in the comments). Of course this assuming you want the simpler option - if your goal is to learn ansible, that avenue also looks like a fun project :) ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 17:01 ansible has a module for docker-compose or you can could just use any command you'd normaly use (if you find nothing). ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 17:11 ansible code IS the documentation, i know you heard this before but since ansible is no programming language its actually true. So unless you leave out important manual steps, the next run will do exactly the same. actually i am a admin but i cant remember either all the small details and i dont write ansible every week. but still i can ready back my playbooks and roles easiliy. ansible is imperative, not declarative, but i guess i mean it like "writing" (code). ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 17:21 a well written bash scripts is far more complex than ansible, most people are too lazy (or not experienced enough) to handle errors (trap, pipefail, ...) and you would need to implement something like a dry-run ("check-mode"), debug output of vars etc. which ansible has already built-in. also templating ist not fun with all the escaping. not to mention the different shells and their special Syntax. I would not go back to shellscripts anymore unless its a very simple task (like a loop with one task or so). ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 17:28 just use playbooks instead (with different vars), tasks goes into roles. all you need to is remember the command (ansible-playbook), inventory is defined in ansible.cfg, playbooks and -C (check-mode) -D (diff- mode) or -l or --limit to run it on specific hosts/group (better place is in playbooks). i would say thats not too much to remember? ---------------------------------------------------------------------- themonstersarecoming · 2023-12-19 17:42 For someone getting one shot exposure to it and then having to come back to try to fix something, it was a lot of mental overhead to try to figure out how it worked again. Like anything, I'm sure if you're more familiar with it, or organize it better than I did then it's easier to follow, but again that requires putting in the time. My brain just works better with well layed out visual information like GUIs with helpful tooltips, instead of searching through a bunch of files in folders and lines of text, which is truly unfortunate because I conceptually much prefer IoC type systems (Ansible, and NixOs, for example), and they have the added bonus of working a lot better with LLMs. However, my experience was that it ended up not being worth me spending weeks learning how to do everything I wanted Ansible to do, just to come back to it and be lost again months later. Of course it was a learning experience and was fun, so I don't mind that I did it, but I ended up going back to how I was doing it previously. For people in my use case that aren't trying to get a job in networking or system administration, I'd recommend against overcomplicating things with stuff like Ansible, Kubernetes, Docker Swarm, etc. because although I believe they are truly better systems, they have knowledge and learning overhead that isn't going to sink in if you're just setting up one network and forgetting about it. At least that was my experience of a lot of time wasted learning how to do Ansible and Docker Swarm over a couple of weeks. Basically I set it up to be more stable, and found it was much less stable because of my lack of experience with the systems. ---------------------------------------------------------------------- AdventurousSquash · 2023-12-19 17:50 True! It’s in my “read” bookmark folder for a rainy day 😅 ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 17:55 Dont call tool useless unless you understand what they do. I could just as easily say k8s is useless because you can do everything with ansible/VMs/docker but i know there are pros and cons for every decision (and complexity is one of the main issues here). I would recommend ansible as basis (automation basics...), whatever you manage after that will be a lot harder than that. ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 18:11 Configuration management tools are not a "layer", they are bootstrap/management tools to build the house, whereas you focus in the interior work i guess? ansible is more like a tool kit with all basic tools you need, the others are more complex (more like programming, would not recommend to start with them). ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 18:15 thats totally fine, from all the tools imho ansible was the best mix of features vs. complexity, but you are right: complexity is the end- boss. ---------------------------------------------------------------------- blind_guardian23 · 2023-12-19 18:20 One service, one VMs. Snapshots, migration, rollback etc. per App (as you would do with Containers). On bare-metal its not great to manage (conflicting) apps unless you add some layer (like containers). ---------------------------------------------------------------------- Fluffer_Wuffer · 2023-12-19 19:27 I use custom python scripts, that spit out packer configs to build my VM's, then dumps a.script.which leans on K3Sup to stand up Kubernetes and core components (CNI, 1Password, ArgoCD), then ArgoCD to deploys everything else... I got fed up with some stuff breaking, such as Gitlab etc.. so I moved everything critical and non ephemeral to external services, to Github, 1Password, and my Registry etc. So now I can simply pull 1 script, tell it the cluster I want stood up - and it will rebuild each of them in about 2-3 minutes... ---------------------------------------------------------------------- HighMarch · 2023-12-19 19:29 I try the K.I.S.S methodology: I want my environment to be as simple to manage/maintain as possible. I've avoided homebrewed tools since I rarely have a need that's niche enough to justify them, and for everything else I either create backups, or can replace it with minimal hassle. A lot of stuff I have, I just expect I'll wind up replacing/rebuilding in time, and expect to do that every 6-24 months, depending upon what it is. So importing backups or just replacing with a later version, and reconfiguring are expected chores. Probably equally labor-intensive, if I'm being honest, but it's the approach that feels simpler to me. ---------------------------------------------------------------------- NeoJackOfBlades · 2023-12-19 19:37 I use docker-compose "managed" by ansible. I use ansible to install packages, create users, folders, samba share and up/down composes services ---------------------------------------------------------------------- ebrious · 2023-12-19 20:15 I use both ansible and docker compose. My goals are to make it easy to completely start fresh on new metal if something breaks and to enshrine how I did something in case I want to refine or replicate later. The gist of it is: 1. One role per service (e.g., immich, or paperless_ngx). Most of my services are deployed using docker (compose). So the ansible role uses a [template](https://docs.ansible.com/ansible/latest/collections/ansib le/builtin/template_module.html) to produce a docker-compose.yml and [ handlers](https://docs.ansible.com/ansible/latest/playbook_guide/playb ooks_handlers.html) to start/stop the service via docker compose. 2. A "base" role that all roles have as a dependency. This mostly just makes sure they all have an base system with docker, some other dependencies, and a single parent directory `/mnt/homesever` 3. Horizontal roles that some services use as a dependency. Things like certbot or a nginx reverse proxy to add a layer of abstraction for configuring web access. 4. Other horizontal roles that depend on my services. Right now, this is just snapshotting with BTRFS and backing up using restic. I'm pretty happy ---------------------------------------------------------------------- themonstersarecoming · 2023-12-19 20:28 No shade on Ansible, it's one of the best software I've ever used. It made me wish I was managing a bunch of complicated infrastructure for multiple clients, because I'm pretty sure I could do it with Ansible. It gives you superpowers and integrates with everything. In the setup I had, I was having it dynamically make docker-compose.yml files with variables, setting up rsync, creating a docker swarm, managing tailscale and firewall rules, and tons of other stuff. It was super cool! ---------------------------------------------------------------------- driversti · 2023-12-20 07:31 Thank you for your reply. Can you please clarify how you use Ansible for managing docker-compose, and up/down services? ---------------------------------------------------------------------- driversti · 2023-12-20 07:34 Yep, I can be considered as a pretty simple approach. At the same time, I'm too lazy to do the majority of work manually. If something can be automated - automate it. And I agree with you: as many as possible actions should be simplified. Even automation. ---------------------------------------------------------------------- naik83 · 2023-12-20 07:39 I basically have a repo where I store Ansible playbooks and docker- compose projects. In the docker directory, each project is in a separated folder, a project being a coherent set of containers related to a service (eg. immich and its database, traefik and its internal network, homeassistant and its database, etc.). The Ansible playbook applies basic system configurations and hardening to the OS, then copies required docker-compose projects to the target and start them. The "tuning" is made through Ansible host_vars and vaults, where I declare several things, including the list of docker projects to deploy on the target, the host name, or the VPN config. I only have to use a few Ansible commands to deploy, which I wrote down in a file aside. I'm comfortable with docker-compose and I'm currently using it regularly, so no retroengineering required when trying to recall what a project does. To some extent, ChatGPT helped me a little to setup Ansible and understand some of its principles. ---------------------------------------------------------------------- driversti · 2023-12-20 08:17 Do I understand correctly that you run your services in separate VMs? ---------------------------------------------------------------------- driversti · 2023-12-20 08:34 Thank you for your response. Actually, I don't want to create a backup in Portainer and restore it on my mini PC. The installation on my RPI was kinda first try. I do not do well some things. Also, some configurations are not clear to me and I would like to dig deeper and set up services from scratch using docker-compose, as this is a tool I know more or less well. As I mentioned in another comment, it seems to me Ansible is not a proper tool for these particular needs. I guess I will create a separate repo with folders per service/stack and will run what I really need. Portainer is going to be a tool for monitoring what is running and reading logs. ---------------------------------------------------------------------- driversti · 2023-12-20 08:35 I'm not familiar with terraform and help yet. I guess it will be an overkill here. But thank you for your approach. ---------------------------------------------------------------------- driversti · 2023-12-20 08:36 This is very interesting! Thank you a lot for the repository 🤝 ---------------------------------------------------------------------- driversti · 2023-12-20 08:45 Wow! So interesting thoughts! I agree with you. I have a few playbooks for setting up dozens of Alpine Linux for my pet projects, but when it comes to self-hosted services (Nextcloud, \*arrs, and so on), using Ansible indeed can be an overkill. I need a simple and robust approach. And multiple docker-compose files are what I need, I guess. ---------------------------------------------------------------------- naik83 · 2023-12-20 08:52 I do have several VMs, but most of them run multiple services. I have several geographic locations where I have VMs and bare-metal hosts running various amounts of services each. For my current needs, I have "low-power" hosts (Optiplex, i5) on each location. They run a single VM where my "always-on", low-demanding services are deployed (pihole, traefik, homeassistant, physical gateways,...). I also have some other hosts with other hardware specs where I deploy services with more specific needs, like a "large" storage space, more computing power, etc. As long as I can, I set up my services on the same low-power host. I don't require any redundancy here, none of these services are absolutely vital, and they can be redeployed from scratch easily using Ansible and Docker. At some point I will (probably) setup some replication tasks in between my locations, but it would require some hardware upgrades. ---------------------------------------------------------------------- driversti · 2023-12-20 08:52 Thanks for your ideas. I will start with option 1 but will keep Portainer as it is a great tool for visualization. In the future, when I am comfortable with restoring my services whenever and wherever I need them, I will try to migrate to k8s (k3s). Not sure if it makes sense with just a mini PC, though. Ansible is a great tool for some of my needs. Setting up/updating dozens of services is so painful and time-consuming. And it is hard to keep all my services in the same state. Ansible gives me that ability and I am falling in love with it. No more manually setting up new servers. ---------------------------------------------------------------------- driversti · 2023-12-20 09:12 Oh, now I understand. Quite a complex setup :D ---------------------------------------------------------------------- driversti · 2023-12-20 09:16 Yep, good point. I will start with docker-compose files and a repo for them. Thanks :) ---------------------------------------------------------------------- driversti · 2023-12-20 09:16 Agree. This simplicity is exactly what I need right now. Thank you! ---------------------------------------------------------------------- driversti · 2023-12-20 09:18 I will go this way too, but with GitHub. I don't feel Gitea will bring me more benefits than GitHub. ---------------------------------------------------------------------- driversti · 2023-12-20 09:32 I have one mini PC right now and 2 RPI 4. I'm going to use Ansible to keep my machines in the state I expect and store Ansible stuff in a repo. This is actually what I do with my pet projects on one of my RPIs. All current services from my RPI 4 I will move to a mini PC and here dozens of docker-compose files in a single repo will suit my needs. Don't see how can I use Ansible here, to be honest. ---------------------------------------------------------------------- driversti · 2023-12-20 09:33 How many hardware servers do you use for k8s? Do you have one powerful unit and many VMs? ---------------------------------------------------------------------- driversti · 2023-12-20 09:38 Hm, maybe this can be a good idea. Keep in Ansible the state of my server I want and let it clone the repo, and do the rest to prepare my machine for running composes. ---------------------------------------------------------------------- driversti · 2023-12-20 09:39 This is the main idea of the thread. I wanted to know how other people manage their homelab in code. ---------------------------------------------------------------------- driversti · 2023-12-20 09:40 Agree. I came to the same conclusion after reading all Redditors' answers. ---------------------------------------------------------------------- driversti · 2023-12-20 09:45 I will use multiple docker-compose files to configure my services and stacks. I consider a "main" docker-compose, that will prepare networks, volumes, maybe some other stuff. Then in sub-folders (named as a service or stack), I will store docker-compose files needed for particular services or stacks of them. For example, if I want to set up Jellyfin, Sonarr, Radarr and so on, it is a good idea to create a docker-compose with all these services, volumes, networking and isolate them from other services. The "main" docker-compose will provide a single point for communication between services and stacks if needed (e.g., Traefic, or NPM, or PiHole) ---------------------------------------------------------------------- driversti · 2023-12-20 09:46 Thanks. I'm considering a similar approach. ---------------------------------------------------------------------- Bill_Guarnere · 2023-12-20 09:52 That's fine, Portainer or Rancher are good tools to visualize or check things, but I strongly suggest to avoid any changes with those tools for the reasons I explained before. Regarding Ansible, maybe I was too harsh in my previous post, I didn't mean that Ansible is useless at all, I was referring to your objective. I'm not a big fan of it because imho it's useful only for a small amount of people that have to manage hundreds if not thousands of hosts. For the rest of the world imho it's something you can easily do with a few bash lines or with the tools the OS already provide (think about updates, why not use unattended upgrades or yum-cron/dnf-automatic? ---------------------------------------------------------------------- ZaxLofful · 2023-12-20 09:56 “Self-hosted” ---------------------------------------------------------------------- driversti · 2023-12-20 10:16 Currently I don't feel I need self-hosting it :) ---------------------------------------------------------------------- driversti · 2023-12-20 10:18 Agree with both statements. I don't want to manage my services with Portainer, only read-only mode. Ansible I use to manage dozens of my micro servers, but in terms of this topic it is not a tool I need. To set up the host - yes, for managing my services - no. :) ---------------------------------------------------------------------- NeoJackOfBlades · 2023-12-20 17:50 I have an ansible variable listing all my services to be configured, each item has its own task: nexcloud has nextcloud.yml that configures the container appropriately. ​ docker_services: [ redis, plex, nextcloud, gitea, heimdall, paperless, immich, nginxproxymanager, #adguardhome, portainer ] ​ I had a main playbook with setup\_docker\_services.yml task defined: tasks: - import_tasks: tasks/essencial_task.yml - import_tasks: tasks/setup_restic_task.yml - import_tasks: tasks/setup_zfs_task.yml - import_tasks: tasks/setup_volumes_task.yml - import_tasks: tasks/setup_rsync_task.yml - import_tasks: tasks/setup_users_task.yml - import_tasks: tasks/samba/setup_samba_task.yml - import_tasks: tasks/setup_docker_task.yml - import_tasks: tasks/services/setup_docker_services.yml - import_tasks: tasks/scheduler/setup_scheduler_tasks.yml Inside setup\_docker\_services.yml, for each service, calling its task file dynamically, I configure database, directories, configuration files to be placed on volumes, etc. After that, I deploy the services via Ansible: - name: deploy Docker Compose stack community.docker.docker\_compose: project\_src: "{{ docker.compose\_dir }}" files: "{{services\_files}}" It's difficult to copy the code here because it's breaking haha If I no longer want a service, I comment or remove the variable that represents the service and the next time I run Ansible, the script will stop the application. I still have a docker-compose.yml for each service, but this way, ansible automatically copies it to my server via ssh and is in charge of doing the up/down ---------------------------------------------------------------------- Varnish6588 · 2023-12-20 20:45 In your case, the Docker provider would be the equivalent to Docker compose plus ansible together https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs ---------------------------------------------------------------------- tenekev · 2023-12-21 01:36 What people say when they don't know how to leverage Portainer. On the contrary, Portainer adds a fuck-ton of functionality and connectivity. You have a whole stack lifecycle API at your disposal. Combined with a version control service, it's all you can ever need for Docker. You can literally never touch Portainer, if you don't want to. ---------------------------------------------------------------------- driversti · 2023-12-21 14:52 Thanks. Will check it out 🤗 ---------------------------------------------------------------------- driversti · 2023-12-21 14:55 Wow, does this mean that you define your services in appropriate docker-compose.yml files (and keep them up-to-date), and use Ansible for copying them to containers/VMs and execute? ---------------------------------------------------------------------- NeoJackOfBlades · 2023-12-21 22:02 exactly. I found it simpler than letting Ansible generate them dynamically ---------------------------------------------------------------------- driversti · 2023-12-22 17:42 Awesome! ---------------------------------------------------------------------- Then-Quiet-5011 · 2023-12-24 18:33 3 nodes, Intel i3/i5 + 16GB Ram each ---------------------------------------------------------------------- driversti · 2023-12-24 19:05 Impressive. Are they located at your home? ---------------------------------------------------------------------- Then-Quiet-5011 · 2023-12-24 21:50 Yes, i have small rack cabinet