How do you manage multiple dockers: multiple compose ymls, one super long one with everything in it, individual txt files containing the docker run string, etc? reddit / selfhosted ---------------------------------------------------------------------- SpinCharm · 2023-12-13 23:56 I’ll currently using one compose yml file per container then use separate ‘docker compose -f up -d’ commands to recreate each one as needed. But that seems slightly awkward and perhaps there’s a better way. And every time I use that approach it returns a warning about orphaned objects even though they aren’t, so I just ignore that. How do you manage yours? ---------------------------------------------------------------------- Rocketsx12 · 2023-12-14 00:06 One compose file per stack. A stack might be one container or multiple related containers such as app + db. You're probably getting the orphan warning because of multiple compose files in the same directory, there are a few solutions to that which your can Google. ---------------------------------------------------------------------- SillyLilBear · 2023-12-14 00:16 This, just to add, use one folder per stack. I use \~/stacks/homepage/docker-compose.yml. If you use portainer, it will manage it all for you, just create a new stack for each group of apps that work together (nextcloud & nextcloud-postgres for example). ---------------------------------------------------------------------- labm0nkeys · 2023-12-14 00:20 I use extends and include. Everything is explained here https://github.com/labmonkey/docker-compose-project-example ---------------------------------------------------------------------- Square_Lawfulness_33 · 2023-12-14 00:40 Give Dockge it’s from the same developer of uptime-kuma. ---------------------------------------------------------------------- LastTreestar · 2023-12-14 00:50 Portainer ---------------------------------------------------------------------- mattman0123 · 2023-12-14 00:53 Portainer Multiple stacks/docker compose files ---------------------------------------------------------------------- SpinCharm · 2023-12-14 00:55 I use portainer, but just for starting stopping deleting containers. Is there a way to use it to handle docker compose files too? ---------------------------------------------------------------------- user_luca · 2023-12-14 01:29 I am using a GitHub repo where i put all my compose Files. Each compose file holds exactly one Stack (one for my Proxy with authentication (Traefik + authelia), one for my gamservers (pteridactyl Panel + db), and so on). Each compose file is also in its own directory so that i can mount config Files specific for the Stack. If Something in this whole construct changes, GitHub actions is triggerd and deploys it via ansible to my docker server. ---------------------------------------------------------------------- user_luca · 2023-12-14 01:35 The main point of my Idea is that i can easily replicate my infrastructure if something goes wrong or if i need to change my Provider. ---------------------------------------------------------------------- maudat · 2023-12-14 01:43 One folder per project, all files related to that project stays there (docker-compose.yml, config file, env file, etc). For now it does the trick. ---------------------------------------------------------------------- weischin · 2023-12-14 01:47 Docker compose is call Stack in Portainer. ---------------------------------------------------------------------- prabirshrestha · 2023-12-14 01:58 Was using ansible but now migrating to nomad although it internally uses docker under the hood. Been adding features to https://github.com/jonasvinther/nomad-gitops-operator. It is now at a point where I can rely 100% on it. ---------------------------------------------------------------------- revereddesecration · 2023-12-14 02:18 I didn’t know this was an option. Cool. Nice repo, thanks for setting it up. ---------------------------------------------------------------------- CactusBoyScout · 2023-12-14 02:20 Is there a way to export an up-to-date compose file if I’ve made changes to a container after deploying it via Portainer Stacks? ---------------------------------------------------------------------- mattman0123 · 2023-12-14 02:22 The compose.yml is only for deploying the stack. If you modify the container itself after the deploy it won't result back to the compose. So I would assume you need to edit and update the compose file yourself. ---------------------------------------------------------------------- CactusBoyScout · 2023-12-14 02:23 That’s what I thought. I wish there was a way for Portainer to update the stack itself when I do something like add a new bind mount. But I’ve been trying to keep track of them myself manually. ---------------------------------------------------------------------- mattman0123 · 2023-12-14 02:24 Agreed there are times I wish it could. ---------------------------------------------------------------------- Spider-One · 2023-12-14 02:41 Podman with systemd units. Can go a few weeks between looking at any of it. Podman autoupdate once a week. Ideally I'll set up a notification for when a health check fails or such but it's been working and I'm not touching it lol. ---------------------------------------------------------------------- GSpanFan · 2023-12-14 02:44 I'm not positive that it supports Portainer Stacks, but you may want to check out docker autocompose which will basically create a yaml file from a running container. I think you can set it do do it for a group of containers. Word of caution, it seems like it saves every setting, so the yaml file is going to be much longer than one you'd have written yourself and may benefit from some pruning. ---------------------------------------------------------------------- CactusBoyScout · 2023-12-14 02:45 That sounds like exactly what I want. I just keep changing network settings, env, labels, etc and it’s annoying to keep track. ---------------------------------------------------------------------- yakk0 · 2023-12-14 03:25 I edit the compose files in portainer after deploying and then redeploy. It works great. ---------------------------------------------------------------------- mattman0123 · 2023-12-14 03:28 I'm intrigued now!! ---------------------------------------------------------------------- ScratchinCommander · 2023-12-14 04:29 What about data for containers? ---------------------------------------------------------------------- neumaticc · 2023-12-14 05:33 interesting - link: https://dockge.kuma.pet/ ---------------------------------------------------------------------- steveiliop56 · 2023-12-14 06:45 I used to just have them all in folders and stuff but I decided to switch to something more automated like runtipi. ---------------------------------------------------------------------- TheRealSeeThruHead · 2023-12-14 06:48 I’m currently doing portainer stacks manually. But I think I will setup a got repo of stacks as separate yaml files and have them deploy as part of a ci step ---------------------------------------------------------------------- Traditional_Wafer_20 · 2023-12-14 06:56 Is it a Portainer-like project? ---------------------------------------------------------------------- DeusExMaChino · 2023-12-14 07:14 I actually just wrote this to only trigger when a docker-compose.yml is committed. ``` name: push-to-docker run-name: ${{ github.actor }} pushed config change to Kevflix on: push: branches: - main paths: - '*docker-compose.yml' jobs: update-docker-configs: runs-on: [self-hosted, Linux, X64] steps: - name: Check out the repository to the runner uses: actions/checkout@v4 - name: Get changed yml files uses: tj-actions/changed-files@v40 id: changed-yml-files with: files: | *docker-compose.yml - name: If changed docker compose yml files, update stack if: steps.changed-yml-files.outputs.any_changed == 'true' run: | for file in ${{ steps.changed-yml- files.outputs.all_changed_files }}; do echo "$file was changed" docker-compose -f $file pull docker-compose -f $file up -d --remove-orphans done docker image prune -a -f ``` ---------------------------------------------------------------------- maomaocake · 2023-12-14 07:53 one compose file per stack and all the files are on gitlab ---------------------------------------------------------------------- Krax0x · 2023-12-14 08:01 Personally I stack apps with same purpose in one compose file. Example - media server \*arr stack - jellyfin, sonarr, radarr, bazarr, prowlarr, qbittorrent (I know it's not arr app) etc. Another stack for monitoring and so on. I find it easier to manage. Also portainer is great for docker UI. ---------------------------------------------------------------------- wowkise · 2023-12-14 08:40 Personally i prefer single container per docker-compose file, i don’t run multiple database containers i usually have one of each required type and it’s shared between all containers. There are some containers that wont work without other container those get bundled for example a VPN container and a proxy service. I use simple python script to manage control of starting stopping etc via systemd. ---------------------------------------------------------------------- KD_done · 2023-12-14 09:09 When it comes to compose files? sure. But that's where the similarities end.. Portainer goes far beyond just compose yml management (think volume, image and container mangement, kbn and swar and the sorts). It has functions for it's features, and options for it's selections in functions ;) ---------------------------------------------------------------------- FuriousRageSE · 2023-12-14 09:11 >This, just to add, use one folder per stack. I use \~/stacks/homepage/docker-compose.yml. A question about this file. Do docker scan all sub directories for this file? Because i dont really get it how else "docker up -" (or what the command is to up it), and it seems to start everything. ---------------------------------------------------------------------- FuriousRageSE · 2023-12-14 09:16 >Dockge Is it a "portainer replacement" (or close to)? Yesterday i had alot of issues getting portainer working (again), might look into this one instead. ---------------------------------------------------------------------- FuriousRageSE · 2023-12-14 09:23 Yesterday i was thinking in settig something like this up because i reinstalled the VPS a few times trying to setup, and had to reset because im still learning docker and stuff around it. ​ I thought it would been nice to perhaps use github to store done/working files at, but perhaps use brnaches, so i could "only" check out this or that container file. ---------------------------------------------------------------------- clearlybaffled · 2023-12-14 09:41 I moved to kubernetes with argocd as a gitops solution ---------------------------------------------------------------------- ibnunowshad · 2023-12-14 10:04 Quick question. Is it worth to run portainer in a docker itself or standalone cluster across different zone? Something similar to kubernetes cluster across a region and availability zones ---------------------------------------------------------------------- driesmp · 2023-12-14 10:05 I am trying includes but doesn’t work … ---------------------------------------------------------------------- FuriousRageSE · 2023-12-14 10:10 Until yesterday before bed, i always ran portainer in their docker. Maybe it would be easier for me, to run it on the host. ---------------------------------------------------------------------- LastTreestar · 2023-12-14 10:12 The control is limited when the stack is created outside portainer. I asked on the portainersub what the impact was, and I don't recall anything substantial. Iinstall docker, then portainer, then I don't touch docker again. ---------------------------------------------------------------------- thelittlewhite · 2023-12-14 10:30 You can use a simple script to start /stop / restart all your containers. Just change the $dir to your liking. https://github.com/thatg33khub/dockarr_stack/blob/master/start-stop- restart_all_containers.sh ---------------------------------------------------------------------- Square_Lawfulness_33 · 2023-12-14 11:04 Is a docker compose manager that simpler than portainer. ---------------------------------------------------------------------- FuriousRageSE · 2023-12-14 11:11 Then i'll definately will be checking it out when i get home later today. Portainer was nice and all, but it felt like it didn't "match" with me ---------------------------------------------------------------------- dercavendar · 2023-12-14 11:56 When you run docker compose up it scans the folder you are in for a file called docker-compose.yml (or compose.yml or maybe a couple of other examples). If it doesn’t find one it throws an error. You can also pass a file path to it like docker compose up -d /home/containers/compose.yml. This method would allow you to name the file whatever you want and to start your container from anywhere in your file system, but you have to know where the yml file you need is. ---------------------------------------------------------------------- FuriousRageSE · 2023-12-14 12:30 Ah oki, makes more sense. ---------------------------------------------------------------------- TheNoah · 2023-12-14 12:41 Depends on what version you're on but I use a couple files and use include to pull in the other ones. So all I have to do is run docker- compose up -d ---------------------------------------------------------------------- user_luca · 2023-12-14 12:54 For me each Stack hast its own action file so that for some stacks i can Automaten some other things with Ansible. But automating those deployments completly with one action file also Sounds interesting. ---------------------------------------------------------------------- mauvehead · 2023-12-14 12:56 Portainer. https://github.com/portainer/portainer ---------------------------------------------------------------------- user_luca · 2023-12-14 12:58 My setup is not complete yet and likely never will be complete. But i plan on automatically backing up data volumes to my NAS that i am going to build in some time. ---------------------------------------------------------------------- user_luca · 2023-12-14 13:05 For me it is the perfekt solution, because i can mess up everything and be still able to get every back up again in like 5 min. (requiers all Data volumes to be persistent and saved somewhere) In the last few months I experimented a lot and IT was never that easy to restore my services after i messed up. ---------------------------------------------------------------------- Bamny · 2023-12-14 13:07 I’ve got folders in /opt/ for each container to use for volume mounts. I use Portainer to manage each container, and I spin them up via stacks within Portainer. ---------------------------------------------------------------------- FuriousRageSE · 2023-12-14 13:14 >For me it is the perfekt solution, because i can mess up everything and be still able to get every back up again in like 5 min. > > > > (requiers all Data volumes to be persistent and saved somewhere) This is my problem, i havent put my self into much of the docker and config, how/where to stora the permanent data, and how to be able to back it up etc. ​ >In the last few months I experimented a lot and IT was never that easy to restore my services after i messed up. I should probably make a github:ed bash script that makes all the zsh settings i need/want because only yesterday, i had to re-do all the zsh shell changing stuff i wanted to do, because i kept messing stuff up.. or stuff stopped working as expected.. old config files left over somewhere etc. ---------------------------------------------------------------------- willquill · 2023-12-14 13:25 Definitely don’t do one compose file per service! Use one compose file per host, generally. I have a “homelab” git repo that spins up all of my homelab services with one docker compose file. I push to GitHub, and that git repo gets cloned to my main server for homelab services. I have a “media” git repo with a single docker compose file for all media services, and it is cloned to and executed on my media host. Another example: I just upgraded the CPU/mobo in my NAS and added a PCI-E card in it that will be utilized by my Frigate docker service in my “frigate docker compose file/git repo”. In this instance, Frigate is the only docker service running on my NAS, so it is alone in the compose file. ---------------------------------------------------------------------- willquill · 2023-12-14 13:37 Do you keep your docker compose file and supporting files in a git repo? I found this to be extremely useful. I just use GitHub. I have some public and some private. This way, I will create a branch called “add-traefik”, add the Traefik service to my compose file, git commit and git push to GitHub, and then I create a Pull Request so it neatly shows my diff - the changes between pre-Traefik and post-Traefik. Even after merging to the main branch and deleting the add-traefik branch, I have a history of my PRs in GitHub. Also, I can use “git checkout” to checkout ANY previous git commit, i.e. any “state” of my compose file. This makes it super easy rollback to a previous version of my compose file - any previous version that I committed. Also, if you work in IT, or even if you just work with files as a hobby, learning git is incredibly useful! ---------------------------------------------------------------------- sirquack0 · 2023-12-14 13:57 Synology uses a old version so it might be related ---------------------------------------------------------------------- MadeWithPat · 2023-12-14 13:59 I do the folder-per-app with some ansible to manage. I have separate roles defined in an ansible repo where each one creates necessary dirs on my app server, copies a compose template and env file / handles substitution, and starts the service(s) ---------------------------------------------------------------------- CorporateDirtbag · 2023-12-14 14:15 By not using it in the first place. ---------------------------------------------------------------------- labm0nkeys · 2023-12-14 14:44 Try to update to latest docker. Although as I remember some recent version had issues with extends. Can't help much without more details. ---------------------------------------------------------------------- driesmp · 2023-12-14 14:45 What does your include directive look like in your file? And what does the included file look like? ---------------------------------------------------------------------- CactusBoyScout · 2023-12-14 14:51 I don’t know how to do any of this but sounds interesting. I just keep my compose files in a notes app. ---------------------------------------------------------------------- SillyLilBear · 2023-12-14 15:05 You run docker up from that folder so it knows which one you are working with. ---------------------------------------------------------------------- rasg · 2023-12-14 16:56 i have one yaml per stack (arr.yml, plex.yml, jelly,yml, portal.yml, etc...) and one script to quickstart everything #!/bin/bash SCRIPTPATH=$(readlink -f "$0") SCRIPTDIR=$(dirname "$SCRIPTPATH") for f in $SCRIPTDIR/compose/*.yml ; do n=$(basename $f .yml) docker stack deploy --prune -c $f $n done sleep 1 docker system prune -f ---------------------------------------------------------------------- policarp0 · 2023-12-14 17:37 I use podman in rootless mode managed with systemd units. Every service has his user. I use bindfs to mount folders in the home directory with the correct permissions. ---------------------------------------------------------------------- Clanktron · 2023-12-14 18:04 can we stop calling containers “dockers” Jesus ---------------------------------------------------------------------- RedKomrad · 2023-12-14 19:16 I don’t. I let Rancher k3s manage them by running them as TrueNAS custom apps. ---------------------------------------------------------------------- Biog0d · 2023-12-14 21:32 One solution you might like if you’re looking for compose management is dockge. It can be set to monitor a shared directory mounted as a bind, for compose files etc. it can also convert docker run commands into compose files. I run it as a container and actually don’t use it since I use vscode for everything and upload to my nodes that way but I see the use case for you maybe