docker-compose question reddit / selfhosted ---------------------------------------------------------------------- opcivy · 2023-12-15 18:40 is there a reason to use a separate compose file for each container, or should i just put them all in the same file? i keep finding services that have their installation instructions as "download this compose file and run it" instead of adding it to an existing one, so i figured i'd check if there was a reason ---------------------------------------------------------------------- thekrautboy · 2023-12-15 18:45 Again copy/pasting from a previous comment of mine: ___ **Stuffing everything into on giant compose file is a nightmare and removes a lot of the advantages that compose actually provides. Unfortunately there must be some Youtuber or blogger out there that teaches this approach, youre not the first person to do this.** **See the end for a TL;DR** My suggestion, and i believe this is common best practice: **Use a seperate folder and one compose for each stack, as in a group of services that absolutely need to be together. Keep bind mounts like data, config, db etc folders for the services also within those folders.** #Structure As a example of folder structure: mydockerstuff/ ├── immich/ │ ├── cache/ │ ├── docker-compose.yml (contains only immich itself with its own postgres, typesense and redis) │ ├── postgres/ │ ├── upload/ │ ├── typesense/ │ ├── redis/ ├── outlinewiki/ │ ├── data/ │ ├── docker-compose.yml (contains only outline itself with its own postgres and redis) │ ├── postgres/ │ ├── redis/ ├── portainer/ │ ├── data/ │ └── docker-compose.yml (contains only portainer itself) And each stack (folder) also has their own **Docker network** defined inside the compose, so the postgres db from immich doesnt get to be in the same network as the redis from outline for example. When using a **reverse proxy**, only the frontend of a stack that requires to be proxied becomes member of the proxy network. Everything else of the stack stays **seperate**. There is no need for the db container to be part of the proxy network, and the db also doesnt need any ports mapped to the Docker host. Of course a container like Portainer as example that doesnt have any other containers to rely on doesnt *need* its own network, when its being proxied its enough to make it a member of the proxy network and thats it. #One-db-for-all or One-db-for-each? This also speaks to the question of "**one db container that all services use together**" or "**one separate db container for each service**". It might seem logical to run one central MySQL db container and then have all your services connect to that and use it, to keep it simple and save on RAM and CPU. However people who go with that approach are basically guaranteed to run into problems longterm when, especially with databases as example, different versions are required. For example you are running Postgres 14 for all your services, and it works great. Then a update for one service comes up and that update now requires a major version upgrade to Postgres 15. What do you do? You can either spend a lot of time researching if all your other services are also compatible with the higher version of the db, how to migrate each of them, run backups before hand and then risk the upgrade for everything. Or you can start up a second container, keep the old v14 and run a new v15 in addition. Over time that wont just be two containers with that approach, you will end up with a handful of Postgres containers, another handful of MySQL, some InfluxDB and etc. Soon you will realize that sticking each service with its own dedicated db is much simpler to keep track of. And when a service requires a db upgrade, you can just upgrade that one single db and everything else is not impacted by that at all. Also, especially for db containers you should pin them to a specific version in your compose files, and do not use *latest* for example. Databases are almost always finnicky with major version upgrades, so i would absolutely not recommend to have them use *latest* as tag, and even worse, auto-update them with tools like Watchtower. If you deploy a new project, and they recommend version X of a specific db, then you should stick to that and keep using that version, until the maintainers of that project recommend the next major version of the db. It may sound like a lot of extra work, but it isnt, and it will save you a lot of headache in the future. The **CPU usage** impact is near nothing by running multiple db containers, because they almost only cause load when they are being used. And then it doesnt make much difference if that load is on one single db or spread out over multiple smaller ones. The workload stays the same. There is of course some overhead, but i would claim its extremely minimal for typical homelab setups. **RAM usage** i could see more of a reason to go for a single main db instead of one-per-service, a naked MariaDB by itself uses something like 100MB, so when someone is using a host with very limited RAM available (Raspi3 with 1GB as example) i could understand that they dont want to run 3x MariaDB for a total of 300MB just for those. They would "save" 200MB by running a single db instead. Of course they would still need to face all the disadvantages i mentioned already by using a single db for multiple services. Now back to the original topic, to manage these stacks its very simple. Some examples: cd immich docker compose up -d cd .. cd portainer docker compose pull && docker compose down && docker compose up -d Or docker compose --f immich/docker-compose.yml up -d docker compose --f portainer/docker-compose.yml pull && docker compose --f portainer/docker-compose.yml down && docker compose --f portainer/docker-compose.yml up -d It helps to define some often used docker compose commands as shorter alias in `.bashrc` (or whatever you use). So `docker compose up -d` becomes just `dcupd` for example. You can also still use [Portainer](https://www.portainer.io/) to manage stacks that are deployed manually, you only miss out on a few specific features. Compare yourself and make your own choice. But imo using **only** Portainer leads to not learning actual Docker and when the time comes when Portainer cannot handle something, people end up clueless. (For those completely new to Portainer, you can grab a [free license of the Business Edition](https://www.portainer.io/take-3) for 3 nodes for free, and it offers a few extra features). #TL;DR Keep every group of services as a separate stack and folders, follow K.I.S.S, you will be thankful for it longterm ---------------------------------------------------------------------- opcivy · 2023-12-15 19:06 got it, i'll sort through my stuff and get it more organized soon, i see how this would be easier to keep track of \^\^ thank you! ---------------------------------------------------------------------- PaleMongo · 2023-12-15 20:53 Only knit togther those that are integrally dependent upon one another... for example, running bookstack on mariadb. If you make an all-inclusive compose file, you'll need to restart every container when you need to update only one. And networks... O my... ---------------------------------------------------------------------- thekrautboy · 2023-12-15 21:16 > If you make an all-inclusive compose file, you'll need to restart every container when you need to update only one Technically they could do `docker compose up -d ` to only up that specific service out of a larger compose file, but still this gets needlessly complicated and plenty of other things dont work as compose was meant to be, so its a bad idea. ---------------------------------------------------------------------- Untreated3763 · 2023-12-15 23:27 This is the best summary of docker organization for self-hosted set up that I've been able to find. Thanks for sharing. ---------------------------------------------------------------------- thekrautboy · 2023-12-15 23:29 Youre welcome, glad i could help. And thats the reason i copy/paste that comment. Maybe i should take the time some day to further refine it, dont know, but i think it gets most things across and explains them okay. ---------------------------------------------------------------------- PaleMongo · 2023-12-16 02:54 Yep.