how to save my containers guide reddit / selfhosted ---------------------------------------------------------------------- geezcustard · 2023-10-11 17:30 hello :) my docker containers number is increasing and I was thinking which should be the best way to save all the configurations, descriptions and the processes needed to put all the containers up and running. since now I was using a simple text file, since I had not so much to remember, but the configuration is becoming more and more complex. I was thinking maybe at a kanban board. what would you recommend? (I'm not a programmer, that's also why I'm asking) :) ---------------------------------------------------------------------- NekoLuka · 2023-10-11 17:37 I personally have a github repo with a folder for each deployment, and the files needed for deployment with a docker-compose.yml, and comments in the files for what they do. ---------------------------------------------------------------------- hardypart · 2023-10-11 17:42 BookStack is your friend! ---------------------------------------------------------------------- finikwashere · 2023-10-11 18:35 With ports, access tokens to the databases and your usernames in a public github repo? Brave ---------------------------------------------------------------------- joecool42069 · 2023-10-11 18:45 That should be in a .env. And then exclude .env from your git files. ---------------------------------------------------------------------- finikwashere · 2023-10-11 18:48 This is clear, but losing env file is causes sometimes a lot of work to recover everything ---------------------------------------------------------------------- joecool42069 · 2023-10-11 18:49 I keep a copy in keepass. Cause you know.. passwords. ---------------------------------------------------------------------- geezcustard · 2023-10-11 19:02 >BookStack thanks, I didn't know it ---------------------------------------------------------------------- thekrautboy · 2023-10-11 19:10 * Use Docker Compose * Write important things as comments into the compose files * Backup the compose files to wherever you like * Done Many people use something like a selfhosted Gitea to store their compose files as code there, plus you get versioning. Backup the persisent data of containers with whatever backup tools you like, this has been asked a million times already here. Backup database containers ideally by doing a db dump, depending on the type of db. Then backup that dump file. For storing code snippets and such, [snippet- box](https://github.com/pawelmalak/snippet-box) and [memos](https://usememos.com/) are useful. ---------------------------------------------------------------------- Reasonable-Ladder300 · 2023-10-11 19:34 I use infisical for secret management. Also offers option your git repo for secret leaks ---------------------------------------------------------------------- Minituff · 2023-10-11 19:36 I use Portainer with Docker Compose stacks that are pulled from a private GitHub repo. ---------------------------------------------------------------------- NekoLuka · 2023-10-11 19:53 It has the files where the passwords are stored in, edit the password after the git clone and docker compose up, that's all. Of course the same procedure for usernames and the likes. ---------------------------------------------------------------------- Mean_Einstein · 2023-10-11 20:36 Ansible in a git repo ---------------------------------------------------------------------- Reverent · 2023-10-11 20:47 Keep all data including db data in subfolders of each compose file spot, use a COW system like btrfs or zfs, then just snapshot away. ---------------------------------------------------------------------- PaleMongo · 2023-10-11 21:10 Bookstack is super... except, I'm not sure if it's the best way to back up docker containers... since it is a docker container app. ---------------------------------------------------------------------- Dalem246 · 2023-10-11 21:28 Hey is there a guide anywhere for storing these variables in a .env file? I am still new to best security methods for config files and yamls, plus I’d like to also use git to backup my containers. ---------------------------------------------------------------------- Spuxilet · 2023-10-11 21:40 Generally it's better to have important data in volumes not in filesystem. ---------------------------------------------------------------------- joecool42069 · 2023-10-11 21:40 you use git for your dockerfile(if you're not just pulling images from docker hub) and docker compose files. Your containers, make sure to backup your volumes, be they mounts or named. As for how to use a .env? https://docs.docker.com/compose/environment-variables/set-environment- variables/ you simply create a .env file in the same directory as your docker- compose.yml and set your variables. example: UID=1000 Then you can use that in your docker compose: UID: ${UID} Then just use .gitignore to git to ignore .env files. Is this the 'correct' way to do things? I dunno, i'm no expert. Docker also has secrets and vaults for storing sensitive data, but I haven't dove into that yet. I just don't want any api keys, passwords, or other sensitive data to upload to github, even if it is a 'private' repo. ---------------------------------------------------------------------- Reverent · 2023-10-11 21:42 What do you think a volume is? it's a bind mount that defaults to `/var/lib/docker`. Only difference is with a bind mount you typically have to pre-seed the permissions on folders where containers are not running as root (which should be most/all of them). ---------------------------------------------------------------------- Dalem246 · 2023-10-11 21:43 Yeah of course I am not securing anything super secret but that .env would be to insecure for. It just seems like a more professional/proper way to do it. Does the .env file matter the name? Or just has to be in the directory any name? ---------------------------------------------------------------------- joecool42069 · 2023-10-11 21:47 literally just ".env". if you really wanted to name it something else, you can... but then you have to reference it manually like docker compose --env-file /path/to/my/superduper.env ---------------------------------------------------------------------- Spuxilet · 2023-10-11 21:54 Volumes have many advantages and are safer and easier to work with. Also they are more flexible. They work both on windows and linux. They can be easily shared between containers. They can be easily backed up and restored. You can interact with them by docker CLI or API. They are isolated from host and managed by docker. You only use it's name in compose files not full path like when using bind mount. ---------------------------------------------------------------------- Dalem246 · 2023-10-11 22:05 Ahh I see awesome, thank you for all the help! ---------------------------------------------------------------------- Reverent · 2023-10-11 22:06 Sure, if you don't understand how any of that works, volumes can abstract most of it away. Or if you do understand how that works and care about how you back up your data, you can bind mount it to a specific location to snapshot it appropriately. ---------------------------------------------------------------------- joecool42069 · 2023-10-11 22:07 happy journeys. ---------------------------------------------------------------------- Smiles_OBrien · 2023-10-11 23:16 ***DISCLAIMER: I HAVE NOT TESTED THIS, ONLY RAN IT AND CREATED THE COMPOSE FILE. YOUR MILEAGE MAY VARY:*** I came across this: [https://github.com/Red5d/docker- autocompose](https://github.com/Red5d/docker-autocompose) It takes your existing docker configs and turns them into one docker- compose with existing settings. Can be tweaked if / when you move to a new server. Looks interesting. Per my disclaimer, I haven't actually re-deployed my stack using this compose, but it DOES capture my running configs quite nicely. ---------------------------------------------------------------------- thekrautboy · 2023-10-11 23:39 It makes zero difference tho. Using a volume instead of a bind mount just gives you the advantage that Docker is handling all user permissions for you inside the volume. But the files are stored exactly the same way. ---------------------------------------------------------------------- thekrautboy · 2023-10-11 23:41 Snapshots are popular and easy for db backups yes. But i would argue that still the **ideal** way is to do a db dump. And then maybe a snapshot additionally. ---------------------------------------------------------------------- thekrautboy · 2023-10-11 23:43 But you were talking about "important data better to keep in volumes" which is simply not true. The data is not any safer there then it is in a more simple bind mount. You can of course say that volumes have advantages, and that you like those and prefer them. Thats absolutely fine. But they have nothing to do with "better to have important data" in them. ---------------------------------------------------------------------- thekrautboy · 2023-10-11 23:44 Who said anything about a **public** repo? ---------------------------------------------------------------------- thekrautboy · 2023-10-11 23:56 I see this more of a emergency tool, or helpful if someone started with Docker and used only run commands and now they want to switch over to compose. A one-time thing. Also it would dump all containers into one giant compose file, by default. Which is terrible practice to do (as has been discussed here countless times). So in order to avoid that mess, one would need to manually tell the tool to save each container one-by-one into their own compose, which can be very tedious. Still a option worth trying in a situation where nothing else works. The biggest downside is tho that it doesnt distinguish between stacks and just single containers. Everything is just a single container. So if you initially deployed apache+mongodb as one stack, this would save them as individual apache, plus individual mongodb. You loose the relation between them. Again, in a relative large deployment, this becomes very tedious to fix up later. I can also imagine that there would be some rare scenarios where it isnt able to replicate a stack 100% and some more unique options that were used to deploy a stack or container would get lost in the "backup". It definitely is not a form of a backup of a entire Docker host with all deployed stacks. ---------------------------------------------------------------------- Reverent · 2023-10-11 23:59 "best practice" sure. In reality every popular db on the planet now has journalling, restoring from a snapshot has never failed me in the past. and if shit really hit the fan and somehow the db couldn't recover from the WAL, you'd just step back an hour and try the previous hourly snapshot. ---------------------------------------------------------------------- thekrautboy · 2023-10-12 00:03 Yarp. ---------------------------------------------------------------------- anticronista · 2023-10-12 00:37 Selfhost gitea and save all there with git ---------------------------------------------------------------------- bufandatl · 2023-10-12 05:37 Ansible or any other configuration management. ---------------------------------------------------------------------- hardypart · 2023-10-12 06:25 I think you misread OPs text. > the best way to save all the configurations, descriptions and the processes needed to put all the containers up and running He was using a text file up until now: > since now I was using a simple text file, since I had not so much to remember But I see where you're coming from, the title of OPs post is a little misleading. ---------------------------------------------------------------------- stupv · 2023-10-12 07:18 Easy method? Back up your docker-compose file(s), and put your /config (or equivalents) into locations that are backed up ---------------------------------------------------------------------- Spuxilet · 2023-10-12 07:30 Volumes are managed solely by docker and bind mounts are host dependent. Any service can modify or delete them. It IS safer in volumes. ---------------------------------------------------------------------- sowhatidoit · 2023-10-12 07:32 Can you elaborate? I've been learning how to do this. I'd love to hear how you are doing it. ---------------------------------------------------------------------- thekrautboy · 2023-10-12 07:46 And in your reality its impossible to control who can access the bind mounts? *smh* ---------------------------------------------------------------------- Spuxilet · 2023-10-12 07:47 It's linux one can do anything, but like i said it's safer, easier and more flexible to use volumes then bind mounts. ---------------------------------------------------------------------- thekrautboy · 2023-10-12 07:50 > but like i said it's safer, easier and more flexible to use volumes then bind mounts. Your initial comment was "its better" which is simply not true. Now you keep adding things like "more flexible". Its not safer, its not "better". Goodbye, blocked. ---------------------------------------------------------------------- Spuxilet · 2023-10-12 07:51 Go get a life little boy. ---------------------------------------------------------------------- redoubledit · 2023-10-12 08:26 GitHub? No. Public? No. Ports & Access tokens? No. ---------------------------------------------------------------------- redoubledit · 2023-10-12 08:31 Since the "issue" of secrets/passwords came up in the thread: I store a `.env.example` file with it, too. It is built like my real `.env` file but with all "critical" information removed. That one gets committed to the repo as well, so if I have to (or want to) start over, I know what I have to change. Could be ``` # .env.example PORT=8080 USER=user PASSWORD=password # .env PORT=8426 USER=redoubledit PASSWORD=FHSjre8374fhajwo4748efheh # .gitignore .env* !.env.example ``` ---------------------------------------------------------------------- Sir_Squish · 2023-10-12 09:19 Docker Autocompose does generally work, though it generates really messy output. It includes all of the usually runtime generated comments and tags, and for example if you have CAP\_ADD xyz, it seems to generate a CAP\_REMOVE for every other thing instead of just adding the original one. Try it on a container and see what your output looks like compared with your original compose. It's still handy though. ---------------------------------------------------------------------- TBT_TBT · 2023-10-12 10:21 Also using a private Github Repo. Don't use Portainer with the stacks, start them from the console. That is the "clean" way. ---------------------------------------------------------------------- Direct_Emotion_1079 · 2023-10-12 10:39 I’m using gitea to store all docker compose files and configs. Then I use portainer for container deployment, which uses gitea as a repo. ---------------------------------------------------------------------- FatalV0rt3x · 2023-10-12 11:38 I recently had an issue with my server that caused me to have to format my Ubuntu drive and lost my docker setup. I stupidly didn't keep a log of all the docker configs for each container. I've now built a cron that will export the run commands as shell executables for me, so all I need to do in future is run the .sh files on the server to continue where I left off last. May not be the most secure way, but works for me. [https://www.reddit.com/r/selfhosted/comments/1764naz/export\_docker\_ run\_commands/](https://www.reddit.com/r/selfhosted/comments/1764naz/e xport_docker_run_commands/) ---------------------------------------------------------------------- Smiles_OBrien · 2023-10-12 11:58 Did not realize one compose for everything was bad practice - I'll do some reading on that. ---------------------------------------------------------------------- thekrautboy · 2023-10-12 12:12 From just a few days ago: https://www.reddit.com/r/selfhosted/comments /1738p0z/help_a_beginner_get_started/k41e552/ ---------------------------------------------------------------------- phin586 · 2023-10-12 12:43 Just use something like git-crypt and and a private repo. ---------------------------------------------------------------------- javijuji · 2023-10-12 13:05 I am using obsidian.md for this and everything else homelab related. ---------------------------------------------------------------------- shardul- · 2023-10-12 13:26 I was looking at this YouTube video earlier and found it helpful. It outlines how to backup the containers manually or set up watchtower which will back then up automatically. https://m.youtube.com/watch?v=KfpMUBnmmq4 ---------------------------------------------------------------------- KerkiForza · 2023-10-12 15:06 Or you know, just not upload it to the internet in the first place and take a risk. ---------------------------------------------------------------------- phin586 · 2023-10-12 15:08 not disagreeing with you. just stating there are ways to take precautions. you can use a hook to prevent the upload if its not encrypted. ---------------------------------------------------------------------- Sgt_ZigZag · 2023-10-12 15:56 Remember to also backup your docker volumes (container data). Having a copy of the configuration checked into git is great but many containers rely on persistent data that if lost you'll have to reconfigure the application and will lose your databases, etc. I use autorestic for backing up my volumes. ---------------------------------------------------------------------- NanobugGG · 2023-10-12 17:18 I hope it's okay to use this post for my question, since it's somewhat in line of it. Would it be better to use Docker Swarm and just have some VMs or whatever as "docker hosts" to start and stop containers around different hosts, or what is a good way to do it? ---------------------------------------------------------------------- Mean_Einstein · 2023-10-12 20:17 I wrote an Ansible playbook, with a powerful docker-compose template. I basically copy paste a public docker-compose into an Ansible var file, then I run my role with an extra variable, let's say gitea. The template gets merged with changes from my host vars. So if host A uses another image than host B, I just change the image like so: docker_container_gitea_service_gitea: image: gitea:customlabel I hardly ever change the template once everything is set, still I am able to tweak every little bit of it dynamically. ---------------------------------------------------------------------- sowhatidoit · 2023-10-12 21:04 Thanks for the explanation. From the sounds of it if anything were to happen to your hardware, you can simply just replace it and run the playbook. You can be up and running in no time! In essense, instead of making configuration changes directly to your enviornment - you do it through the playbook? That is so freakin cool! ---------------------------------------------------------------------- Mean_Einstein · 2023-10-13 06:12 Yes, I manage all my servers through Ansible. If a VM is misconfigured or broken or whatever, I can just install a new one, let Ansible run over it, restore my back for /data and that's it. From a freshly installed OS to the same state as it was before it just takes a couple of minutes. In regards to containers, I can just install, let's say a docker Webserver on all my machines simultaneously, with tweaks, like special environment vars, volumes, Labels or whatever to every single server, with just one command. And without ever connecting to the OS Terminal. ---------------------------------------------------------------------- PaleMongo · 2023-10-13 20:26 Agree, It was unclear. Thanks for the clarification.