Ways to backup your docker volumes ? reddit / selfhosted ---------------------------------------------------------------------- NothingInTheVoid · 2023-11-23 06:41 I bought a second hand NUC to have a little more horsepower to run various services. I have it connected to my NAS, but almost all of the docker volumes reside on the SSD in the NUC. It would be nice to be able to backup those volumes to my NAS in case the NUC fails. I have Debian 12 running on it. What are my options ? Should I just backup my docker volumes or does it make more sense to backup the entire NUC ? (I'm less tech savvy then I might appear. Please be generous with your explanation, I still have a lot to learn) ---------------------------------------------------------------------- nik_h_75 · 2023-11-23 06:56 Recommend you use docker compose to define you containers. Setup a structure with a folder per stack/container. In each folder put your docker-compose.yml file - and in your compose definition have all volumes setup as ./[folder]:[docker folder]. That way each stack definition (compose file) and volume data is fully contained in a folder. For backup, stop all containers and use your preferred file backup to backup you entire docker stacks folder. After backup - start all containers back up. (yes, doing file backup of volumes is not as good as doing DB export - assuming you have stacks that use DB's - practically I've never had trouble with restoring a container volume with DB files). ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 07:29 Use XFS as file system and use --reflink when you copy the volumes, creates an instant CoW copy no matter how big the folder is. You can then move, copy or whatever that folder to anywhere, or use a VM and simply backup the VM. ---------------------------------------------------------------------- sevlonbhoi1 · 2023-11-23 07:32 I use restic. Stop the container, run backup, start container. you can script it to run every night or something. ---------------------------------------------------------------------- This-Gene1183 · 2023-11-23 07:36 I do this too ---------------------------------------------------------------------- [deleted] · 2023-11-23 08:37 [deleted] ---------------------------------------------------------------------- Akash_Rajvanshi · 2023-11-23 08:40 Can you share the script to push on gitea server! ---------------------------------------------------------------------- Akash_Rajvanshi · 2023-11-23 08:40 Do you have any detailed guide on this? ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 08:43 You need a guide to use cp -R --reflink /source/volumes /backup/volumes ? ---------------------------------------------------------------------- nakkaya · 2023-11-23 10:19 You can mount NFS shares to the container. See \[1\]. That way data lives on the NAS and computations happen on the NUC. ​ \[1\] https://stackoverflow.com/questions/45282608/how-to-directly- mount-nfs-share-volume-in-container-using-docker-compose-v3 ---------------------------------------------------------------------- MundanePercentage674 · 2023-11-23 10:56 I use zfs mirror 2x SSD for cache data snapshot no need to stop or restart docker 0 downtime and super easy to restore data ---------------------------------------------------------------------- BulMaster · 2023-11-23 11:02 You can create NFS Docker Volumes that are mounted through docker and reside on the NAS. ---------------------------------------------------------------------- Obvious_Employee · 2023-11-23 11:59 Databases don’t line nfs. What are some other options? ---------------------------------------------------------------------- ChiliWombat · 2023-11-23 12:01 I use btrfs snapshots and restic. ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 12:02 cp -R --reflink /source/volumes /backup/volumes ---------------------------------------------------------------------- BulMaster · 2023-11-23 12:09 This 👆🏻 ---------------------------------------------------------------------- Faith-in-Strangers · 2023-11-23 12:44 Is there a way to generate a docker compose of every that’s already setup ? ---------------------------------------------------------------------- New_d_pics · 2023-11-23 13:07 [docker-autocompose](https://github.com/Red5d/docker-autocompose). Handy as hell. ---------------------------------------------------------------------- dtk20 · 2023-11-23 13:07 RemindMe! 1 day ---------------------------------------------------------------------- RemindMeBot · 2023-11-23 13:08 I will be messaging you in 1 day on [**2023-11-24 13:07:38 UTC**](http ://www.wolframalpha.com/input/?i=2023-11- 24%2013:07:38%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://www.reddit.com/r/selfhosted/comments/181v210/ways_to_backu p_your_docker_volumes/kafn31u/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=Remin dMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2F selfhosted%2Fcomments%2F181v210%2Fways_to_backup_your_docker_volumes%2 Fkafn31u%2F%5D%0A%0ARemindMe%21%202023-11-24%2013%3A07%3A38%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete %20Comment&message=Delete%21%20181v210) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remind mebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?t o=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside %20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Yo ur Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot& subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](ht tps://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot %20Feedback)| |-|-|-|-| ---------------------------------------------------------------------- CactusBoyScout · 2023-11-23 13:21 Wow people are recommending a lot of things I don’t do and now I’m worried I’m doing something wrong. I just have a folder on my Ubuntu boot drive called Docker with all of the persistent data from my containers. And I just tell Duplicati to backup that folder to BackBlaze. I don’t stop the containers to do that. Am I doing something wrong? ---------------------------------------------------------------------- pcs3rd · 2023-11-23 13:30 I have a disk that I have my container appdata, including portainer. I can just tar it every month and throw it elsewhere. Restoring is as easy as decompress and deploy portainer on that config dir. All my stacks get pulled back in. ---------------------------------------------------------------------- Ebrithil95 · 2023-11-23 13:34 I dont use volumes at all and instead use bind mounts so i can just back up those folders ---------------------------------------------------------------------- NiftyLogic · 2023-11-23 13:42 IMHO not really. There is the slight chance that DBs get inconsistent with backing up hot DB files, but in a homelab with minimal load this is usually not an issue. Same for NFS. Just make sure you have older backups, too. Just in case the last backup was not good. ---------------------------------------------------------------------- Big-Finding2976 · 2023-11-23 13:49 Some people would say that you're doing something wrong by using Duplicati, because they've had problems restoring data and it's very slow, so if you've never had to restore data before you should test that to check that it works, and maybe switch to something else like Borg to be safe. Also, backing up the folder without stopping the containers first might result in any backed up databases being corrupt, so if you're running anything that uses databases, you should stop those containers before backing up the folder. ---------------------------------------------------------------------- CactusBoyScout · 2023-11-23 13:50 How would you automate the stopping of the containers? ---------------------------------------------------------------------- Big-Finding2976 · 2023-11-23 13:56 I'm not sure, but I think you'd just need to create a script which stops them, runs the backup, and then restarts them. ---------------------------------------------------------------------- avdept · 2023-11-23 15:00 Thats how I provide all my content to ARR apps and also torrent client. ---------------------------------------------------------------------- zunfire7 · 2023-11-23 16:22 Be aware that databases might be corrupted by snapshots ---------------------------------------------------------------------- lizzard7 · 2023-11-23 16:24 Apart from what others have already mentioned: stuff in docker volumes can easily be copied using \`docker cp\`. IMO the easiest way to copy directly from the running container, no matter where it lives inside the container - not suitable for all purposes, but handy as hell. ---------------------------------------------------------------------- Minituff · 2023-11-23 16:34 If you just need to backup Docker Volumes, I recommend [Nautical](https://github.com/Minituff/nautical-backup). You can use it to backup to an NFS share if you need to go between Machines. ---------------------------------------------------------------------- Faith-in-Strangers · 2023-11-23 16:59 amazing, they even have a command to print it for all containers. I just " > backup.txt" and done. Thanks a lot ---------------------------------------------------------------------- mrmclabber · 2023-11-23 18:10 That’s a bind mount, not a volume. ---------------------------------------------------------------------- mrmclabber · 2023-11-23 18:11 I just use veeam to backup my host nightly. Worst case I’m out one day of data, and for my services that’s just fine. ---------------------------------------------------------------------- plastrd1 · 2023-11-23 19:47 Those backing up bind mounted volumes, what user does your backup program run as? The data inside bind mounts can have very random user IDs depending on the container including files owned by root. Does your backup have to run as root in order to capture all files and retain permissions? ---------------------------------------------------------------------- Reverent · 2023-11-23 20:45 Functionally the same thing. ---------------------------------------------------------------------- Reverent · 2023-11-23 20:47 No. Any database in the last 20 years supports WAL, this advice is straight out of 2005 ---------------------------------------------------------------------- christancho · 2023-11-23 20:48 Nop, docker volumes bring me anxiety since I have no clue where are the files located. I always go with directory binding, I retain full control, master of my files, captain of my soul. ---------------------------------------------------------------------- zunfire7 · 2023-11-23 20:53 That’s good to know then ---------------------------------------------------------------------- Frozen_Gecko · 2023-11-23 23:54 It most definitely is not. https://docs.docker.com/storage/volumes/ In most scenarios, it will be the same, but to somebody new to docker, it might be very confusing why a random container won't start. There are images that use volumes to pre-populate it with data, which will not happen when they are mapped using bind mounts. ---------------------------------------------------------------------- Reverent · 2023-11-24 00:39 Prepopulating volumes is poor practice (pretty much precisely for this reason). I can't think of a single container that doesn't perform a check to repopulate the volume if it's empty. A more pertinent reason that bind mounts can be problematic is permission mismatches between the folder and docker user, which IMO is something that people should understand and fix (if you don't know what file permissions your container needs, you probably have no idea why running your container as a root user is poor practice). Volumes can be more flexible in that they abstract away the actual location the files are kept. Volumes can also be incredibly misleading in that they abstract away the actual location the files are kept. In a thread literally about how to back up docker volumes, knowing where your files are and are not is pretty darn important. ---------------------------------------------------------------------- Frozen_Gecko · 2023-11-24 00:46 Oh yeah, I completely agree with you that's it's poor practice to not check. I just wanted to emphasize that volumes and bind mounts are not functionally the same, as there is a most definite difference in their functioning. Recently, I ran into the issue I described when setting up tandoor recipes. ---------------------------------------------------------------------- mrmclabber · 2023-11-24 19:09 They both provide storage, that’s about where the similarities end. He specifically asked about volumes which are *not* bind mounts. Volumes have benefits over bind mounts, the only downside is they are “harder” to backup and you can’t just manipulate the data on your host. It’s like saying a Tesla is the same as a Honda civic because they are both cars. True, but functionally very different. ---------------------------------------------------------------------- mrmclabber · 2023-11-24 19:22 So the answer to bind mounts and not being able to deal with permissions issues is “learn how.” But when it comes to volumes it’s “incredibly misleading” and “knowing where your files are is incredibly important,” also something that is super easy to find out in the docker docs. Bind mounts are simpler for n00bs, but they are inferior to docker volumes. ---------------------------------------------------------------------- This-Gene1183 · 2023-11-25 06:50 I googled but here is the base line https://unix.stackexchange.com/questions/226677/pushing-to-a-git-repo- from-shell-script Create a git inside your main docker folder. It will upload everything inside to your git.