How are you managing backups of your containers? What about Backups of Containers to https://hub.docker.com/repository/docker reddit / selfhosted ---------------------------------------------------------------------- loltrosityg · 2023-10-25 22:53 Hi There, I am just getting started with self hosting and docker containers. I have found that I can backup all my containers to this location easily via script [https://hub.docker.com/repository/docker](https://h ub.docker.com/repository/docker) My largest container compressed size in my private repository is 3GB. I was wondering if backing up to a hub.docker private repository is okay and if there is going to be cost/limitations to doing that? What are you using for backing up your containers? I noticed people often expressing importance of backups but then not sharing how they are doing their backups and their backup setup. ---------------------------------------------------------------------- thekrautboy · 2023-10-25 23:05 https://www.reddit.com/r/selfhosted/search?q=container+backup&restrict _sr=on > via script https://hub.docker.com/repository/docker That link doesnt seem to work. > I was wondering if backing up to a hub.docker private repository is okay and if there is going to be cost/limitations to doing that? Free accounts are limited to 1 private repo on the Docker Hub. And the Docker Hub is not a place for backups, at all. You do not backup containers. I dont know what youre trying to do, maybe youre exporting them? Thats not a backup. You backup the persisent data that each container requires (if any). And you backup the container config, very easy when using Docker Compose because you simply backup the compose file, done. Containers are not Virtual Machines. You dont stuff them into a zip and back them up in their entirety. There isnt a very specific tool to backup containers, because of that. You backup just the files and folders that your containers actually require. And for that thousands of backup tools exist, they dont need to be Docker specific at all. Use whatever suits you to backup those files and folders and the compose file. Nothing more is needed. Special attention should be paid to container of databases, like MySQL, MariaDB, Postgresql, etc. Its good practice for databases to not only just copy the plain files to some place, but to perform a actual database dump (exporting everything from the database into a special file which can later be restored). Each database type is unique, there is no general rule, but most of them provide a basic command line tool to perform a dump. So run that and save the resulting dump file with the backup of the other files. ---------------------------------------------------------------------- Minituff · 2023-10-25 23:13 Don't back up the container itself, instead backup the volumes the container is mapped to. I also backup my Docker Compose files to a GitHub private repo. I use [this](https://github.com/Minituff/nautical-backup) took to backup my container volumes. If you need custom tools inside a container, consider making a Dockerfile for it. ---------------------------------------------------------------------- Firestarter321 · 2023-10-25 23:13 I just back up the entire host that has Docker running on it. ---------------------------------------------------------------------- thekrautboy · 2023-10-25 23:15 And when you only need to restore a single container and its data? You restore the entire host? ---------------------------------------------------------------------- ScribeOfGoD · 2023-10-25 23:21 I too use a shotgun to hunt flies 😂 ---------------------------------------------------------------------- Firestarter321 · 2023-10-25 23:31 Yup. I back them all up daily so the data loss is minimal and acceptable to me. I do back up the docker-compose file when I make changes though. ---------------------------------------------------------------------- thekrautboy · 2023-10-25 23:33 Exactly. We have a common saying here thats like "using a cannon to shoot birds". ---------------------------------------------------------------------- thekrautboy · 2023-10-25 23:34 Thats a horrible setup to me, but if youre happy with it, okay. ---------------------------------------------------------------------- Firestarter321 · 2023-10-26 00:11 I may change in the future but this has worked fine for me for the last year so I'll continue to roll with it for the time being with what I have running in Docker. ---------------------------------------------------------------------- thekrautboy · 2023-10-26 00:27 *shrug* ---------------------------------------------------------------------- Obvious_Employee · 2023-10-26 00:36 Backup the files/persistent data with a backup solution like duplicati for example. Backup the compose files via git if you can. Imo, restoring containers is better done in a modular way where there are minimal dependencies. ---------------------------------------------------------------------- Firestarter321 · 2023-10-26 00:39 Something just occurred to me... Since I back up the entire Docker host nightly I can simply restore the host backup (or one from a month ago if needed) as a different VM to my Proxmox node and then grab what I need off of it at a file level using WinSCP to put back into the production Docker host. It may not be as fast as having the file system backed up ready to somewhere, however, I don't really see it as a big deal since my backups are done over a 10Gb network. ​ I may also look at PBS to back up just the Docker container volumes that aren't shared storage. ---------------------------------------------------------------------- thekrautboy · 2023-10-26 00:48 > Since I back up the entire Docker host nightly I can simply restore the host backup (or one from a month ago if needed) as a different VM to my Proxmox node and then grab what I need off of it at a file level using WinSCP to put back into the production Docker host. Yes, you can make the complicated approach even more complicated xD ---------------------------------------------------------------------- Firestarter321 · 2023-10-26 00:56 Different strokes for different folks I guess 🤷‍♂️ They don’t take long to restore over a 10Gb network with SSD’s. ---------------------------------------------------------------------- thekrautboy · 2023-10-26 00:57 Do whatever the f you want. > They don’t take long to restore over a 10Gb network with SSD’s. But thats not the point at all. ---------------------------------------------------------------------- flyingvwap · 2023-10-26 01:55 https://kopia.io/docs/installation/#docker-images ---------------------------------------------------------------------- CringeGinge666 · 2023-10-26 02:22 I don’t backup containers, I backup volumes with [nautical](https://github.com/Minituff/nautical-backup). ---------------------------------------------------------------------- ORUHE33XEBQXOYLZ · 2023-10-26 03:43 [They're just operating at scale](https://xkcd.com/1737/) ---------------------------------------------------------------------- PaulEngineer-89 · 2023-10-26 03:45 It’s called Hyperbackuo on Synology. It’s basically standard Unix incremental backup with a nicer front end. ---------------------------------------------------------------------- loltrosityg · 2023-10-26 03:59 Nice, thanks for sharing. ---------------------------------------------------------------------- random_hitchhiker · 2023-10-26 04:20 I know you can backup your containers to gitlab also ---------------------------------------------------------------------- yakultisawesome · 2023-10-26 05:53 I don't think backing up the container is the right strategy, as containers (I think\*\*\*) is built for portability. So you can spin one up in seconds, or migrate to another machine with very little hassle. For me I backup the docker host itself and the docker config files in the volumes with a script. I make the somewhat redundant back up of the config files in addition to the host is because I like to tinker things, and usually nuke something in the process. Having a more frequent backup of the docker configs allows me to easily restore if something goes wrong. ---------------------------------------------------------------------- JanBurianKaczan · 2023-10-26 06:15 I use resilio to clone all volumes every day ☝️ ---------------------------------------------------------------------- zoredache · 2023-10-26 06:22 I also backup the entire host. My host is using ZFS, and so I use zfs send via syncoid. So if I want to restore something, I access the remote, change into the snapshot directory for the particular point in time I want to restore to, then I copy the data (scp/rsync) back to the host. Of course that assumes I don't still have the data in a local snapshot, which I do, about 95% of the time I have had to restore something. In that case I just copy stuff from the old snapshot back to the production. I have also used borg, and restic to do full computer backups. Again, to restore data for a particular volumes or directory, you just mount the particular backup snapshot you want, and then copy files. Of course the important piece is that all data is in volumes or bind mounts. I have my compose files, and ansible playbooks in a git repo. So I can rebuild all the containers anywhere I need them as long as I can get back my data. I also have a task that runs mydumper occasionally against my mariadb container so restoring my databases is easy. ---------------------------------------------------------------------- Lanten101 · 2023-10-26 06:42 My container are on lxc aswell as on Ubuntu VMs, so I just back up the entire vm or lxc container, when restoring it. Everything is just there as it was ---------------------------------------------------------------------- tweek91330 · 2023-10-26 07:05 I won't say you cannot or shouldn't backup a container since it it wrong imo, but as your using docker as a base i'd say just backup mapped volumes and configuration files. Docker is built around breaking and rebuilding your container and keep working anyways as long as your mapped volumes and docker configurations are the same (asuming you use a well maintained image for updates or something of your own). That's probably the sane way to do it and will save you space. I don't see the benefit of a full container backup (unless it's an lxc, lxd container or another full persistent container type). As for the backup repository you can use whatever. If it were me i'd use another server or a NAS, but most cloud storage provider would fit. Storage is never free though. ---------------------------------------------------------------------- Jealy · 2023-10-26 07:42 I do the same, I have nightly backups with Veeam of the docker host that keeps 2 weeks worth of snapshots. I can either restore the entire VM or do a file level restore of individual containers. ---------------------------------------------------------------------- bufandatl · 2023-10-26 08:17 I either use public available containers like from docker hub or other registry. Or if I build them myself I have them pushed to my own self hosted registry with a minio(s3 compatible) backend and mirror the MinIO instance to a VPS. All dynamic data is saved with the VM backup the container runs on or is backed up with rdiff-backup to an offsite location. ---------------------------------------------------------------------- djc_tech · 2023-10-26 09:36 I don’t back up containers but I have bind mounts, so all of my data is in a ZFS data set that I rclone to another location. But people seem to like nautical ---------------------------------------------------------------------- CrispyBegs · 2023-10-26 09:48 do you happen to have a guide handy on backing up compose files via git? i just occasionally back them up from portainer by hand, but that's a really stupid approach, obviously ---------------------------------------------------------------------- CrispyBegs · 2023-10-26 09:49 i was doing this but there seem to be some log files somewhere that become huge and swamp the back up server's disk space. any tips for avoiding that? I'd appreciate knowing how your set up is configured. ---------------------------------------------------------------------- DasKraut37 · 2023-10-26 10:41 Oh hey, what up, KrautBoy! How many of us are there ‘round these parts anyway? ---------------------------------------------------------------------- Minituff · 2023-10-26 15:16 Developer of nautical here. Thanks for using my project!! ---------------------------------------------------------------------- betahost · 2023-10-26 16:20 I actually use [Restic.net](https://Restic.net) to backup to my NAS, I also host a local Docker registry to backup images. Docker hub removed their free tier. ---------------------------------------------------------------------- sv-giampa · 2023-10-26 17:39 No container backups, just volume backups. When you run a program on your PC, say a CAD or an IDE, you won't backup the software itself, but just data of your projects. The same is for containers. A container is the software, the volumes are your project data. You should store all the configuration of your container (e.g. config files, yaml files, databases directories, deployment files,etc. etc.) in one or more volumes that are persisted on your disk. By the way, the container lose all files not stored in volumes each time your server reboots, so the volume itself is a form of backups between power-ons. Backing up containers is not a practice. It is not needed, it must not be done, never. Sometimes you want change the internal environment of your container, and in that case you want create/commit an image to restore that environment. This practice is more correctly called 'image creation' whose purpose is clearly not backing up something. In ordet to recovery from some sort of disaster, you should backup your volumes and that's all. ---------------------------------------------------------------------- JanBurianKaczan · 2023-10-26 17:55 Hm, I have 150gb of my containers configs so yeah. Log files would have to be huuuuuuuuge for you to notice. Databases are bigger as well as other app data. To answer your question, just back everything up, less of a chance to have issues with restore. ---------------------------------------------------------------------- isleepbad · 2023-10-26 18:01 I have Synology and back up the entire volume. ---------------------------------------------------------------------- junialter · 2023-10-26 19:04 Nobody backs up containers. Why would someone do that? For the volume data I go for either borg or restic but that's in no way specific to containers whatsoever. ---------------------------------------------------------------------- hnyazi · 2023-10-27 04:27 I have Synology, backup docker vm. You can restore files/volumes only or entire vm from backup for business. Or you can use hyperbacup to backup volumes only. ---------------------------------------------------------------------- RadiantTangerine4898 · 2023-11-13 15:16 As noted, there are some limitations regarding the number of private repositories on dockerhub. Furthermore, dockerhub limits the number of free image pulls (https://docs.docker.com/docker-hub/download-rate- limit/). For self-hosting and managing images, I recommend Pulp. You can run it from a container as a standalone application and push images to it as needed (https://pulpproject.org/pulp-in-one-container/). The images are stored on your device and you can do with them whatever you want to. Thus, in this case, Pulp serves as a backup if you run it on your next computer. It has a nice CLI too. Here is the video I recorded today about Pulp: [https://www.youtube.co m/watch?v=grDXPtnJu4U](https://www.youtube.com/watch?v=grdxptnju4u)