Backup software for Docker volumes and bind mounts? reddit / selfhosted ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-16 22:01 I have a docker host w/Portainer that runs most of my homelab services and I'm looking to update my backup methods. For a long time I've been using [https://github.com/offen/docker- volume-backup](https://github.com/offen/docker-volume-backup) for creating sidecar containers that backup the data from the bind mounts or volumes of my main containers to my NAS. It works well but it doesn't scale well. I need to remember to go in and modify the Dockerfile for the stack and add the offen config and a bunch of the environment variables I have abstracted to the stack config and that has to be manually added each time. It's getting annoying at this point since I have 30+ containers running with maybe 1/4 to 1/3rd being offen containers and the whole process is getting tiresome. I'd like to move to something that has a central interface where I can configure the backups for each of the containers individually (just tell it which bind mount on the host to backup). I've spent a ton of time over the last couple days trying to find an app that meets my criteria and keep coming up short. Looking for suggestions. **Criteria** * Runs in Docker * Has GUI `(1)` * Backs up regular files. No forced deduplication, encryption, etc * Standalone client (doesn't require backup server software) * Can backup to SMB share `(1)` I'm 100% comfortable on the CLI but I'm tired of having to use it for so much stuff and I really don't feel like going in and running a bunch of CLI config each time I want to backup a new docker container and I'd like to have a UI where I can easily see and monitor the status of my backups. ❌ **Duplicati:** * Still in beta after many years and [many threads](https://news.ycombinator.com/item?id=33450632) about how unstable it is * Over 800+ open issues in github seem high * May only provide deduplicated storage, not sure. ❌ **Restic:** * [Compatability issues with SMB](https://github.com/restic/restic/iss ues/2659#issuecomment-1546701199) * Primarily CLI with only some 3rd party UIs and unclear if any of them run in Docker. ❌ **Borg:** * Requires borg server side software for backups? ❌ **Kopia:** * Fits most of my criteria but has forced encryption which [cannot be disabled](https://kopia.discourse.group/t/is-there-any-way-to-disable- encryption/807/2) ❌ **Duplicacy:** * Forced deduplication ❌ **ElkarBackup:** Works in basic tests but unclear if it's literally just rsyncing to the backuplocation or if there is some kind of snapshotting or incremental option. A bit on the heavy side with 3 containers including a mysql db. [Project is abondened](https://github .com/elkarbackup/elkarbackup/issues/659#issuecomment-1673442885) as well. ❌ **Rclone:** * Got it mostly working with [this](https://forum.rclone.org/t/how-to- set-up-rclone-webgui-server-as-a-docker-container/14330/12) but event most recent posts I could find [from 2022](https://forum.cloudron.io/topic/7159/rclone-web-gui) say it doesn't have all features and is still experimental. * Rclone doesn't seem to be well suited for backups anyway and is more for just copies? ❌ **LuckyBackup:** * Tried via Unraid app (docker based) but super dated UI that uses VNC web view into the GUI? Strange. Unclear if this can be run in vanilla docker [https://github.com/ich777/docker- luckybackup#](https://github.com/ich777/docker-luckybackup#) ❌ **UrBackup:** * Client / Server model. Unsure about rest of features. ❌ **Syncthing:** Strongly recommended to not use it as a backup tool. Too much risk of misconfugiring it and accidently syncing unwanted changes in the wrong direction, etc. ❓ **Veeam:** * Keeps getting brought up in threads but unclear to me if / how it could fit my use-case. No GUI / web interface? ---------------------------------------------------------------------- ElevenNotes · 2023-12-16 22:48 A container is the sum of it's volumes and a single run file (yaml or whatever you prefer). Why is this hard to backup? Simply *cp -R --reflink* the volumes and then rsync to **any** storage. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-16 23:07 It's not "hard" to backup via various CLI methods but as per the post I to move to a GUI for various reasons and have a central place to manage all the backups, see status, error, etc, etc. ---------------------------------------------------------------------- ElevenNotes · 2023-12-16 23:12 Why people need a GUI for everything is beyond me. A script running has an exit code. If not 0 throw an error to your GUI monitoring tool. ---------------------------------------------------------------------- anony_mous_me · 2023-12-16 23:20 It's unfortunate to see answers like this that come off condescending. OP stated what they're looking for and said explicitly they're not interested in working on the command like so these kinds of comments that basically scoff at the idea of working anywhere but the command line are just rude and unhelpful. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-16 23:22 This is a bit silly IMO. There is value to both CLI and GUI tools. If you want to do literally everything you can on the CLI then fine. That doesn't fit everyone's preference or use cases. ---------------------------------------------------------------------- d4nm3d · 2023-12-16 23:58 Veeam gas a gui, but it's not a web based one. I use it to do exactly what you're proposing.. but it's heavy and requires a Windows machine to run on. Are you literally just wanting to sync the contents of the bind mounts to elsewhere.. over writing each night? Or do you want some kind of retention period (i ask because your requirement of no dedupe confuses me.. the only reason i can think of is that you want the backups to be softrware agnostic.. in which case you're just looking to sync the files each day to a dated folder and after x days, delete the oldest folder?) ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 00:08 > Or do you want some kind of retention period (i ask because your requirement of no dedupe confuses me.. the only reason i can think of is that you want the backups to be softrware agnostic.. in which case you're just looking to sync the files each day to a dated folder and after x days, delete the oldest folder?) Right, I could have elaborated a bit more. Currently with offen backup I’m doing a “snapshot” in a sense where it zips the contents of the bind or volume mount for each container, sends it to the NAS over SMB, and removes old ones based on the defined policy. Each container has its own policy defined as each app has its own needs etc. I’d like to keep that overall model. I’d like to avoid Windows for sure. ---------------------------------------------------------------------- rafipiccolo · 2023-12-17 00:23 im sorry but i love this ipv6 joke :) as long as it's a joke its pretty nice ---------------------------------------------------------------------- nik_h_75 · 2023-12-17 00:26 1. Install proxmox 2. Create VM to run your Docker stacks 3. Setup VM backups in proxmox (to nfs share) - fully done in gui with retention policies (including "test" of policies to test backup plan) ---------------------------------------------------------------------- rafipiccolo · 2023-12-17 00:36 i run rsync every hour from a cron, from the backup server to the other host via ssh. i decided to go with hardlinks mode (not incremental because it's way more convenient to me to see actual full folders, and it's using the same disk storage on the backup host) i like the fact that it's only using unix classic / safest tools. borg and some others can add a layer of quality (deduplication, compression, encryption) and problems (load, lag, push mode only, ...) my cron is a docker container and it has a beautiful UI to report failed and winning jobs. my backups are plain folders, so one can do a "du" or "ncdu" or anything useful to gain insights. rsnapshot can do rsync with hardlinks for people who dont understand this or dont want to try it themselves. it has some gui available too. ---------------------------------------------------------------------- Sammeeeeeee · 2023-12-17 00:41 I'm perfectly comfortable with cli, but I do a lot of stuff remotely, from my phone, and cli like that is a bitch. Also, my boyfriend is dyslexic and he finds cli extremely difficult, and he much prefers gui. (He's not a techie, I'm just teaching him how to use all this stuff). Therefore, I also heavily prefer anything I'm gonna be touching often to have a gui. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 00:43 Seems kind of overkill to install a whole VM stack just to leverage its backup features but I’ll keep it in mind if nothing else comes up. ---------------------------------------------------------------------- d4nm3d · 2023-12-17 00:44 gotcha.. but i doubt anything like what you want exists not with a gui anyway.. it sounds very much like a scriptable job that can just be run on Cron.. if you have all your bind mounts in a root folder then there would be no need to update any config.. just recurse through them creating a zip file of each.. blah blah .. you want a GUI... The closest thing i can think of is maybe continue what you're doing with offen but have webmin coupled with it so you can edit the config file in a gui.. not sure that would hit your requirements.. it also wouldn't let you monitor what's going on really.. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 00:46 > my cron is a docker container and it has a beautiful UI to report failed and winning jobs. Which docker image are you using for this? ---------------------------------------------------------------------- d4nm3d · 2023-12-17 00:50 the more i think about this.. the more it's something i could likely use myself... I am by no means an expert but i've been known to put together some gui's in the past.. maybe this is something i could try.. It was my work Christmas party last night so give me a bit of time to recover from that and ill see what i can do ---------------------------------------------------------------------- BrodyBuster · 2023-12-17 00:54 No need do over complicate things, right? My backup procedure is the same. Little rsync script with hardlinks. Hasn’t failed me yet ---------------------------------------------------------------------- rafipiccolo · 2023-12-17 01:05 i'm sorry for the false joy you may experience. I started with this : [https://github.com/mcuadros/ofelia](https://git hub.com/mcuadros/ofelia) and loved the idea of setting new crons via docker labels. it worked good but doesnt have a gui. Then i developed my own cron with nodejs. That way i could have the monitoring and front i wanted. maybe one day i'll be able to set it public, but i would need to write the proper amount of doc and provide a more generic solution for people to use it. You could stil go with a normal cron and [https://healthchecks.io/](https://healthchecks.io/) ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 01:34 > I do a lot of stuff remotely, from my phone, and cli like that is a bitch. Was going to raise that same point but didn’t want to waste any more time responding to that person. My phone is always on me and allows me to conveniently check on things at any time, respond to alerts, make config changes, start / stop things, etc. Often doing it in the moment when I’m thinking of it or need to address it is way more important and valuable than doing it on a “real” device like a PC or laptop. Even then many things are just much more convenient, faster, and easier in a GUI vs the CLI, not everything but lots of things. There’s a reason GUIs were invented and are used as the primary interface for most things. ---------------------------------------------------------------------- Minituff · 2023-12-17 02:01 I have been using [this](https://github.com/Minituff/nautical-backup) image, but it doesn't have a GUI. ---------------------------------------------------------------------- nik_h_75 · 2023-12-17 02:13 True, but depends on your "needs GUI" priority. Proxmox brings a lot of other benefits (not least that the VM backup is the whole VM, not just docker - so easier to restore). ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 02:35 That would be pretty cool. I'm sick as a dog right now and can't seem to shake it but when I'm well again I can try to pitch in if you end up starting something. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 02:48 Yeah that's seems like a round about and heavy overhead solution for my use case. Installing an entire VM platform, running Docker in the VM and snapshotting the entire VM just to backup some mostly text data for what is designed to be an ephemeral system like Docker is very bulky. It also doesn't let me setup separate backup schedules and related tasks for each container volume, it's all or nothing all the time. So for example if data for most of the containers isn't changing very often but one or two containers change their data often and I want to back that up more frequently I have to backup *everything* which is really inefficient. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 02:51 It's hard to tell from the documentation (examples seem thin) but is intended to be added to each docker compose file you want to backup containers for, like how effen volume backup works? ---------------------------------------------------------------------- Minituff · 2023-12-17 02:55 It should be similar to Watchtower if you use that. You should be able to add labels to each compose file if you want custom configs for that container. Otherwise everything is contained within the Nautical compose. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 03:10 Thanks. I may give this a look. Sounds similar to effen but centralize. I can get away without a GUI if I can at least have feature parity with effen and have things in a central place. GUI would still be nice for monitoring and such though. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 03:34 This could be a decent starting point. It's got a good foundation with rsync doing the heavy listing. Mostly it's lacking incremental / snapshot / version type features. I believe in it's current state it just syncs everything over top of the same backup each time. https://furier.github.io/websync/ ---------------------------------------------------------------------- Ejz9 · 2023-12-17 03:43 Might I ask why you do not want encryption? I use Kopia myself so I’m just curious. I’ve had to restore already too but if you keep track of your login for the repository I don’t see where this is an issue…. I guess I’ve never heard of not wanting encryption even if no one’s expected to access your data. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 03:58 >Might I ask why you do not want encryption? > >what do you not like about deduplication either? Good questions that I thought would come up sooner. I'm not opposed to either for offsite or other types of backups strictly for this backup case I'd prefer the backups be readily readable and portable. One reason being that if the docker host blows up I want to be able to easily just copy the backup folder(s) for the container data onto thew new host and be up and running without having to install and setup specialized backup software just to read and restore the backups. Another user describe similar and other reasons here. I've also been burned before by software that does funky things to the backups and then they've become unreadable down the road for whatever reason and you can just reconstruct the data yourself. Kopia is actually really high on my list for lots of reasons so I may consider if for my cloud / offsite backups but for this case it's actually a hinderance of how I want to work. Honestly I may give up my "standard file" requirement and go with it if other else presents itself. ---------------------------------------------------------------------- TerminalFoo · 2023-12-17 04:08 Why do you need SMB for Restic? It looks like Go has a problem with SMB and Rclone seems to have a solution. So, if you really needed SMB support for Restic, you can get it via Rclone. Also, I use this docker container to handle my docker container and bind mount backups. [https://github.com/djmaze/resticker/](https://github.com/djmaze/resti cker/) ---------------------------------------------------------------------- Ejz9 · 2023-12-17 04:24 Understandable. I guess I’ve yet to get burnt and hopefully don’t 🤞. *famous last words* I mean there’s always just taking a manual backup every so often 😁. Hopefully you can find your solution though. Another idea I have is making a SMB of the specific folder for the file you need and just copying it/them. Simple, you see what’s happening, all regular. No special software but I dont know what does this otherwise. Using backblaze comes up too but I dont think they have docker for their stuff and I think they require an S3 client. I’m not sure if there app works on windows for personal backups though, I mean I believe you get 15gb free so… Best of luck and I’d be interested to hear what you eventually go to. ---------------------------------------------------------------------- psicodelico6 · 2023-12-17 05:31 O use proxmox and proxmox backup. Deduplication reduce 127tb to 5tb ---------------------------------------------------------------------- Acceptable-Jump-8332 · 2023-12-17 05:52 I have been running restic backups on an SMB share with no difficulty. It's all automated now and I have tested restores as well. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 05:53 Someone else suggested Proxmox but seems very overkill. I don't currently run Proxmox or any VM host. Running a backup server also doesn't really fit my use case for several reasons. Looking for something that's just client side. Looks like Proxmox backup requires a server component? [https://www.proxmox.com/en/proxmox-backup- server/overview](https://www.proxmox.com/en/proxmox-backup- server/overview) ---------------------------------------------------------------------- psicodelico6 · 2023-12-17 05:54 Proxmox Backup install in vm o bare-metal. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 06:04 People don't get jokes on Reddit. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 06:06 Because people on Reddit are very lazy at even learning the basics. They want pre fab solutions that solve all their needs for free and to absolute perfection at all times. How dare anyone who does not provide the perfect answer right away, and how dare them who want OP to actually use his brain and not just copy/paste the next thing. Why do you think NPM exists? Because Nginx is difficult? No, because people are even too lazy for that. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 06:08 It's not about the CLI, it's about your unwillingness to solve your specific problem yourself or even try. You want a pre made perfect solution made for you, for free on top of that. For a problem that isn't really difficult to solve. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 06:10 I doubt you need or want to check the backups of your containers on the go and all the time from your phone. That's neurotic. This is coming from a professional that monitors thousands of servers and containers. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 06:10 Why do you work from your phone? ---------------------------------------------------------------------- anony_mous_me · 2023-12-17 06:36 Except the tone is very much not a joke and even if it had a joke-like ton it's unfunny. I just comes across very asshole-ish, which this person seems to be based on a lot of their other comments. ---------------------------------------------------------------------- anony_mous_me · 2023-12-17 06:40 You sound extremely bitter. I hope you can turn that around for your sake and the sake of people around you. Good luck. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 06:46 Nobody here has said they "work" from their for but it's 2023 FFS and there are times where a phone is the most immediately available device and there are plenty of cases where certain tasks can be easily done on a phone. Many modern systems have well designed mobile interfaces for exactly that reason. Stop pretending like everything should be done like it's 1993. You're just being combative at this point and embarrassing yourself. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 06:50 Ah don't worry my life is pretty much perfect in every aspect, not here to compare dicks, because that's a competition you would probably lose on every aspect of yours. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 06:52 If you have to need to check servers or even fix issues from your phone, maybe you should lay off off tech for a while and go out and live a little, without your phone constantly checking your portainer UI. ---------------------------------------------------------------------- marmata75 · 2023-12-17 07:04 Proxmox backup server is a separate install and doesn’t require proxmox. Although, it needs a server and doesn’t store files ‘as is’ so doesn’t match your other requirements. Still a great piece of software! ---------------------------------------------------------------------- lilolalu · 2023-12-17 07:52 rsnapshot, a backup script for rsync https://rsnapshot.org/ GUI for rsnapshot https://www.elkarbackup.org/ Ticks all your boxes. I see that you mentioned Elkarbackup already, and assume that it has been abandoned. i don't use the GUI but rsnapshot from the CLI - it literally just creates directories, copies the files in there and links duplicates to their originals, so ANY file browser really is a GUI to rsnapshot Backups. Also take into consideration that rsync is one of the oldest Linux tools available, rsnapshot exists for years ... sometimes tools don't receive frequent updates because they just do what they are designed for, without flaws. ---------------------------------------------------------------------- bigahuna · 2023-12-17 09:53 rsync and / or rsnapshots. No need for more. ---------------------------------------------------------------------- digitalindependent · 2023-12-17 10:16 Borg with borgmatic. No docker, but… Very simple install Just one yaml to configure The machine running Borg pulls backups and saves them to mounts via sshfs or other means You don’t have to have Borg running on the targets, you can work around that. One more thing: If you don’t like encryption, why not set the password to „1“. ---------------------------------------------------------------------- MegaVolti · 2023-12-17 10:29 Why? The whole premise seems overly complicated. What's the point of backing up individual bind mounts and configuring backups for containers individually? I simply have all my bind mounts in /mnt/containers/service_name and have exactly one backup config for /mnt/containers Whenever a new bind mounts gets added, the backup script will include it automatically. I can't forget to add something or adjust a config, it just happens automatically. And of course I can configure overrides for individual sub-folders if necessary (if I ever need different retention periods for different services) but so far this hasn't come up, I'm happy with one setting for all. ---------------------------------------------------------------------- Thutex · 2023-12-17 10:49 why not urbackup? assuming you use volumes on the docker host, you just need to install the client on your docker host and tell it to backup "/data" for example, to have the data from the hosts backed up. true you need a client on the host to backup, and the server running somewhere, but iirc you can just run urbackup server in a docker container, so on the same host - and use some mounted storage to backup to. ---------------------------------------------------------------------- KubeGuyDe · 2023-12-17 11:07 Imo that's the only way to do it. ---------------------------------------------------------------------- Sammeeeeeee · 2023-12-17 11:20 I'm out for the majority of the day, I don't want to carry a laptop with me in case something at my home server needs doing. ---------------------------------------------------------------------- Simplixt · 2023-12-17 11:25 Exactly.Just a simple rsync of the /mnt/containers/ folder.I also wrote a bash script, that stops all container before, and restarts afterwards, to not run in any problems with backups of databases. No need for an GUI, backup docker container or what ever. ---------------------------------------------------------------------- maretoni · 2023-12-17 11:27 why not just put all docker volumes in one location and use rsync on crontab...super simple ---------------------------------------------------------------------- Simplixt · 2023-12-17 11:34 *I'm 100% comfortable on the CLI but I'm tired of having to use it for so much stuff and I really don't feel like going in and running a bunch of CLI config each time I want to backup a new docker container and I'd like to have a UI where I can easily see and monitor the status of my backups* Same mistake I did in the past. GUI's a great for beginners, but for such simple tasks, just another app you must maintain. Don't over complicate things. Just keep it simple. Point all bind mounts to the same folder, e.g. /mnt/containers. No need to touch your script after adding any additional docker container, as you are backing up the parent folder. For notification, there are great solutions like [Healthcheck.io](https://Healthcheck.io). Just integrate a Healthcheck Ping in your script, and you get notified if anything goes wrong (e.g. the ping was not received) If you need a GUI to write your script, just let ChatGPT write it for you. It's working really great and the code quality is perfectly fine for such basic tasks. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 11:37 Your home server is that important? ---------------------------------------------------------------------- Sammeeeeeee · 2023-12-17 11:39 I have implemented redundancy. I have a couple (not many) people using stuff on it, but I like to keep an eye on what's going on so I haven't automated it yet. ---------------------------------------------------------------------- ElevenNotes · 2023-12-17 11:55 See, there is the truth. You did not do a proper job, that's why you need to check in on your infra from your phone. Why not be honest from the start? ---------------------------------------------------------------------- kameleon25 · 2023-12-17 13:01 Have a look at backup pc https://backuppc.github.io/backuppc/ I have been using it for over a decade now and it us awesome for Linux hosts. Will does works but is a little more setup. I use it as a docker instance on my synology for my home lab stuff and I have another instance in a vm for my work Linux machines. I love the work flow to restore especially vs Veeam which we use at work for everything else. ---------------------------------------------------------------------- mtucker502 · 2023-12-17 13:43 Duplicati has been working fine for me. ---------------------------------------------------------------------- sk8r776 · 2023-12-17 17:09 I think your criteria is way too specific. While I wish a tool that fit all of them existed, I don’t think you are going to get everything you want here. You have basically ruled out all the bigger projects one way or another. I personally run all my hosts as vms on proxmox and a pbs bare metal host. I do have multiple locations though and it makes more sense for me that way. You may have to loosen your criteria on what you are looking for, especially the gui one since the one area open source projects lack to me is the UI. Lots have them, rarely they are good or fully fleshed out. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 19:24 That’s definitely the easiest approach but it trades setup simplicity for storage (and other) inefficiencies. For example some containers such as Plex and other media related ones have quite large caches that don’t need to be backed up and just bloat the backups. They are often gigabytes in size and not compressible because they’re media. We could exclude those with the --exclude flag but that doesn’t scale too well and can get unwieldy as a single command if you have to too many. Then there is the question of different backup schedules and retention policies. Some containers only require being backed up say once week while others need to be backed up several times a day and have longer retention. Doing it in a single sweep means either backing up everything way more frequently than necessary to make sure the more important stuff gets its needs met. Same issues on the retention side in keeping backups for longer than necessary for most containers just to make sure you get the retention you need for the important stuff. There are of course various solutions to all these issues but point being it’s not as simple as it seems on the surface and why I prefer not to treat all the containers with a single backup policy / command, etc. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 19:26 Sounds like you've solved some of the [issues I have](https://www.redd it.com/r/selfhosted/comments/18k1scz/comment/kdsgplz/?utm_source=share &utm_medium=web2x&context=3) with the 'shotgun' approach of backup up the entire bind mount directory such as excluding large cache subfolders and setting up different retention policies or backup frequencies. What tool are you using and what does your config look like? ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 19:41 I may revise Elkar. I think I skimmed over it a bit hastily in between some others because the retention mechanism wasn't super obvious at the time. Looks like that's handled under the policies section so I'll experiment with that a little. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 20:05 I found this after making the post but the documentation is unclear about setup. They mention an http interface but don't show one anywhere and there's mention of needing your own Apache instance? Also it seems like it's entirely server side and you have to connect from there to each client over SSH or other protocols? I'm not necessarily fully opposed to that but it's not my preferred workflow. ---------------------------------------------------------------------- kameleon25 · 2023-12-17 20:13 It's not a backup if it's on the same host, keep that in mind. Hence the ssh to the backup targets. The documentation could use a little cleanup but the instances I run have everything included. I'll have to check which one I'm running when I get home. ---------------------------------------------------------------------- MegaVolti · 2023-12-17 20:42 I use btrfs as file system and btrbk for snapshot management, including sending it to all my backup targets. File system based tools are simply the best way of doing backups, if only for their atomic nature - no need to stop containers during backups. As for exclusions etc: I do use a single database directory that has all my container names with a database and CoW turned off (since database operations and CoW don't play well together). You could do the same with bind mounts for caches. Simply put that mount point on a different subvolume and it won't be backed up at all. Or just don't bind mount the cache at all - if it stays in the container, it will never be backed up and automatically cleared on every image update. Personally, I'm not using different retention policies for my containers. I do have different ones for media, which of course has different mount points, but the container-specific files are usually just about its config which tends to have a very small footprint. ---------------------------------------------------------------------- maretoni · 2023-12-17 20:52 I guess Plex offers different volumes for cache and db, so that's easy to separate. different schedules should be easy with multiple crontab settings... Am just trying to see the simplest setup that satisfies your needs...managing that many container I don't see how a gui would make it easier to manage than just a bunch of text files with battle proven tools. 🤔 ---------------------------------------------------------------------- maretoni · 2023-12-17 20:54 of course there are more enterprisee needs and tools, but hope you earn money with it at that point 😅 ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 21:08 Everyone suggesting to avoid the GUI definitely isn't wrong, it's a nice to have vs a hard requirement in most cases but I greatly appreciate having a GUI for several reason one of which is helping avoid my own mistakes. With the CLI and config files things are wide open for copy / paste errors, typos, etc. Most of the time they will be obvious because the command will fail but not always and things could silently fail in the background. Having a GUI abstracts a lot of those issues away because you're using fixed values via drop downs, buttons, etc. and can't mess certain things up. Additional it's not to have a visually well presented set of info to see the status of all jobs at a glance, if they've failed, when they ran last, how much space they're taking, etc. That's all very doable via the CLI too but requires manual work and being at a PC with a proper terminal and makes it difficult to do on mobile, etc. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 21:18 >I use btrfs as file system... File system based tools are simply the best way of doing backups Interesting. More feature rich file systems like btrfs, zfs, etc are not something I have much exposure to yet. I'm currently building an Unraid server so learning a bit but my Docker host is just using standard ext4. Maybe I'll consider something like brtfs next time I rebuild the host. ​ >You could do the same with bind mounts for caches. Simply put that mount point on a different subvolume I thought of that shortly after making my previous comment and it's actually not a bad idea for other reasons too, aside from backups. >the container-specific files are usually just about its config which tends to have a very small footprint. I may rejig my docker-compose files next time I do maintenance to separate the key configs, etc which would definitely simplify backups. If I can get the main bind mount folder down to just text based configs then backing up more frequently than necessary for some of the containers isn't an issue, especially if I use compression. Gives me something to think about. Appreciate your kind and thorough response. Some people get really arrogant and condescending and pretend like their way is the only right way and everyone else is stupid for having different criteria and preferences. There are definitely some approaches that are objectively better than others but we're also human and have different personal preferences and ways we like to work so there are 1000 ways to slice a lot of these problems and that's also ok sometimes. ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-17 21:35 You're literally gaslighting and berating people now. Just stop. You clearly need to "spend less time on Reddit" like you're telling others to do. ---------------------------------------------------------------------- d4nm3d · 2023-12-17 22:43 that issue could be rectified by firing a script after the fact to rename the folder to the days date (potentially)... i mean for keeping some kind of history. I've not found anything that can do incremental style backups that doesn't repackage the files in to their own blocks like Kopia / duiplicacy etc.. ---------------------------------------------------------------------- maretoni · 2023-12-17 23:32 yeeahh alright, I see your points re GUI. still think it's overcomplicated and I would do it differently, but I get your points. thanks for elaborating 🙏 will follow this thread, am curious what the others come up with 🤔 ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-18 01:36 Just came across [Cronicle](https://github.com/jhuckaby/Cronicle). Sounds similar to what you've built. Seems pretty flushed out and there are some people who have [wrapped it in docker](https://github.com/soulteary/docker-cronicle). Might give this a try since it provides the GUI element for monitoring and a like but is more flexible than the purely backup tools I was looking at. ---------------------------------------------------------------------- KubeGuyDe · 2023-12-18 07:17 Normally I'd agree that there are 100 ways to do something and it all comes down to personal preferences. But in this case I don't. It sounds a lot like you're running containers like vms. So the root of your problem wouldn't be a problem of choose, but a false approach. E.g. containers shouldn't depend on any state. Everything stateful should be mounted as volume, but only that. Also the required config should rather be injected instead of hard coded into your Dockerfile. In most cases you shouldn't be required to have custom Dockerfile for any open source program but simply run the upstream base image without any changes. I'm aware that not every application is designed to run perfectly in a container yet. And that there are tools that support you running them as vms. Yet I'd always try to do it the cloud native way. Not trying to lecture you or anything, just want to point out that you might be missing an important point about the discussion, based on what I've read. ---------------------------------------------------------------------- Quubee79 · 2023-12-18 08:30 How do you handle which backups to keep? ---------------------------------------------------------------------- CrispyBegs · 2023-12-18 11:50 > the required config should rather be injected instead of hard coded into your Dockerfile. this is interesting. can you expand on it a bit more? i use portainer and write my compose scripts straight into the stacks section, and that sounds like what you're saying shouldn't be done. i'd def be interested in hearing a better way of doing things if you have a moment! ---------------------------------------------------------------------- BrodyBuster · 2023-12-18 12:57 here is the script I wrote to perform the task, including stopping docker containers if you want to. https://pastebin.com/nC2D0jwQ It uses a combination of sort (by folder name which contains the date) and find to determine which backups to delete. The number of backups is defined in the script. ---------------------------------------------------------------------- Simplixt · 2023-12-18 13:10 I think he is just irritated about the statement: *"If I can get the main bind mount folder down to just text based configs then backing up more frequently than necessary for some of the containers isn't an issue, especially if I use compression."* As your persistent files shouldn't be that big with most docker projects out there. Just separate your mounts into e.g. 4 categories: /mnt/config /mnt/media /mnt/db /mnt/cache When you can define a different backup retention each. But it wouldn't be a problem at all, if you backup your /mnt/media folder every hour, as long as you use a backup solution that supports deduplication and versioning, e.g. Kopia or BorgBackup. ---------------------------------------------------------------------- KubeGuyDe · 2023-12-18 14:16 Think of a basic nginx with an nginx.conf in `/etc/nginx` and some html in `/usr/share/nginx/html `. You could package that into your container via Dockerfile `add`. Any change would need a docker build/run with new tag. Or you have it in git, checked out on your host and mounted into the container at the right place, running a vanilla upstream nginx:latest. A change to the nginx.conf requires only a pull on the host and a restart of the container (or a hot reload). Portainer even provides that by using a git repo when creating a stack. But you could probably also just have those stack files on the host and mounted into the right directory (untested though). That's what I mean by injection. Create and backup (git) the config outside of the container and mount it into the container. I think the official term is "externalized configuration". ---------------------------------------------------------------------- KubeGuyDe · 2023-12-18 14:21 No, it's the other way around. Writing the config outside the container on the host and mounting those files into the right directory. That way you don't need to create any backup of the container, but just a backup of the config files, e.g. via git (assuming the rest of the container data can be lost). ---------------------------------------------------------------------- guesswhochickenpoo · 2023-12-18 16:46 You’re misunderstanding my comments. I’m not running them like VMs. They’re all basically “off the shelf” images and compose files from places like linuxserver.io and run with basically default settings. What I mean is that rather than using the default bind mounts where, say Plex for example, lumps all the persistent data into a single mount point (/config) split out the important configuration bits from say the non-text caches, etc into different mount points on the host to simplify the backups. i.e. as you suggested.