How do you back up all your selfhosted services reddit / selfhosted ---------------------------------------------------------------------- muguli222 · 2023-11-23 17:25 Hi Everyone I need some help I'm currently selfhosting some of my applications on Digital Ocean and i run a container using Portainer CE. I was wondering how you guys keep backups for the applications running on docker. I'm currently using Digital ocean's snapshots feature but is there a better way i could use, any help on this is highly appreciated. ---------------------------------------------------------------------- punintended2 · 2023-11-23 17:40 RemindMe! 3 days ---------------------------------------------------------------------- RemindMeBot · 2023-11-23 17:41 I will be messaging you in 3 days on [**2023-11-26 17:40:12 UTC**](htt p://www.wolframalpha.com/input/?i=2023-11- 26%2017:40:12%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://www.reddit.com/r/selfhosted/comments/18267f2/how_do_you_ba ck_up_all_your_selfhosted_services/kagoimk/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=Remin dMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2F selfhosted%2Fcomments%2F18267f2%2Fhow_do_you_back_up_all_your_selfhost ed_services%2Fkagoimk%2F%5D%0A%0ARemindMe%21%202023-11- 26%2017%3A40%3A12%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete %20Comment&message=Delete%21%2018267f2) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remind mebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?t o=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside %20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Yo ur Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot& subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](ht tps://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot %20Feedback)| |-|-|-|-| ---------------------------------------------------------------------- Xenkath · 2023-11-23 17:41 Use volume mounts for persistent data in your containers. Use a script to shutdown the containers, copy data directories with your preferred backup solution, and restart containers. Done. ---------------------------------------------------------------------- iammilland · 2023-11-23 18:22 Google Buckets is awsome, you control the encryption and you control the retention, and it is dirt cheap. Gsutil works on most nix systems. ---------------------------------------------------------------------- Conscious-Fault-8800 · 2023-11-23 18:35 This! Cronjob with a simple bash Script. Some Application containers I stop and backup weekly, some only monthly ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 18:53 --reflink, no need to shutdown ---------------------------------------------------------------------- emmebi14 · 2023-11-23 19:04 I use docker in Proxmox and i backup all container ---------------------------------------------------------------------- Proximus88 · 2023-11-23 19:09 I use [Restic](https://restic.net) to backup to a local NAS and Backblaze B2. And I use bind volumes to a docker path folder. The backup script is run by cron every day. First I backup my databases in a folder only accessible by root. Then I backup the docker folder that includes my backup up databases and docker bind volumes. I have it ping my Healthcheks server at start and finish. ```/etc/restic-password``` is owned by root and only readable by root, for security reasons. Output the results to ```restic.log``` to check later for any errors. My backup script, run daily by root via cron: ``` #!/bin/bash set -e #Cron health start curl -fsS --retry 5 -o /dev/null https://health.domain.tld/ping/xxx- xxx-xxx-xxx/start #Define a timestamp function timestamp() { date "+%b %d %Y %T %Z" } ##Local Backup #Load env export RESTIC_REPOSITORY=/mnt/backup/restic/ export RESTIC_PASSWORD_FILE=/etc/restic-password #Wait sleep 2 #Insert timestamp into log printf "\n\n" echo "---------------------------------------------------------------- ---------------" | tee -a /home/user/backup/restic.log echo "$(timestamp): local started" | tee -a /home/user/backup/restic.log #Wait sleep 2 #Backup Nextcloud restic backup /path/to/nextcloud-data --tag nextcloud #Wait sleep 2 #Backup Docker restic backup /path/to/docker-data --tag docker #Wait sleep 2 # insert timestamp into log printf "\n\n" echo "---------------------------------------------------------------- ---------------" | tee -a /home/user/backup/restic.log echo "$(timestamp): local finished" | tee -a /home/user/backup/restic.log #Wait sleep 2 ##B2 Backup #Load env export B2_ACCOUNT_ID="xxxxx" export B2_ACCOUNT_KEY="xxxxx" export RESTIC_REPOSITORY="b2:Bucket-Name" export RESTIC_PASSWORD_FILE=/etc/restic-password #Wait sleep 2 #Insert timestamp into log printf "\n\n" echo "---------------------------------------------------------------- ---------------" | tee -a /home/user/backup/restic.log echo "$(timestamp): b2 started" | tee -a /home/user/backup/restic.log #Wait sleep 2 #Backup Nextcloud restic backup /path/to/nextcloud-data-data --tag nextcloud #Wait sleep 2 #Backup Docker restic backup /path/to/docker-data --tag docker #Wait sleep 2 # insert timestamp into log printf "\n\n" echo "---------------------------------------------------------------- ---------------" | tee -a /home/user/backup/restic.log echo "$(timestamp): b2 finished" | tee -a /home/user/backup/restic.log #Cron health curl -fsS --retry 5 -o /dev/null https://health.domain.tld/ping/xxx- xxx-xxx-xxx/$? #NTFY notification apprise -t "Server-Name" -b "Backup complete" ``` Exmple of ```restic.log```: ``` ---------------------------------------------------------------------- --------- Oct 13 2022 11:47:04 CEST: local started using temporary cache in /tmp/restic-check-cache-268868756 create exclusive lock for repository load indexes check all packs check snapshots, trees and blobs [1:53] 100.00% 56 / 56 snapshots no errors were found ---------------------------------------------------------------------- --------- Oct 13 2022 11:54:28 CEST: local finished ``` Database backup script: ``` #!/bin/bash set -e # Cron health start curl -fsS --retry 5 -o /dev/null https://health.domain.tld/ping/xxx- xxx-xxx-xxx/start # Nextcloud maintanance mode on docker exec nextcloud occ maintenance:mode --on # Stop containers docker stop photoprism immich-server immich-microservices immich- machine-learning immich-proxy # Wait sleep 2 # Backup nextcloud database docker exec nextcloud-mariadb mysqldump -uroot -p --all-databases > /home/user/docker/database/nextcloud/nextcloud-dump-$(date +%F_%H-%M-%S).sql # Wait sleep 2 # Remove oldest rm "/home/user/docker/database/nextcloud/$(ls -t /home/user/docker/database/nextcloud | tail -1)" # Wait sleep 2 # Backup photoprism database docker exec photoprism-mariadb mysqldump -uroot -p --all-databases > /home/user/docker/database/photoprism-dump-$(date +%F_%H-%M-%S).sql # Wait sleep 2 # Remove oldest rm "/home/user/docker/database/photoprism/$(ls -t /home/user/docker/database/photoprism/ | tail -1)" # Wait sleep 2 # Backup immich database docker exec immich-db pg_dump -U immich immichdb > /home/user/docker/database/immich/immichdb-dump-$(date +%F_%H-%M-%S).sql # Wait sleep 2 # Remove oldest rm "/home/user/docker/database/immich/$(ls -t /home/user/docker/database/immich/ | tail -1)" # Wait sleep 2 # Nextcloud maintanance mode off docker exec nextcloud occ maintenance:mode --off # Start containers docker start photoprism immich-server immich-microservices immich- machine-learning immich-proxy # Cron health curl -fsS --retry 5 -o /dev/null https://health.domain.tld/ping/xxx- xxx-xxx-xxx/$? #NTFY notification apprise -t "Server-Name" -b "DBBackup complete" ``` ---------------------------------------------------------------------- TheFumingatzor · 2023-11-23 19:34 Uuuh...timeshift and borg?? ---------------------------------------------------------------------- Xenkath · 2023-11-23 19:42 Is —reflink supported on ext4 file systems, and does it avoid database corruption from copying during a write? ---------------------------------------------------------------------- alphabuild · 2023-11-23 19:44 Hmm. Never heard of this. But not supported on ext4 it looks like. ---------------------------------------------------------------------- alphabuild · 2023-11-23 19:46 Not sure why you got downvoted. It’s like all of r/selfhosted doesn’t see the value of cloud services. ---------------------------------------------------------------------- krimsonstudios · 2023-11-23 19:52 Proxmox Backup Server (PBS) snapshotting all my VM's / LXC's. External VPS' and anything that can't run PBS-Client I am rsync'ing important data into my home network first, then doing a file based backup of that data to PBS via PBS-Client tool. Those backups then get sync'd to a 2nd datastore for a bit of redundancy. ---------------------------------------------------------------------- Catsrules · 2023-11-23 19:55 Hey that is the plot to First Contact. ---------------------------------------------------------------------- flyingvwap · 2023-11-23 20:08 Kopia has been great. https://github.com/kopia/kopia ---------------------------------------------------------------------- carlinhush · 2023-11-23 20:21 Unraid with Duplicacy and Appdata Backup incremental to Backblaze ---------------------------------------------------------------------- trigger2k20 · 2023-11-23 20:26 I use diplicati to backup to a secure off site location. Useful for something like vaultwarden. https://github.com/duplicati/duplicati ---------------------------------------------------------------------- billiarddaddy · 2023-11-23 20:32 Most of mine are lightweight so private repos on git. For big data I have two NAS that sync on the daily. ---------------------------------------------------------------------- 2lach · 2023-11-23 20:41 Cronjobs to backup important folders to a separate disk Git repo(s) for services & configs with weekly automated commits and pushes ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 20:42 You're not seriously using docker with ext4 instead of XFS? ---------------------------------------------------------------------- dually · 2023-11-23 20:42 Backup with sanoid and then syncoid to the backup server in another building. Every virtual machine image has its own zfs dataset. ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 20:44 No. --reflink is supported on XFS and it's CoW so no corruption possible. Using docker without using XFS robs you of many features of overlay2 like a limit on volume size. Don't use ext4 in 2023, use XFS. ---------------------------------------------------------------------- alphabuild · 2023-11-23 20:47 Yeah. Why shouldn’t I? Only ever used ext file systems for 20 years. Haha. ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 20:48 Are you also using a 56k modem since 20 years? Use XFS. ---------------------------------------------------------------------- alphabuild · 2023-11-23 20:55 Hit me up on AOL Online so we can discuss further ---------------------------------------------------------------------- Theon · 2023-11-23 21:11 As others said, use volume mounts, and I incrementally backup those with [borg](https://www.borgbackup.org/) to minimize storage space requirements ---------------------------------------------------------------------- Xenkath · 2023-11-23 21:12 I run my docker containers in Alpine VMs on Proxmox backed up via Proxmox Backup Server. For persistent storage I mount host directories, not docker volumes. I don’t see any benefit to redeploying docker on XFS in my case. ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 21:14 That okay, it not like docker specifies to use overlay2 with XFS or anything like this. ---------------------------------------------------------------------- bufandatl · 2023-11-23 22:03 I use rdiff-backup to backup the volumes directory of my VPS to a local machine via VPN. Containers are stored in some public registry anyways. Also use ansible with all the configurations and container settings. ---------------------------------------------------------------------- iammilland · 2023-11-23 22:50 I do understand why :) it is the concern about google that most selfhosters have. selfhosting is a way not to use the consumer based services. When i recommend google for backup is kind of silly i get it. But google bucket has noting to do with the consumer or business suite that google offers. this is i a backend like s3 used for devs, you can create you own keys or even encrypt you backup at 0.0012$ pr GB this is a no brainer ---------------------------------------------------------------------- unconscionable · 2023-11-23 22:51 I often end up with corrupted sqlite databases when I try to hot copy. They are generally recoverable, but still seems problematic. Can tools like reflink avoid that somehow? ---------------------------------------------------------------------- ElevenNotes · 2023-11-23 22:59 Yes, because –reflink uses CoW from XFS, ZFS or BTRFS. ---------------------------------------------------------------------- mosaic_hops · 2023-11-23 23:39 I do the reverse… all configs are ansible scripts and files and I just push them to the servers. That way I can spin up a new machine from scratch, completely automated within minutes… just the time it takes the machine to set itself up. ---------------------------------------------------------------------- ManiacMog · 2023-11-23 23:56 I use Kopia. The cli is very easy to use and I have backups scheduled nightly. I backup all external mounts and my entire Portainer directory. Has helped in a pinch to restore busted databases. I point Kopia cli to backup to a WebDAV location I host on my NAS. For off-site backups I run daily backups of that kopia repository to Google cloud. I'm not sure if Google cloud is the best off-site backup solution, but I did a price comparison when I first selected it and it was the best capacity for the price that I could find at the time. ---------------------------------------------------------------------- pbrom · 2023-11-24 00:00 I.... don't. No space for more than my Thinkpad Proxmox server and one VPS in the cloud. I also save on coffee bc anytime something seems slightly off, my HR spikes off the charts. ---------------------------------------------------------------------- modscleo4 · 2023-11-24 00:11 Almost everything I run is a Docker container, so I made `/var/lib/docker` a btrfs subvolume, and then I make every day incremental snapshots (cron job) and copy them to a secondary disk (also btrfs, using btrbk). Since they are btrfs snapshots they don't use a lot of disk space and if I really need to rollback an entire day I can ---------------------------------------------------------------------- maretoni · 2023-11-24 00:17 rsync ---------------------------------------------------------------------- maretoni · 2023-11-24 00:18 this, plus rsync! ---------------------------------------------------------------------- DentedZebra · 2023-11-24 00:18 Haven't used digital ocean but I run 2 Proxmox servers and 2 NAS one of each at a different location. I backup containers and VMs which run in Proxmox to the NAS via NFS and then have a nightly script to copy the backups from there to my remote NAS. It works, haven't lost any data yet. Still thinking about a third backup in another location as well but money is a thing 🤷 ---------------------------------------------------------------------- Service-Kitchen · 2023-11-24 00:39 What’s Reflink? ---------------------------------------------------------------------- CactusBoyScout · 2023-11-24 02:40 How would I go about writing said script? I’ve just been backing up my containers while they’re running. ---------------------------------------------------------------------- clericc-- · 2023-11-24 02:47 Borgbackup, borgmatic to two backup targets: one in my home and a Hetzner Storage Box. Amongst other things, i include /var/lib/docker/volumes, covering the not-filesystem-bound mounts. ---------------------------------------------------------------------- eidetic0 · 2023-11-24 03:34 For great priced bucket storage that isn’t one of the big players (Amazon, Google or MS) Backblaze is worth checking out. Their B2 bucket storage is S3 compatible and they are a very transparent company. I have had my backups and git lfs stored there for years. ---------------------------------------------------------------------- Juxsta0 · 2023-11-24 03:54 I use an Ubuntu vm for all my containers in proxmox and make backups of the vm ---------------------------------------------------------------------- TheBezac · 2023-11-24 06:07 Use [resticker](https://github.com/djmaze/resticker/) to add an additional backup service to each compose allowing me to customize some pre/post backup actions. Works like a charm 👍 ---------------------------------------------------------------------- iammilland · 2023-11-24 06:10 Yes, but at 6$ a tb with backblaze it is almost 6x the price for archival at Google buckets 😊 ---------------------------------------------------------------------- stefantigro · 2023-11-24 06:13 I have a kubernetes cluster, so I use https://github.com/vmware- tanzu/velero. Essentially it backs up whole volume mounts (persists t volume claims) of the containers. Does so periodically during quiet hours. This is the schedule https://github.com/Michaelpalacce/HomeLab/blob/ma ster/cluster/homelab/configs/velero/schedule.yaml and here is an example of how a pvc is defined for backup ( look at the annotations) https://github.com/Michaelpalacce/HomeLab/blob/master/Helm/apps/postgr esql/templates/postgresql.yaml ---------------------------------------------------------------------- armaver · 2023-11-24 07:09 Borg Backup to Hetzner Storage Box. ---------------------------------------------------------------------- tech2but1 · 2023-11-24 08:03 Not sure where you're getting prices from but this is wildly inaccurate, and there's more to it than that anyway. Google charges in tiers, B2 charges for data used so Google Buckets for me would be more expensive than My $2-3 I spend on B2 per month for some hundreds of GB. ---------------------------------------------------------------------- Jacksaur · 2023-11-24 08:58 This is exactly what I'm setting up with Borg right now. What kind of retention do you run though? I'm doing daily backups, and considering 7 daily copies (for easy restoring through the week) and then potentially 2-3 monthly ones just in case there's a real long term issue that I missed. ---------------------------------------------------------------------- Jacksaur · 2023-11-24 08:59 What retention do you run? I'm setting up the same system, but don't know how far back I need. Currently considering 7 daily backups, so I can restore to any point within the week, and 2-3 Monthly backups in case there's an issue I miss for a real long period. ---------------------------------------------------------------------- iammilland · 2023-11-24 09:01 From the official websites. 0.0012$ fo archive in one region, 6$ at blackblaze pr 1 tb. Ingress is free, at Google it cost to retrive but that is not the point with a backup. https://www.backblaze.com/cloud-storage/pricing and https://cloud.google.com/storage/pricing#asia ---------------------------------------------------------------------- tech2but1 · 2023-11-24 09:07 You've misunderstood entirely. $6 per TB yes that's right, but with Google you pay per TB regardless from what I have seen. I do not have 1TB so do not pay $6, I pay per GB. You are ignoring the way Google prices their buckets to make it look like Backblaze is more expensive so I can only assume you are a Google shill. ---------------------------------------------------------------------- clonecharle1 · 2023-11-24 09:57 A few hard drives that are stored offsite and rotate every few weeks. ---------------------------------------------------------------------- yowzadfish80 · 2023-11-24 10:22 Everything running on Proxmox Virtual Environment in one house. Backing up daily to Proxmox Backup Server running in second house. ---------------------------------------------------------------------- clericc-- · 2023-11-24 11:11 entirely up to your feelings i guess - i run 7 dailies and 2 weeklies ---------------------------------------------------------------------- Xenkath · 2023-11-24 14:53 I keep daily copies for a week, weekly copies for a month, and monthly copies for a year. ---------------------------------------------------------------------- sapl84 · 2023-11-24 15:11 rsnapshot to pi-based backup system with Daily, Weekly, Monthly and Yearly Backups. That system is Backed up via borg, encrypted and rsynced to a NAS of a friend (offsite-backup). He does sync his encrypted Data to me in reverse. ---------------------------------------------------------------------- king_hreidmar · 2023-11-24 15:16 I just run a pg_dump through kubectl exec and pipe the stdout to a file on my master node. The same script then runs restic to send encrypted backups over to s3. I use the host name flag on the restic command as kind of a hack to get backups per service name. This eliminates the risk of overwriting files or directories with the same name. ---------------------------------------------------------------------- Hammertoggl · 2023-11-24 16:32 I backup all the mounted docker volumes once every hour (snapshots). Additionally i create dumps from all databases with https://github.com/tiredofit/docker-db-backup (once every hour or once a day depending on the database). ---------------------------------------------------------------------- cbunn81 · 2023-11-24 17:17 ZFS snapshots. ---------------------------------------------------------------------- fjch1997 · 2023-11-24 18:00 duplicati to take live, crash-consistent backups of all my windows servers and VMs with Volume Shadowcopy Service (VSS) ---------------------------------------------------------------------- lgLindstrom · 2023-11-24 19:38 When backing up Docker volumes, should not the docker container be stopped first.?? I can't se any support for that in the backup tools mentioned. ---------------------------------------------------------------------- Minituff · 2023-11-24 20:36 Yes the containers do need to be stopped. I actually built a [project](https://github.com/Minituff/nautical-backup) that does exactly that. ---------------------------------------------------------------------- Minituff · 2023-11-24 20:39 I use [Nautical](https://github.com/Minituff/nautical-backup). It will stop your containers before performing an RSYNC backup. ---------------------------------------------------------------------- eidetic0 · 2023-11-25 00:11 Interesting… I had a look at the Google cloud ‘Archive Storage’ which I think is what you’re referring to at $0.0012/Gb. Regular Google Buckets look a lot more expensive than Backblaze. So I guess it’s comparing a very different storage class. Slower access compared to regular bucket storage? Maybe something similar to Amazon Glacier? Perfect for long term backups though I assume. It’s a good price if you don’t need any kind of frequent or quick access. ---------------------------------------------------------------------- Kltpzyxmm · 2023-11-25 03:13 I have bind mounts to nfs shares that are backed my zfs pools and last snapshots and sync jobs to another storage device. All containers are ephemeral. ---------------------------------------------------------------------- lgLindstrom · 2023-11-25 06:43 Thanks,, I will look into this. ---------------------------------------------------------------------- iammilland · 2023-11-25 07:46 It is like glacier, as you say only for backups that you dont need to acces to, i think you are right about the speed, but it not something that bad. In my case i hope i never have to. 😊 but the speed sending data will max my 1gbit connection ---------------------------------------------------------------------- CarefulNerve8923 · 2023-11-25 08:55 On Proxmox i use for my Backup Solution - Hetzner Storage Bix ---------------------------------------------------------------------- CarefulNerve8923 · 2023-11-25 08:55 Box* ---------------------------------------------------------------------- Ok-Power7033 · 2023-11-27 08:18 For databases and data I use [restic-compose- backup](https://github.com/ZettaIO/restic-compose-backup) because you can use labels in your docker compose files. For config files I use a git repository. ---------------------------------------------------------------------- Mr_Brightstar · 2023-12-09 19:26 > Google cloud ‘Archive Storage isn't duplicati on an eternal Beta state?