best services-coldspare-clone setup ? reddit / selfhosted ---------------------------------------------------------------------- Thutex · 2023-12-16 17:13 So, i have a few services running, which from time to time change places (i.e. hosted at home, hosted on a server i put in a DC somewhere, hosted on a VM i have somewhere, ....) and so far this has been fine for years - never had a real issue, and i hope that stays this way. HOWEVER.... i have come to a point where "i would feel much better' if i knew these systems were cloned to another system and all i'd have to do to restore services, if the current ones go down, would be pointing the dns somewhere else. (also used to self-host dns, but that has been retasked to cloudflare because, why not) so what i'm looking for is basically: \- something that runs on the current systems (mostly VMs) \- knows when something changes and can copy that change to another VM somewhere else (so, ssh/sftp would be the best option to transfer/update) \- can make sure that these changes are correct (i.e. if an IP would be different, this would have to be updated in the copied config) i know i could go overkill and setup a proxy somewhere and have that point to the services, then link the vm's over one of the wireguard options, and use rsync between them.... but i'm hoping there is an "easier, less maintenance requiring option" ---------------------------------------------------------------------- BraveNewCurrency · 2023-12-16 21:55 There is a spectrum of ideas you can use: 1) Something like Puppet/Chef/Ansible to "declare" what your server setup looks like. These are used less and less as people move to Docker containers. But sometimes they can still be used to configure inside your VMs. 2) Something like Terraform can manage "infrastructure" like DNS or network configs. 3) You want to backup your data. But also have a way to "test" that your backups work. Maybe periodically try restoring to a backup server. (Inherent in this: you probably don't need to backup your software. Just capture how to install it from #1.) ​ Load balancers are more complex (and cost money to run). You would only pay that price if you want to eliminate the "downtime" during switch-over. Very likely, being down for 30m or an hour, or even a day is acceptable to you. Make sure you set the TTL on DNS to slightly less than the expected fail-over time. ---------------------------------------------------------------------- Thutex · 2023-12-16 22:48 load balancing is not my idea, and a certain downtime is certainly acceptable. but i would like to not use the concept of "backup" for my services - but rather the cold spare concept: i want to have another machine (other datacenter, other provider) constantly being up to date (with an interval of say 6-12h, which is generally an acceptable amount of dataloss for most of these services) with the main setup. then, if something happens to the main setup, instead of having to recreate everything and then get the data from backups, i want to just manually reconfigure dns to go to the alternate system and be up and running again after a dns cache update. so... that kind of excludes things like terraform, regular backup solutions, and most other items in that category (though i assume ansible might be partially the answer - but that would then require something that's running ansible, and would create a new dependency) ---------------------------------------------------------------------- BraveNewCurrency · 2023-12-17 00:13 >that kind of excludes things like terraform, regular backup solutions I would not say it excludes them. I might agree that they are slightly more complicated and maybe not necessary. Version one could just be "rsync over to another host every night". (If you use BTRFS, you can send snapshots, which is even better.) ​ >i assume ansible might be partially the answer - but that would then require something that's running ansible, and would create a new dependency Yes, it's a trade-off: You have to "do everything" in Ansible to get the benefits of using Ansible. But then the benefit is that you can replicate the installation of all your software trivially. If you don't want to learn Ansible, the next idea would be to put everything in Docker containers. Then you can just copy around some tiny `docker-compose.yaml` files, plus your backups. (You have to decide if you want your data "inside docker", or just volume-mount existing folders from outside Docker to make it easier to backup. The latter has worse performance if you are not on Linux.) If you don't like that, just get good at creating your VMs via "OS+application install scripts" vs "data". Only backup the data, and have a small script create your OS+applications in VMs. Periodically toss your VMs and ensure you can re-create the OS+application and run with your data snapshot. The worst is to try and backup 20 VMs every night, each having a 10GB filesystem..