What do you use for VPS backup? Would improved borg setup - pull mode - be enough? Or, do you use something else? reddit / selfhosted ---------------------------------------------------------------------- mirokra · 2023-12-02 08:35 I've got an app running on a VPS. And, I need to backup `pg_dump` output, and simple files (static, and user uploads). Currently, I'm auto-backing it up with [borg](https://www.borgbackup.org/) (push mode) through wireguard tunnel to NAS behind ISP's CGNAT. The borg takes care of deduplication in SQL file, so incremental update (even in append-only mode) is very small for PostgreSQL dump. What already went wrong? The wireguard tunnel failed, and VPS couldn't push backups for some decent amount of days. The [attack model of borg backup](https://borgbackup.readthedocs.io/en /stable/internals/security.html) is trusted client (VPS being backed up) and untrusted repository (NAS): >The attack model of Borg is that the environment of the client process (e.g. borg create ) is trusted and the repository (server) is not. The attacker has any and all access to the repository, including interactive manipulation (man-in-the-middle) for remote repositories. This doesn't fully suit attack model on VPS, because if VPS gets compromised, then attacker can try to attack borg backup stored in NAS, and the whole NAS itself, via the tunnel. I've mitigated the issue by setting repository to [append-only mode](h ttps://borgbackup.readthedocs.io/en/stable/usage/notes.html#append- only-mode-forbid-compaction), which should protect backup from being corrupted by compromised VPS. The VPS could still try to attack the NAS itself, because a wireguard tunnel is up between VPS and NAS. I would like to get rid of VPS being able to access NAS on its own. Yes, there's firewall on NAS, but no access (to initiate connection) is better than firewall-ed access. So, I would like to get rid of wireguard tunnel. Full attack model I would like to be protected against: * untrusted (compromised) VPS, * untrusted (compromised) NAS, * untrusted (compromised) machine of the (sole) admin - myself. * Gaining access to my machine with key-logger would eventually be able to reach both NAS and VPS, and kill both at the same time. Note, the MFA would help here, but it's a separate topic. So, I plan to add offline backups to the mix. I found [pull mode](https://borgbackup.readthedocs.io/en/stable/deployment/pull- backup.html) of backup via borg. The socat and ssh-agent options seem like pull-push, initiated (pull) by machine storing backup, passing temporary tunnel to machine being backed up, which then pushes backup via this temporary tunnel. The pull mode should enable me to get rid of the wireguard tunnel. Also, pull mode should enable me to do sporadic offline backups to external HDD manually, when I'm sure my machine is not compromised. Ideally, rotate 2 external HDDs, so there's at least one backup offline at any/every point in time. The password for the borg's repository would still be stored on VPS, and not on NAS. If VPS is compromised, it doesn't matter anyway, because attacker would have access to all the data anyway. However, compromise of NAS won't result in gaining access to unencrypted backup data. Same for offline backup to external HDD. I am wondering, whether to put effort into reworking things to pull mode of borg backup, or to go with something else completely. Does anybody have experience with borg backup in pull mode? Does it deliver good results? Or, is there a much better tool to do these backups and protect against the above attack model? ---------------------------------------------------------------------- Scary-Maple · 2023-12-02 09:08 I wouldn’t use a VPS provider that doesn’t do backups (or allow you to export those backups). Really depends on what is on the VPS, but you are better off completely automating the deployment of the services running (including config - or have well documented steps). If persistent data is needed, store it outside the VPS (S3-compat storage, block storage from VPS provider, etc). Basically, your compute node should be ephemeral and anything persistent should be elsewhere. ---------------------------------------------------------------------- lilolalu · 2023-12-02 09:10 I use Borg and rsnapshot ---------------------------------------------------------------------- lilolalu · 2023-12-02 09:13 https://www.datacenterdynamics.com/en/opinions/ovhclouds-data-center- fire-one-year-on-what-do-we-know/ ---------------------------------------------------------------------- Scary-Maple · 2023-12-02 09:24 I don’t know what point you are trying to make. Posting just a link is not helpful. If your complaint is that you shouldn’t rely on a VPS provider because their data centre might burn down, you have heard of geographically- disperse infrastructure, right? In my examples, you could have the S3-compatible bucket be literally anywhere, it’s just an API endpoint that you upload and download files from. Doesn’t even have to be the same provider. ---------------------------------------------------------------------- lilolalu · 2023-12-02 09:29 Ah so you don't only want a VPS provider that does backup but in "geographically disperse infrastructure" that's a lot of expectation and trust, just to not take care of it yourself. Also you would have to make sure that the providers backup is as actually working and just an empty promise. OVH datacenter burning down was a practical case where a lot of people completely lost their data, because not only the servers where gone, but the backups etc. ---------------------------------------------------------------------- Scary-Maple · 2023-12-02 09:54 You missed a couple of parts. 1. Do not store data locally on your compute node, which should be ephemeral and (re)buildable in an automated manner. 2. You can backup data stored in S3-compatible storage (just connect another client and download the data, I’m sure OP’s NAS could do that. 3. I never said to rely only on your VPS provider. You can use however many you want to meet your resiliency and uptime requirements. An example architecture using OP’s details: pg_dump: 1. Use mounted block storage for database directory (really depends on the provider’s network and storage, optional) 2. Dump to mounted block storage from provider 3. Upload the zipped dump to S3-compat storage (you’d probably want to encrypt this first, maybe Borg can do this?) - this may also be versioned 4. (Optional) Pull a copy from your S3-compatible storage somewhere else 5. Test all backup locations routinely to ensure data can be restored This protects against the following: 1. Malicious user on compute - unable to modify versioned backups 2. Catastrophic provider failure (VPS: S3 provider), (S3 provider: VPS/tertiary backup), (Both: tertiary backup) 3. User error (VPS deletion, network misconfiguration) Static files/User Uploads: Pretty similar, but if your app supports S3 storage directly, use it. Less overhead and easier to automate redeployment. Honestly though, it really just depends on what this data is and its threat model. Nextcloud and everyone in your family is backing up phone photos? Yeah, 3 copies would be ideal, preferably with one offline. A bunch of memes? Eh, probably not worth the expense. Linux ISOs? Definitely not worth the expense. ---------------------------------------------------------------------- lilolalu · 2023-12-02 09:55 Ok ---------------------------------------------------------------------- mirokra · 2023-12-02 14:29 What does rsnapshot give you over borg? And, do you use borg in push or pull mode? > I think you are overthinking the "attack scenarios". More probable is ransomware, someone accidentally or on purpose deleting your data, datacenter burning down. Well, maybe. I just want to make it "better safe than sorry". At least, I definitely want to add offline backup to the mix. I don't want to loose years of data, with a guilt, that it could have been avoided, if I did backup properly. Wondering, whether to go to pull mode with borg backups. Or, maybe, just copy borg repository to an external HDD manually, from time to time. ---------------------------------------------------------------------- lilolalu · 2023-12-02 14:32 Its just a script for rsync, so it has all the simplicity of rsync. It doesnt have a lot of the nice features of borg like deduplication or repository encryption. but depending on your use case, simple can be an advantage imo. ---------------------------------------------------------------------- FeelingPapaya47 · 2023-12-02 18:29 I also backup my VPS via WireGuard to my NAS via [Borgmatic](https://torsion.org/borgmatic/) (push). Although it has never failed for me, I would get notified if it happens by [Healthchecks.io](https://healthchecks.io) because I call it in the `after_backup` section of the Borgmatic `config.yaml`. If you want to use pure borg without Borgmatic you can also append the Healthchecks- call to your `borg` Command with `&&`. E.g.: `borg create ... && curl https://healthchecks.io/XXX`. If the `borg` command fails, the command behind the `&&` is not executed, therefore no ping is sent to Healthchecks.io. Healthchecks.io will then send you an email that an expected ping did not occur. You can also selfhost it if you want. I also do not SSH directly into my NAS. I use an [OpenSSH- Container](https://docs.linuxserver.io/images/docker-openssh-server/). This container has only the backup-folder on the NAS as a mount point. So if someone were to hijack the SSH login, they would only see the backups of the VPS and nothing else. You ideally also have a backup of the NAS itself, so even if some tampers with that repo, you still have your backups. You could achieve a similar thing with permissions on the NAS but I like the extra layer of a container inbetween. In the `before_backup` section of Borgmatic I also have this line: `ssh HOSTNAME -l USER -p PORT -i /root/.ssh/key.pem sudo apk add borgbackup`. This installs borg inside the Alpine-based OpenSSH- Container. It's not necessary but makes borg faster. The install is only required after rebuilding the container but I just keep in there so I don't have to worry about it after updating. You also mention you use `pg_dump` for database backups. Perfectly fine, just a heads up, [Borgmatic supports this as well](https://torsion.org/borgmatic/docs/how-to/backup-your- databases/), so no need to do it manually. ---------------------------------------------------------------------- mirokra · 2023-12-02 19:22 Thank you for many hints and ideas. >I also backup my VPS via WireGuard to my NAS via Borgmatic (push). Didn't track what caused it, but I got infinite logs of: Temporary failure in name resolution: `hostname.com:54321'. Trying again in 20.00 seconds... Even if domain name was resolvable by `dig`, when I ssh-ed to the machine. >Although it has never failed for me, I would get notified if it happens by Healthchecks.io because I call it in the after\_backup section of the Borgmatic config.yaml. Thanks, good idea. I'm using just plain shell script on client (VPS) side, periodically executed by crontab. So, should be straightforward to add a healthcheck to it. >I also do not SSH directly into my NAS. I use an OpenSSH-Container. This container has only the backup-folder on the NAS as a mount point. So if someone were to hijack the SSH login, they would only see the backups of the VPS and nothing else. I am running borg (separate openssh) in a container, mounted only backup directories, too. Though, I'm using just [NixOS container](https://nixos.wiki/wiki/NixOS_Containers), which is based on native systemd-nspawn containers. AFAIK, less secure than docker, but container escape is possible only via root user in container, which client (VPS) won't get anyway. Client (VPS) does ssh to this container, but I'm using [borg serve](https://borgbackup.readthedocs.io/en/stable/usage/serve.html) with forced ssh commands. Those are generated by NixOS's options - [se rvices.borgbackup.repos](https://search.nixos.org/options?channel=23.1 1&from=0&size=50&sort=relevance&type=packages&query=services.borgbacku p.repos) \- and, using `authorizedKeysAppendOnly` for the append-only mode. Though, if a bug in borg would enable attacker to escape repository and change system files, that would allow client to ssh as root in borg container, then attacker could escape the container. Wondering, whether such sophisticated (automatic) malware even exists. It would maybe require a human hacker to pull such thing, and existence of such bug. However, if wireguard is up, then the whole NAS is reachable from the other side of tunnel. This extends attack surface. Though, maybe I could run wireguard in a container for borg backups. >You also mention you use pg\_dump for database backups. Perfectly fine, just a heads up, Borgmatic supports this as well, so no need to do it manually. From what I've read, `pg_dump` is a valid approach for backup of (non- scaled, non-replicated) PostgreSQL, because it is atomic (edge-cases only on schema changes), and it won't backup any effects of half-done transactions, that weren't committed before backup started. ---------------------------------------------------------------------- ChadTheAssMan · 2023-12-02 23:27 restic ---------------------------------------------------------------------- mirokra · 2023-12-03 06:09 Do you use it for VPS backups to offline media? Restic is push mode, same as Borg. So, I guess, VPS can't push backup to locally (temporarily) mounted HDD. ---------------------------------------------------------------------- ChadTheAssMan · 2023-12-12 12:56 You can backup to a local device, yes. I push backups to a cloud block store