[HN Gopher] Let's Talk OpenZFS Snapshots
___________________________________________________________________
Let's Talk OpenZFS Snapshots
Author : rodrigo975
Score : 114 points
Date : 2021-07-29 09:03 UTC (13 hours ago)
(HTM) web link (klarasystems.com)
(TXT) w3m dump (klarasystems.com)
| infogulch wrote:
| Can you somehow associate snapshots from separate machines? Lets
| say I have two servers that provide services for an app, one runs
| a postgresql db and the other is just a front for a directory
| that stores attachments (it's an example ok). Both servers are
| running ZFS and each service is running on its own volume which
| has regular snapshots taken. If I'm not too worried about the
| snapshots being in _perfect_ sync (its ok if I have an extra
| attachment or have to pull an attachment from the next snapshot;
| I 'm not asking for a solution to the distributed consensus
| problem here), is there some particular way I should store or
| manage the snapshots? Or do I need to write some custom scripts
| to associate the snapshots and restore them together in case of a
| recovery situation?
| justinsaccount wrote:
| snapshots are usually just named based on the date, like
| 2021-07-29-10-hour 2021-07-29-10-53-minute
|
| or similar. Then it's just a matter of ensuring that you have
| NTP working and your clocks are in sync.
| pawelduda wrote:
| Does it make sense to snapshot the entire root dir or just the
| home directory? Asking about desktop PC.
| tiernano wrote:
| Probably home did, maybe etc and var, maybe other important
| dirs... full root would include temp and logs that may not be
| needed and folders that don't change often
| vermaden wrote:
| Yes, like that: https://is.gd/BECTL
| solarengineer wrote:
| For Solaris 11 and for Illumos based distros, snapshotting the
| entire root is normal. That way, with an OS update and
| application update, the logs are associated with that
| particular root file system. You can always look into earlier
| snapshots to learn of what the logs contained for that earlier
| root file system.
|
| With Boot Environments (I don't know if Linux based Distros let
| one do this) on Solaris and Illumos, we have multiple active
| root file systems to switch between and can mark snapshots for
| each of them.
|
| Certainly, for Belenix (KDE based opensolaris distro), we
| snapshotted the entire root file system.
| 5wp3 wrote:
| This is implemented for Arch Linux with bieaz or zbm; and
| Ubuntu with zsys.
|
| bieaz: https://openzfs.github.io/openzfs-
| docs/Getting%20Started/Arc... zbm: https://github.com/zbm-
| dev/zfsbootmenu zsys: https://github.com/ubuntu/zsys
| throw0101a wrote:
| > _For Solaris 11 and for Illumos based distros, snapshotting
| the entire root is normal._
|
| FreeBSD stole this feature from Solaris as well:
|
| * https://www.freebsd.org/cgi/man.cgi?beadm
|
| * https://wiki.freebsd.org/BootEnvironments
| vermaden wrote:
| ... and Illumos stole Bhyve and Bootloader from FreeBSD? :)
|
| IMHO implemented or ported is better word here.
| joshspankit wrote:
| I tuned out of ZFS when it was being designed, ducked my head in
| a few years later to read about 8GB+ RAM requirements, and have
| not tuned back in properly (don't kick me off HN lol)
|
| My question is on snapshots vs file-specific backups: If a folder
| is set up for non-destructive writes, does it then _need_
| snapshots? Is there a benefit to using both snapshots and NDR in
| the field?
| unixhero wrote:
| You need a lot of ram for de-duplication. It is not needed for
| general use. 128 bits man, you're missing out!!!
| kissgyorgy wrote:
| ZFS snapshots saved my ass multiple times after a bad Ubuntu
| upgrade. It's pretty easy to roll back.
| unixhero wrote:
| How do you roll back after a botched OS update then? Boot into
| a rescue cd?
| vermaden wrote:
| Like that: https://is.gd/BECTL
|
| By using ZFS Boot Environments.
| unixhero wrote:
| Woot, thanks a lot.
| mvanbaak wrote:
| 'In this article, we'll learn how ZFS boot environments (BEs) are
| used to boot the operating system itself into a previous point in
| time.'
|
| The whole article does not touch the subject of boot environments
| ...
|
| Maybe the author can update the page and state that the article
| is about managing snapshots instead of boot environments?
| vermaden wrote:
| Here: https://is.gd/BECTL
| beermonster wrote:
| I read the entire article as I was interested in that aspect
| too. Misleading title.
| vermaden wrote:
| Here: https://is.gd/BECTL
| 1MachineElf wrote:
| A common pattern with beadm on FreeBSD is for the root
| filesystem dataset to be called "ROOT", and this mirrors
| what's in the main article. It appears this person has boot
| environments enabled, but didn't actually explain anything
| about them.
|
| ZFS boot environments are awesome. You can get them on both
| FreeBSD and Illumos with beadm. On Linux there are a number
| of utilities, but I've only used ZSys. It comes working out
| of the box if you do a ZFS-on-root install with Ubuntu.
| 5wp3 wrote:
| Boot environment is just a fancy term for root filesystem
| clones created from snapshots.
|
| Fyi: snapshots are read-only. Normal, read-writable filesystem
| can be created by `zfs clone root@snap newroot`.
| barrkel wrote:
| I make snapshots every hour for 24 hours, every day for 7 days,
| every week for 4 weeks and every month for three months. This
| keeps the number of snapshots reasonable while keeping most stuff
| around should you accidentally delete or overwrite it. I can see
| a good argument for more fine grained snapshots (e.g. every few
| minutes) but I think I'd limit that to directories (and thus
| FSes) known to have frequent updates.
|
| I sometimes blow away all snapshots if I have a big clear out or
| rewrite of data, or else that stuff sticks around for months.
| Siecje wrote:
| Can you take snapshots more often than hourly? For example can
| you take a snapshot every 15 minutes?
| ptomato wrote:
| Yup; you could do every minute with no real problems if you
| like. I believe you start to run into performance issues with
| high (1000+) snapshot counts, and then mostly when actually
| doing snapshot operations, so you'd want to have something to
| trim snapshots so you only keep your minutely snapshots for
| an hour and so forth.
| throw0101a wrote:
| > _I can see a good argument for more fine grained snapshots
| (e.g. every few minutes) but I think I 'd limit that to
| directories (and thus FSes) known to have frequent updates._
|
| Fun fact, the ZFS send-receive functionality was born with
| snapshots being created every few minutes in mind:
|
| > _ZFS send and receive wasn't considered until late in the
| development cycle. The idea came to me in 2005. ZFS was nearing
| integration, and I was spending a few months working in Sun's
| new office in Beijing, China. The network link between Beijing
| and Menlo Park was low-bandwidth and high-latency, and our NFS-
| based source code manager was painful to use. I needed a way to
| quickly ship incremental changes to a workspace across the
| Pacific. A POSIX-based utility (like rsync) would at best have
| to traverse all the files and directories to find the few that
| were modified since a specific date, and at worst it would
| compare the files on each side, incurring many high-latency
| round trips. I realized that the block pointers in ZFS already
| have all the information we need: the birth time allows us to
| quickly and precisely find the blocks that are changed since a
| given snapshot. It was easiest to implement ZFS send at the DMU
| layer, just below the ZPL. This allows the semantically-
| important changes to be transferred exactly, without any
| special code to handle features like NFSv4 style ACLs, case-
| insensitivity, and extended attributes. Storage-specific
| settings, like compression and RAID type, can be different on
| the sending and receiving sides. What began as a workaround for
| a crappy network link has become one of the pillars of ZFS, and
| the foundation of several remote replication products,
| including the one at Delphix._
|
| * https://www.delphix.com/blog/delphix-
| engineering/zfs-10-year...
| jiggawatts wrote:
| > rsync ... incurring many high-latency round trips.
|
| If I remember correctly, the rsync protocol is actually very
| nearly optimal in terms of minimising round trips.
| ptomato wrote:
| If you have to actually compare the files on both ends,
| however, that's still many more round trips than necessary
| for zfs send, which is just sending all blocks with a birth
| time after the previous snapshot
| ptomato wrote:
| It remains a _delightfully_ straightforward way to do
| backups; I do incremental zfs send with hourly snapshots to
| my local NAS and also to an rsync.net ZFS account
| (https://www.rsync.net/products/zfsintro.html); it's
| incredibly resource un-intensive relative to your traditional
| scan-everything backup methods and I can trivially mount a
| past snapshot for browsing or spot restore or just do zfs
| send on the whole thing for catastrophic recovery.
| throw0101a wrote:
| As an added bonus, with (Open)ZFS encryption, you can send
| the 'raw' encrypted _zfs send_ stream (over an encrypted
| SSH connection) to a destination where the file system is
| encrypted and where the _zfs receive_ command does not have
| to decrypt it to save things.
|
| That is, the remote destination does not need to have the
| key/passphrase to the file system to have an (encrypted)
| copy of the data.
|
| If your production server goes down, you can restore the
| encrypted file system _without the encryption key_ , and
| only when you try to mount the restored ZFS file system
| will you be prompted for the key/passphrase.
|
| > _This means that you can use ZFS replication to back up
| your data to an untrusted location, without concerns about
| your private data being read. With raw send, your data is
| replicated without ever being decrypted--and without the
| backup target ever being able to decrypt it at all. This
| means you can replicate your offsite backups to a friend 's
| house or at a commercial service like rsync.net or zfs.rent
| without compromising your privacy, even if the service (or
| friend) is itself compromised._
|
| * https://arstechnica.com/gadgets/2021/06/a-quick-start-
| guide-...
| cyberpunk wrote:
| Do you happen to know if rsync.net supports this?
| throw0101a wrote:
| They support zfs send-recv and snapshots:
|
| * https://www.rsync.net/products/zfsintro.html
|
| * https://www.rsync.net/resources/howto/snapshots.html
|
| Whether their ZFS implementation supports OpenZFS
| encryption is something you'll have to e-mail them about.
| They have an HN account:
|
| * https://news.ycombinator.com/user?id=rsync
|
| Edit: Yes, it seems that they do:
|
| * https://news.ycombinator.com/item?id=26506263
| rsync wrote:
| "Do you happen to know if rsync.net supports this?"
|
| Yes, we do.
|
| If you get a zpool from us (which is a bit different than
| a "normal" rsync.net account) it is running the latest
| stable ZoL codebase and, thus, supports encryption and
| "raw send", etc.
|
| _However_ ...
|
| As elegant and performant as 'zfs send' is, the benefits
| are realized in large datasets where efficiency and
| performance really matters. If you're just sending 200 or
| 400 or 800 GB of data to an untrusted destination (like
| rsync.net) you should probably just use borg[1][2].
|
| [1] https://borgbackup.readthedocs.io/en/stable/
|
| [2] https://www.stavros.io/posts/holy-grail-backups/
| cyberpunk wrote:
| Hmm, well currently all my machines, phones etc backup to
| a nas (zol) which I then reclone up to an encrypted b2
| bucket, but tbh the restores suck and they charge for
| traffic so I was contemplating just throwing em on
| rsync.net, you recon Borg->rs.n is the way to go instead
| of zfs send? Under 2tb..
| rsync wrote:
| Well, it's really up to you but let me put it this way
| ...
|
| In order to give you a zpool of your own we need to give
| you a full blown VM (bhyve) with resource guarantees and
| your own IP address, etc.
|
| So there is a 1TB minimum and _no discounts_.
|
| Alternatively, if you just get a plain old rsync.net
| account the minimum account size is much smaller and if
| you're an expert and don't need any (borg specific)
| support there is a discounted plan[1].
|
| [1] https://rsync.net/products/borg.html
| infogulch wrote:
| Interesting I wasn't aware of the borg discount pricing,
| that may change my default position. I now know of three
| pricing schemes, basic, ZFS, and borg, the last two of
| which don't seem to be listed anywhere on the site. Got
| any other products hiding behind my ignorance? :)
|
| If OpenZFS supported some kind of "virtual zpool" feature
| (I guess it doesn't exist yet) that you could provision
| out of your main ZFS infrastructure and hand off to the
| customer, would that be useful to reduce the resource
| overhead related to offering ZFS directly?
| rubatuga wrote:
| It's so handy! And I have personally tested a restore using
| it. It's a good idea to check out the read-only flags so
| you don't accidentally touch the file-system and invalidate
| an incremental backup.
| SkyMarshal wrote:
| Out of curiosity, what do you use to automate deleting of old
| snapshots? I'm aware of zfs-prune-snapshots, but what's the
| best way to automate it (or something like it)? Cron job, or
| something else?
|
| (For anyone wondering, https://github.com/bahamas10/zfs-prune-
| snapshots)
| rodgerd wrote:
| sanoid/syncoid.
| Alexqw85 wrote:
| I use zfsnap.[1] It's been forever since I've worked on it
| (hence why it's now "archived" on GitHub), but it's stable
| for my use and I use it on all ZFS systems I manage.
|
| [1] https://github.com/zfsnap/zfsnap
| ncrmro wrote:
| Checkout sandoid
| anotherhue wrote:
| If you're using a recent Ubuntu, docker/zfs can lead to a large
| amount of snapshots. 'docker system prune' may help but has had
| zfs issues for me.
| yjftsjthsd-h wrote:
| > but has had zfs issues for me.
|
| Can you give more details so I know what to watch out for?
| (Recently started running Docker+ZFS in prod, haven't seen it
| break yet but appreciate heads up)
| flemhans wrote:
| The zfs storage driver isn't production ready yet.
| cosmin800 wrote:
| I am pretty sure Canonical will soon release a
| stable,production ready snap package for zfs.
| alyandon wrote:
| That gets updated arbitrarily at random intervals
| (because who doesn't want filesystem modules updated
| frequently!?!?!) and then purges logs every 72 hours so
| when you have a problem you'll never even know what was
| updated!
|
| I have to run snapd on one of my servers and I hate
| everything about it.
| r1ch wrote:
| I love that they say this, then default to using it when
| ZFS is detected.
| e12e wrote:
| (not op, but) I don't recall exactly what I experienced (in
| development- I think it was just terribly slow?) - I ended up
| carving out a volume (zvol) and formatting it with ext4 (or
| xfs?) for docker.
|
| It might actually have been this ms sql issue that led me to
| do that, actually:
|
| https://github.com/moby/moby/issues/33191
| anotherhue wrote:
| there were a number of containers immune to 'docker rm',
| seemed it was trying to invoke 'zfs delete' on something that
| had already been removed (presumably moments ago as part of a
| previous step). Nasty workaround was to manually recreate the
| dataset (and children) so that it could remove them. The
| issue was common enough that I could google it so don't
| stress.
|
| More worryingly, 'prune' locked up for about 10 minutes
| before getting to do it's thing. pinning the cpu the whole
| time. This was on my devbox, so maybe it's my doing but did
| not inspire confidence.
|
| I've run FreeBSD ZFS for about a decade and openzfs/ubuntu in
| prod for kafka clusters so I'm no stranger - this docker
| usage was just particularly strange.
| hddherman wrote:
| I've had bad experiences with Docker+Debian as well when the
| storage was on ZFS. I've had a much better experience by just
| keeping /var/log/containers on something simple, like ext4.
| Valuable data is still on ZFS and mounted inside the
| containers, no issues there.
| eptcyka wrote:
| Is there any good tooling on managing automatic snapshotting and
| sending backups?
| waz0wski wrote:
| sanoid is a good one. I've been on a quest to run mostly
| golang-based utilities and switched to
| https://github.com/zrepl/zrepl which has been running/syncing
| without issue for a couple years now
| indiv0 wrote:
| Personally I just setup sanoid and syncoid on my NAS. They were
| dead simple to get up and running with.
|
| Sanoid makes snapshots according to a schedule you define (e.g.
| 1 monthly, 31 daily, 24 hourly). It also prunes old snapshots
| to ensure that you always have the exact snapshots you need
| available and no more.
|
| Syncoid uses zfs send/recv to copy over snapshots from one ZFS
| dataset to another, typically between machines. You basically
| have two options: either you can:
|
| 1) copy over ALL of the snapshots from the source in the order
| they were created (like cherry picking one git commit after
| another onto another branch), or you can:
|
| 2) configure it to make a new snapshot every time it syncs and
| just copy over the diff since the last time you synced (a bit
| like squashing all the commits into one and just cherry-picking
| that).
|
| The former is good if you want to maintain the same snapshots
| on the source and the destination (you can also use sanoid on
| the destination to prune the snapshots further).
|
| The latter is good if you want to minimize the amount of churn
| sent over the network. For example if you're snapshotting VM
| images you'd be interested in sending over the state of the VM
| as it appears at the end of the day, rather than all the
| intermediate state.
| unixhero wrote:
| Proxmox backup node
|
| https://www.proxmox.com/en/proxmox-backup-server
| ptomato wrote:
| sanoid (https://github.com/jimsalterjrs/sanoid/) is probably
| the most featureful, but there's quite a few I'm aware of -
| zfsnap, zfs-auto-snapshot, zfs_autobackup, znapzend...
___________________________________________________________________
(page generated 2021-07-29 23:01 UTC)