[HN Gopher] OpenZFS deduplication is good now and you shouldn't ...
___________________________________________________________________
OpenZFS deduplication is good now and you shouldn't use it
Author : type0
Score : 50 points
Date : 2024-10-30 21:48 UTC (1 hours ago)
(HTM) web link (despairlabs.com)
(TXT) w3m dump (despairlabs.com)
| nikisweeting wrote:
| I'm so excited about fast dedup. I've been wanting to use ZFS
| deduping for ArchiveBox data for years, as I think fast dedup may
| finally make it viable to archive many millions of URLs in one
| collection and let the filesystem take care of compression across
| everything. So much of archive data is the same jquery.min.js,
| bootstrap.min.css, logo images, etc. repeated over and over in
| thousands of snapshots. Other tools compress within a crawl to
| create wacz or warc.gz files, but I don't think anyone has tried
| to do compression across the entire database of all snapshots
| ever taken by a tool.
|
| Big thank you to all the people that worked on it!
|
| BTW has anyone tried a probabilistic dedup approach using
| soemthing like a bloom filter so you don't have to store the
| entire dedup table of hashes verbatim? Collect groups of ~100
| block hashes into a bucket each, and store a hyper compressed
| representation in a bloom filter. On write, lookup the hash of
| the block to write in the bloom filter, and if a potential dedup
| hit is detected, walk the 100 blocks in the matching bucket
| manually to look for any identical hashes. In theory you could do
| this with layers of bloom filters with different resolutions and
| dynamically swap out the heavier ones to disk when memory
| pressure is too high to keep the high res one in RAM.
| uniqueuid wrote:
| I get the use case, but in most cases (and particularly this
| one) I'm sure it would be much better to implement that client-
| side.
|
| You may have seen in the WARC standard that they _already_ do
| de-duplication based on hashes and use pointers after the first
| store. So this is exactly a case where FS-level dedup is not
| all that good.
| dark-star wrote:
| I wonder why they are having so much trouble getting this working
| properly with smaller RAM footprints. We have been using
| commercial storage appliances that have been able to do this for
| about a decade (at least) now, even on systems with "little" RAM
| (compared to the amount of disk storage attached).
|
| Just store fingerprints in a database and run through that at
| night and fixup the block pointers...
| wmf wrote:
| Fixup block pointers is the one thing ZFS didn't want to do.
| magicalhippo wrote:
| > and fixup the block pointers
|
| That's why. Due to reasons[1], ZFS does not have the capability
| to rewrite block pointers. It's been a long requested
| feature[2] as it would also allow for defragmentation.
|
| I've been thinking this could be solved using block pointer
| indirection, like virtual memory, at the cost of a bit of
| speed.
|
| But I'm by no means a ZFS developer, so there's surely
| something I'm missing.
|
| [1]: http://eworldproblems.mbaynton.com/posts/2014/zfs-block-
| poin...
|
| [2]: https://github.com/openzfs/zfs/issues/3582
| Wowfunhappy wrote:
| I just wish we had "offline" dedupe, or even "lazy" dedupe that
| doesn't require the pool to be fully offline, but doesn't happen
| immediately.
|
| Like:
|
| > When dedup is enabled [...] every single write and free
| operation requires a lookup and a then a write to the dedup
| table, regardless of whether or not the write or free proper was
| actually done by the pool.
|
| To me, this is "obviously" the wrong approach in most cases. When
| I'm writing data, I want that write to complete as fast as
| possible, even at the cost of disk space. That's why I don't save
| files I'm actively working on in 7zip archives.
|
| But later on, when the system is quiet, I would love for ZFS to
| go back and figure out which data is duplicated and use the BRT
| or whatever to reclaim space. This could be part of a normal
| scrub operation.
| magicalhippo wrote:
| The author of the new file-based block cloning code had this in
| mind. A backround process would scan files and identify dupes,
| delete the dupes and replace them with cloned versions.
|
| There are of course edge cases to consider to avoid data loss,
| but I imagine it might come soon, either officially or as a
| third-party tool.
| Dylan16807 wrote:
| The ability to alter existing snapshots, even in ways that
| fully preserve the data, is extremely limited in ZFS. So yes
| that would be great, but if I was holding my breath for Block
| Pointer Rewrite I'd be long dead.
| tilt_error wrote:
| If writing performance is critical, why bother with deduplication
| at writing time? Do deduplication afterwards, concurrently and
| with lower priority?
| klysm wrote:
| Kinda like log structured merge tree?
| klysm wrote:
| I really wish we just had a completely different API as a
| filesystem. The API surface of filesystem on every OS is a
| complete disaster that we are locked into via backwards
| compatibility.
| magicalhippo wrote:
| Internally ZFS is essentially an object store. There was some
| work which tried to expose it through an object store API.
| Sadly it seems to not have gone anywhere.
|
| Tried to find the talk but failed, was sure I had seen it on a
| Delveloper Summit but alas.
| tjwds wrote:
| (dupe) https://news.ycombinator.com/item?id=41961863
| Wowfunhappy wrote:
| The resubmission should be okay because the last one did not
| get significant attention.
| gtirloni wrote:
| HN will automatically redirect the submitter to a recent
| submission instead of allowing a new post... if it had a
| significant number of comments.
|
| https://news.ycombinator.com/newsfaq.html
___________________________________________________________________
(page generated 2024-10-30 23:00 UTC)