[HN Gopher] Chrome Feature: ZSTD Content-Encoding
___________________________________________________________________
Chrome Feature: ZSTD Content-Encoding
Author : croes
Score : 179 points
Date : 2024-04-01 09:16 UTC (13 hours ago)
(HTM) web link (chromestatus.com)
(TXT) w3m dump (chromestatus.com)
| tutfbhuf wrote:
| Finally. I hope we get zstd into the Go standard library. It
| feels kind of strange that we have to fetch a third-party library
| for that.
| roywashere wrote:
| I just contributed zstd decoding to httpx, my favourite python
| http library!
| fl0ki wrote:
| Only if it's backed by security-reviewed and fuzzed asm. Go's
| gzip implementation is slow enough that klauspost is worth
| importing, following that pattern, I would probably still use
| klauspost's zstd even if Go had a slow reference implementation
| in the standard library.
|
| What I'm really hoping for is something as seamless as gzhttp
| but for zstd.
| https://github.com/klauspost/compress/tree/master/gzhttp
| xnx wrote:
| > Zstd is roughly three times faster than Brotli for
| decompression. Combined with zstd being faster at compression,
| this will result in faster page load times.
| linsomniac wrote:
| Part of my infrastructure uses compressed "dd" disc images. Last
| week I switched from using "xz" to "zstd" for them and "zstd -19"
| compresses almost as well as xz (1.8GB vs 1.7GB), but the
| threaded decompression is >10x faster (72s -> 6s).
|
| I made this change coincidentally, the day before the xz
| compromise.
| cogman10 wrote:
| I've been nothing but happy with ZSTD. Really good compression
| trade offs with ratios close to the best of LZMA.
|
| ZSTD 3 is nearly as fast as LZ4 while getting a much better
| ratio. Consiquentially, on my linux boxes I use either ZFS or
| btrfs and use zstd 3 as the default for the file system.
| dralley wrote:
| Nearly as fast to compress, but much slower to decompress.
| Zstd is pretty good on decompression but LZ4 is top-tier.
|
| Zstd is often "fast enough" in decompression for most cases,
| though.
| chronogram wrote:
| It's why I use LZ4 for ZRAM. Only at RAM speeds do LZ4's
| speeds matter.
| wood_spirit wrote:
| (JiaT75 contributed to zstd as well as xz)
| cipherboy wrote:
| Citation needed?
| https://github.com/facebook/zstd/commits/dev/?author=jiaT75
| wood_spirit wrote:
| https://news.ycombinator.com/item?id=39873017
| cipherboy wrote:
| That looks like a(n unofficial) fork, not contributions
| to the official upstream repository.
| geggo98 wrote:
| I think there were some open PRs from that account, that got
| scrubbed from the account after the back door in xz was
| discovered. But probably nothing got merged.
| ikekkdcjkfke wrote:
| Is it possible to retrieve the old branches?
| linsomniac wrote:
| Additional note: I run these on servers that are not very tall
| but are pretty wide: 40 1.8GHz cores or so. So multi-threaded
| decompression really helps. I was using xz before, which
| doesn't support threaded decompression (according tom man
| page), decided to switch to zstd rather than pixz.
| dmw_ng wrote:
| A very happy moment for Yann no doubt, he mentioned Google's
| competing compression efforts directly impeding his work during a
| corecursive podcast interview ( https://corecursive.com/data-
| compression-yann-collet/ )
|
| For the rest of us, another half-decade+ wait to get useful
| levels of compatibility especially on mobile
| jeroenhd wrote:
| > For the rest of us, another half-decade+ wait to get useful
| levels of compatibility especially on mobile
|
| Chrome already ships it with their currently released browsers,
| so you get a significant amount of mobile traffic (about 42%)
| from Android devices supporting it already. I don't know what
| the current status of Blink on iOS is, but once that releases,
| you'll also get iOS users.
|
| WebKit has reacted positively to the proposal, though the
| Github issue documenting their position refers to an internal
| radar ticket so I have no idea what their timeline is.
|
| If you build for Chrome today, compatibility will only grow.
| Chrome doesn't do compression levels above a certain point but
| I doubt Safari and Firefox will be affected by a different
| limitation when support eventually lands.
|
| Plus, if your web server is configured well, you can have the
| gzip/brotli/zstd content available right next to each other,
| and just follow the browser's preference to pick a supported
| compressed asset.
|
| There really are no downsides here.
| olliej wrote:
| If you "build for chrome today" you create a chrome only
| monopoly exactly as web devs did with IE.
|
| Or you could build your site properly and have it work on
| more than a single browser.
|
| You do you.
| lifthrasiir wrote:
| Simple: Brotli was initially shipped as a part of WOFF2
| compressed font file format, and its creators foresaw more
| possible uses in the web browser, tailoring its built-in preset
| dictionary for web contents. It is no doubt that Zstandard is
| faster than Brotli, but both are sufficiently fast enough for
| web contents (100 MB/s and more), so an additional gain from
| Zstandard was much smaller. Given this, I believe Zstandard
| support in web browsers was possible only due to the strong
| push from Meta.
| MatthiasPortzel wrote:
| Because of HTTP content negotiation, you can serve zstd to
| Chrome users today, so "useful levels of compatibility" doesn't
| make sense in this context.
| beeboobaa3 wrote:
| Can we expect xz support next?
| rafaelturk wrote:
| hahaha
| theandrewbailey wrote:
| No. LZMA-based formats like xz are too slow for web transfer
| speeds.
| CodesInChaos wrote:
| Isn't LZMA decompression pretty fast? For static assets its
| slow compression doesn't matter.
| toastal wrote:
| The cost of compression isn't nothing. A lot of Linux
| distros have moved off of it for packages since they heavy
| Zstd compression was close enough on size while saving a
| lot of time & CPU bandwidth.
| pixl97 wrote:
| They are having some problem compiling the test suite in
| Chrome.
| alserio wrote:
| It would be nice if CompressionStreams would also get brotli/zstd
| support. I wonder if the js engine uses the same implementation
| of the browser
| donatj wrote:
| While looking to see how it compares to brotli, I found an
| interesting conversation between a zstd contributor and the
| author of brotli
|
| https://news.ycombinator.com/item?id=19678985
| jeffbee wrote:
| Zstd implementation got a lot better in the last five years.
| Ono-Sendai wrote:
| I implemented this in my server (https://substrata.info/). Works
| great. Results in smaller files than zlib/deflate. One thing to
| watch out for - compression levels >= 20 aren't handled by chrome
| clients.
| executesorder66 wrote:
| I am surprised all major browsers hadn't implemented this years
| ago. Zstd is way better than gzip and brotli.
| Y_Y wrote:
| Hardly surprising when you consider gzip's DEFLATE was around
| in 1990 and developed by one guy as shareware and zstd was
| produced by a megacorp in 2015.
| lifthrasiir wrote:
| Zstandard was also mainly designed and produced by a single
| person, Yann Collet.
| dralley wrote:
| Yann also developed LZ4
| KingMob wrote:
| Yes, but they also work for megacorp Facebook, and
| according to
| https://github.com/facebook/zstd/graphs/contributors, 300+
| other contributors have made 4500+ commits to the zstd
| repo.
|
| It's not quite as small-scale as 90's-style shareware was.
| kzrdude wrote:
| zstd has improved a lot since it was brought in under the
| facebook roof, but those are incremental changes, it
| already existed before then.
|
| Here's the good old blog about that
| http://fastcompression.blogspot.com/
| veltas wrote:
| Yes it's worth taking a minute to appreciate how well DEFLATE
| and gzip has stood up over the years. It's a brilliant format
| and tool. I was definitely a little too ready to believe the
| paper that claimed gzip's ability to classify text was better
| than deep neutral nets, alas it has some limits after all!
| silverquiet wrote:
| How much better is Zstd? Compression seems like one of those
| things where diminishing returns kick in very quickly. Whenever
| I need compression, it's always just been basically, "throw
| gzip at it because it does a lot without much of a performance
| hit". Basically it does the job.
| lifthrasiir wrote:
| If you are not too much constrained in I/O rate, Zstandard
| has no match; it can easily clock more than 1 GB/s with a
| good compression ratio and also can automatically adapt to
| the changing I/O rate. Web browsers typically work with much
| less bandwidth available, though, so both Brotli and
| Zstandard are virtually identical in the clients.
| jorvi wrote:
| It's not so much about IO rate but access times.
|
| If compression and decompression speed are critical, lz4.
| Zstd for pretty much everything else.
|
| There are edge cases where compression time doesn't matter
| but decompression time does. This used to be the case for
| game torrents back in the old days, and UHARC was used for
| that to great effect. Not sure what the current king is for
| that purpose.
| lifthrasiir wrote:
| Brotli has a higher potential for the compression ratio
| due to its internal structure, so that edge case would be
| better served by Brotli than by Zstandard.
| repiret wrote:
| One of the systems I maintain at work uses enormous,
| highly compressible, text files, which need to be
| compressed once and decompressed many times.
| Decompression speed isn't critical, it just needs to keep
| up with processing the decompressed data to avoid being
| the bottleneck. We optimize primarily for compression
| ratio.
|
| For that system, we haven't found something that beats
| `xz -9`.
| sodality2 wrote:
| Something like FreeArc would be used nowadays for
| repacks, typically - there's gains to be made by
| tailoring your procedure depending on the type of game
| asset, and some of those decisions won't apply to general
| compression.
|
| FreeArc Next does actually use zstd as above but it also
| does a lot of tricks with compression, dictionaries, etc
| while taking much longer to process.
|
| As an example, looking at FitGirl's COD:BO3 repack,
| 180GB->42.4GB entirely losslessly. Not sure how regular
| compression would fare on the original game, though.
| derekp7 wrote:
| Try this: time tar -cf /usr/share/doc |gzip |wc -c vs. time
| tar -cf /usr/share/doc |zstd |wc -c
|
| repeat a few times to warm up your disk cache if needed. On
| mine host (with an nvme disk), zstd was about slightly better
| compression ratio than gzip, but took 1 second instead of 9
| seconds to compress. Compare against something like lzop,
| which is about the same speed, but produces much worse
| compression.
|
| Of course, with gzip if you have multiple cores you have the
| option of using pigz which bring the wall-clock time of gzip
| down to comparable to zstd and lzop.
| silverquiet wrote:
| Thank you - that's a great, simple test.
| loeg wrote:
| > with gzip if you have multiple cores you have the option
| of using pigz which bring the wall-clock time of gzip down
| to comparable to zstd and lzop.
|
| (But then you should use zstd -T0 for an apples to apples
| comparison.)
| idoubtit wrote:
| For a benchmark on a standard set: https://github.com/inikep/
| lzbench/blob/master/lzbench18_sort... Of course, you may get
| different results with another dataset.
|
| gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]
|
| zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]
|
| NB1: The default for zstd is -3, but the table only had -2.
| The difference is probably small. The range is 1-22 for zstd
| and 1-9 for gzip.
|
| NB2: The default program for gzip (at least with Debian) is
| the executable from zlib. With my workflows, libdeflate-gzip
| iscompatible and noticably faster.
|
| NB3: This benchmark is 2 years old. The latest releases of
| zstd are much better, see
| https://github.com/facebook/zstd/releases
|
| For a high compression, according to this benchmark xz can do
| slightly better, if you're willing to pay a 10x penalty on
| decompression.
|
| xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
|
| zstd -18 [ratio=25%] [compr=3.6Mo/s] [dec=912Mo/s]
| btdmaster wrote:
| It would be good if zstd had a standardised a static web
| dictionary, like Brotli's https://www.rfc-
| editor.org/rfc/rfc7932#appendix-A. This would mean that the
| overhead for small files would be smaller, for boilerplate like
| <!DOCTYPE html>.
| swinglock wrote:
| Possibly just use the one from Brotli since it's already
| standardized. If it's any good then the work is mostly already
| done, right?
| mjevans wrote:
| They want it as a standard option (pre-shipped with all
| decompressors). Not something they could incorporate into a
| customized fork / client that uses zstd with their own
| extensions.
| a_e_k wrote:
| The zstd API does allow you to supply your own initial
| dictionary, so there's no need to fork it to allow a
| browser implementation to use the brotli dictionary.
|
| Personally, as someone who doesn't work in web, I'm just as
| happy that zstd is flexible this way. For my applications,
| the brotli dictionary is pure overhead that bloats the
| library.
| TillE wrote:
| Yeah zstd is nice on its own, but a proper dictionary will give
| you several times the compression ratio on top of that, plus
| super fast decompression. It's just absurdly good for
| specialized data.
| chronogram wrote:
| Do Linux distributions use a dictionary with their package
| manager? Since their packages are typically zstd compressed,
| every distro (version) release could have its own
| accompanying dictionary.
| vlovich123 wrote:
| Dictionaries only help with really tiny files.
|
| > Typical gains range ~10% (at 64KB) to x5 better (at
| <1KB).
|
| https://www.manpagez.com/man/1/zstd/zstd-1.1.0.php
|
| Files distributed by distros are unlikely to have many
| packages < 64kib so the advantages of a dictionary rapidly
| diminish on this use-case.
| bryanlarsen wrote:
| Or to put it more explicitly `Content-Type: zstd` should have a
| standard dictionary, since that's far easier to add to new
| proposals than something widely used.
|
| The brotli dictionary appears to help with random text, not
| just html/css.
| hinkley wrote:
| Content-Encoding?
| bryanlarsen wrote:
| Yes, thank you.
| saurik wrote:
| There is an issue tracking this with a bunch of links to
| discussions about it, but they continue to not have time it
| seems.
|
| https://github.com/facebook/zstd/issues/3100
|
| This was the first place my mind went when I saw this Content-
| Encoding announcement, so I ran and re-checked the issue :(.
| infogulch wrote:
| Brotli's baked-in dictionary always irked me because it's
| forever fixed and can't be updated (not that I'm implementing
| new hypermedias on a weekly basis but still). I'd much rather
| see broad adoption of plain 'Content-Type: zstd` implemented
| with no dictionary, and later go through a standards process to
| add named content-specific dictionaries like `Content-Type:
| zstd-web` or `zstd-cn` or whatever.
|
| Edit: Actually this is already considered in RFC-8878 [0]. The
| RFC reserves zstd frame dictionary ids in the ranges: <= 32767
| and >= (1 << 31) for a public IANA dictionary registry, but
| there are no such dictionaries published for public use yet.
|
| [0]: https://datatracker.ietf.org/doc/html/rfc8878#iana_dict
| SquareWheel wrote:
| Have you seen this proposal yet? It allows domains to define
| their own dictionaries for future compressions, with delta
| updates for changes.
|
| Still seems a bit complicated to me, but could be meaningful
| for web apps that are required to be large.
|
| https://github.com/WICG/compression-dictionary-transport
| tedunangst wrote:
| It's not a standard dictionary, but you can use a custom shared
| one. https://developer.chrome.com/blog/shared-dictionary-
| compress...
| yread wrote:
| The second document has a link to a benchmark [1] by peazip which
| shows that 7zip is by far the best when you have a bit of time to
| compress (241s) but its the fastest at decompression and has the
| best ratio. Interesting. How is the support in languages? Are
| browsers planning to implement it as well?
|
| https://peazip.github.io/fast-compression-benchmark-brotli-z...
| mik1998 wrote:
| 7z is an archive format, it just uses LZMA2 compression I
| believe.
| arp242 wrote:
| Well it's also an LZMA implementation, and an implementation
| can be fast or slow, good or bad.
|
| From what I gather 7zip tends to be quite a bit faster than
| xz, since it's been worked on/optimized quite a bit more (I
| didn't test this myself to verify, but I've seen several
| people comment on that over the last few years).
| modeless wrote:
| Will zstd be added to the compression streams API for use by
| JavaScript as well?
| OttoCoddo wrote:
| Sane choice, albeit delayed, as Zstandard has been the leading
| standard algorithm in the field for quite some time. I tested
| most of them for developing Pack [1] and Zstandard looked like
| the best alternative to the good old DEFLATE.
|
| The dictionary feature [2] will help design some new ways of
| getting small resources.
|
| [1] https://news.ycombinator.com/item?id=39793805
|
| [2] https://facebook.github.io/zstd/zstd_manual.html#Chapter10
| lxgr wrote:
| Who would have thought that Chrome can add new compression
| formats too as opposed to removing them!
|
| This is nice - but please, bring back JPEG XL too.
| redeeman wrote:
| i hope they do, but i doubt it. they should be ashamed
| lemper wrote:
| as usual, I am always in the skeptic camp when chrome people add
| features. there's an unexplainable suspicion that they do it for
| EEE strategy like web integrity api. remember folks, if you love
| open Web, you'd advocate to use non-chrome browser.
| hyperpape wrote:
| They're implementing a widely used compression standard created
| by a developer at a competing company.
|
| Even if Google is the absolute most evil company in the world,
| they will still make many smaller decisions every year that are
| just reasonable engineering. If you forget that, you end up
| going down a conspiracy rabbit hole and losing any ability to
| understand the world.
| kccqzy wrote:
| Or you can look at whether Firefox will add the same feature.
| They do.
| cpeterso wrote:
| There is some recent development activity on Firefox's zstd bug:
| https://bugzilla.mozilla.org/show_bug.cgi?id=1301878
| fastest963 wrote:
| We have been live with zstd encoding for several weeks now and
| last week public traffic really started to use it. We're now at
| 50% brotli, 40% zstd, 9% gzip. Our payload is almost exclusively
| JavaScript and compression rates are 73% zstd (level 6), 72%
| brotli (level 4) and 65% gzip (level 4). However, for 95th
| percentile encoding speed zstd is 4.3ms, brotli is 2.1ms, and
| gzip is 0.1ms. At the 75th percentile they're all essentially the
| same. This is a Golang application and we use a cgo brotli
| library but a pure-go zstd library.
| saurik wrote:
| Are you unable to cache the zstd-compressed JavaScript in your
| app server (so as to save the 4.3ms of CPU on every request)?
| If not, are you sure the 4.3ms of CPU costs less than the 1% of
| bandwidth? I saw a linked discussion here with a Brotli author
| talking to a zstd author that started making me stare into
| these questions more.
| fastest963 wrote:
| We compile/build JS on the fly per-request so we wouldn't be
| able to cache the JS ahead of time.
|
| The difference between brotli and zstd is 2.2ms at the 95th
| percentile, at lower percentiles the differences are much
| smaller. This is still an area of active
| investigation/development for us. There are possibly content
| sizes where one is better than another. There are also
| probably tweaks on the server-side to improve the encoding
| time.
| netol wrote:
| I fail to understand why this, now. For these minimal gains
| (compared to Brotli), servers and CDNs will need to increase
| memory and disk space to store the cached responses in zstd.
| fastest963 wrote:
| We don't use brotli for any responses under 25kb and in those
| cases, for us, zstd is the clear winner. We are still
| tweaking zstd and expect to improve the performance to bring
| it inline with brotli. Additionally, our biggest expense per
| month is egress, any savings goes a long way for us.
| fl0ki wrote:
| I stan zstd hard. Hopefully now that browsers support it, HTTP
| middleware libraries will finally support it too. I already use
| it everywhere else.
| qingcharles wrote:
| Here's the Microsoft .NET and ASP.NET feature tracking:
|
| Add support for Zstandard to System.IO.Compression:
| https://github.com/dotnet/runtime/issues/59591
|
| Support zstd Content-Encoding:
| https://github.com/dotnet/aspnetcore/issues/50643
| hinkley wrote:
| Another important variable is reverse proxies.
|
| You don't necessarily need Rust or Node.js or Java to support
| zstd but you do need Traefic and nginx and haproxy to do so.
| 2OEH8eoCRo0 wrote:
| Stupid question: Why do we keep adding things like this to the
| browser when they could be a library on the system?
| wmanley wrote:
| I feel like I'm missing something here. How would having a
| library on the system help with serving zstd compressed assets?
| snvzz wrote:
| For "realtime" compression, wouldn't lz4 make more sense?
|
| Much lighter in both the server and the client.
| bastawhiz wrote:
| LZ4 has quite poor compression, its advantage is speed. The
| difference between decompressing ten megs of data with LZ4
| versus zstd is a completely negligible number when the
| difference in bytes sent over the wire is might approach a full
| megabyte.
___________________________________________________________________
(page generated 2024-04-01 23:01 UTC)