[HN Gopher] Consider using Zstandard and/or LZ4 instead of Deflate
___________________________________________________________________
Consider using Zstandard and/or LZ4 instead of Deflate
Author : marklit
Score : 98 points
Date : 2025-08-05 17:18 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| zX41ZdbW wrote:
| Very reasonable.
|
| I've recently experimented with the methods of serving bitmaps
| out of the database in my project[1]. One option was to generate
| PNG on the fly, but simply outputting an array of pixel color
| values over HTTP with Content-Encoding: zstd has won over PNG.
|
| Combined with the 2D-delta-encoding as in PNG, it will be even
| better.
|
| [1] https://adsb.exposed/
| privatelypublic wrote:
| Does deflate lead the pack in any metric at all anymore? Only one
| I can think of is extreme low spec compression
| (microcontrollers).
| adgjlsfhk1 wrote:
| Even there, LZ4 is probably better.
| hinkley wrote:
| You think LZ4 is more portable than zlib? I'm gonna need some
| citations on that.
|
| zlib is 30 years old, according to Wikipedia. And that's
| technically wrong since 'zlib' was factored out of gzip
| (nearly 33 years old) for use in libpng, which is also 30
| years old.
| adgjlsfhk1 wrote:
| not more portable, but probably faster in resource
| constrained environments
| duskwuff wrote:
| A basic LZ4 decompressor is on the order of a few dozen
| lines of code. It's exceptionally easy to implement.
| JoshTriplett wrote:
| The only metric deflate leads on is widespread support. By any
| other metric, it has been superseded.
| atiedebee wrote:
| I'd assume memory usage as well, because it has a tiny
| context window compared to zstd
| JoshTriplett wrote:
| You can change the context window of zstd if you want. But
| yes, the default context window size for zstd is 8MB,
| versus 32k.
| arp242 wrote:
| Comparison of "zpng" (PNG wth zstd) and WebP lossless, with
| current PNG. From https://github.com/WangXuan95/Image-
| Compression-Benchmark : Compressed format
| Compressed size (bytes) Compress Time Decompress Time
| WEBP (lossless m5) 1,475,908,700 1,112 49
| WEBP (lossless m1) 1,496,478,650 720 37
| ZPNG (-19) 1,703,197,687 1,529 20
| ZPNG 1,755,786,378 26 24
| PNG (optipng -o5) 1,899,273,578 27,680 26
| PNG (optipng -o2) 1,905,215,734 4,395 27
| PNG (optimize=True) 1,935,713,540 1,120 29
| PNG (optimize=False) 2,003,016,524 335 34
|
| Doesn't really seem worth it? It doesn't compress better, and
| only slightly faster in decompression time.
| bobmcnamara wrote:
| Am I reading those numbers right? That's like 25x faster
| compression than WEBP-M1, there's probably a use case for that.
| arp242 wrote:
| The numbers seem small enough that it will rarely matter, but
| I suppose there might be a use case somewhere?
|
| But lets be real here: this is basically just a new image
| format. With more code to maintain, fresh new exciting zero-
| days, and all of that. You need a strong use case to justify
| that, and "already fast encode is now faster" is probably not
| it.
| scott_w wrote:
| I don't think it's quite as bad, though? It's using a known
| compression library that (from reading other comments) has
| seen use and testing. The rest of PNG would remain
| unchanged, as the decompression format is a plugin.
|
| I know it needs to be battle tested as a single entity but
| it's not the same as writing a new image format from
| scratch.
| realityking wrote:
| Considering both zstandard and PNG are already web facing
| technologies, would the combination of both really increase
| the attack surface?
| stephencanon wrote:
| "Only slightly faster in decompression time."
|
| m5 vs -19 is nearly 2.5x faster to decompress; given that most
| data is decompressed many many more times (often thousands or
| millions of times more, often by devices running on small
| batteries) than it is compressed, that's an enormous win, not
| "only slightly faster".
|
| The way in which it might not be worth it is the larger size,
| which is a real drawback.
| arp242 wrote:
| The difference is barely noticeable in real-world cases, in
| terms of performance or battery. Decoding images is a small
| part of loading an entire webpage from the internet. And
| transferring data isn't free either, so any benefits need to
| be offset against the larger file size and increased network
| usage.
| fmbb wrote:
| Win how?
|
| More efficiency will inevitably only lead to increased usage
| of the CPU and in turn batteries draining faster.
|
| https://en.wikipedia.org/wiki/Jevons_paradox
| hcs wrote:
| So someone is going to load 2.5x as many images because it
| can be decoded 2.5x faster? The paradox isn't a law of
| physics, it's an interesting observation about markets. (If
| this was a joke it was too subtle for me)
| snickerdoodle12 wrote:
| Might as well just shoot yourself if that's how you look at
| improvements. The only way to do something good it to stop
| existing. (this is a general statement, not aimed at you or
| anyone in particular)
| out_of_protocol wrote:
| Is webp really losses here? As far as i remember its capped at
| 4:2:0 and can't do 4:4:4 files without loosing some of the
| color data
| e-topy wrote:
| Instead of using a new PNG standard, I'd still rather use JPEG XL
| just because it has progressive decoding. And you know, whilst
| looking like png, being as small as webp, supporting HDR and
| animations, and having even faster decoding speed.
|
| https://dennisforbes.ca/articles/jpegxl_just_won_the_image_w...
| jchw wrote:
| JPEG XL definitely has advantages over PNG but there is one
| serious seemingly insurmountable obstacle:
|
| https://caniuse.com/jpegxl
|
| Nothing really supports it. Latest Safari at least has support
| for it not feature-flagged or anything, but it doesn't support
| JPEG XL animations.
|
| To be fair, nothing supports a theoretical PNG with Zstandard
| compression either. While that would be an obstacle to using
| PNG with Zstandard for a while, I kinda suspect it wouldn't be
| that long of a wait because many things that support PNG today
| also support Zstandard anyways, so it's not a huge leap for
| them to add Zstandard support to their PNG codecs. Adding JPEG-
| XL support is a relatively bigger ticket that has struggled to
| cross the finish line.
|
| The thing I'm _really_ surprised about is that you still can 't
| use arithmetic coding with JPEG. I think the original reason is
| due to patents, but I don't think there have been active
| patents around that in years now.
| bawolff wrote:
| > The thing I'm really surprised about is that you still
| can't use arithmetic coding with JPEG.
|
| I was under the impression libjpeg added support in 2009 (in
| v7). I'd assume most things support it by now.
| jchw wrote:
| Believe it or not, last I checked, many browsers and some
| other software (file managers, etc.) still couldn't do
| anything with JPEG files that have arithmetic coding.
| Apparently, although I haven't tried this myself, Adobe
| Photoshop also specifically doesn't support it.
| superjan wrote:
| Arithmetic coding decodes 1 bit at a time, usually in
| such a way that you can't do two bits or more with SIMD
| instructions. So it will be slow and energy inefficient.
| adgjlsfhk1 wrote:
| this isn't necessarily true. zstd uses an ans which is a
| type of arithmetic coding which is very efficient to
| decode
| kps wrote:
| > Nothing really supports it.
|
| Everything supports it, except web browsers.
| jchw wrote:
| JPEG-XL is supported by a lot of the most important parts
| of the ecosystem (image editors and the major desktop
| operating systems) but it is a long way away from
| "everything". Browsers are the most major omission, but
| given their relative importance here it is not a small one.
| JPEG-XL is dead in the water until that problem can be
| resolved.
|
| If Firefox is anything to go off of, the most rational
| explanation here seems to just be that adding a >100,000
| line multi-threaded C++ codebase as a dependency for
| something that parses untrusted user inputs in a critical
| context like a web browser is undesirable at this point in
| the game (other codecs remain a liability but at least have
| seen extensive battle-testing and fuzzing over the years.)
| I reckon this is probably the main reason why there has
| been limited adoption so far. Apple seems not to mind too
| much, but I am guessing they've just put so much into
| sandboxing Webkit and image codecs already that they are
| relatively less concerned with whether or not there are
| memory safety issues in the codec... but that's just a
| guess.
| swiftcoder wrote:
| Apple also adopted JPEG-XL across their _entire_ software
| stack. It 's supported throughout the OS, and by pretty
| much every application they develop, so I'm guessing they
| sunk a fair bit of time/money into hardening their codec
| floxy wrote:
| > >100,000 line multi-threaded C++
|
| W. T. F. Yeah, if this is the state of the reference
| implementation, then I'm against JPEG-XL just on moral
| grounds.
| Zardoz84 wrote:
| You can use a polyfill.
| greenavocado wrote:
| That's because people have allowed the accumulation of power
| and control by Big Tech. Features in and capabilities of end
| user operating systems and browsers are gate kept by a
| handful of people in Big Tech. There is no free market there.
| Winners are picked by politics, not merit. Switching costs
| are extreme due to vendor lock in and carefully engineered
| friction.
|
| The justification for WebP in Chrome over JPEG-XL was pure
| hand waving nonsense not technical merit. The reality is they
| would not dare cede any control or influence to the JPEG-XL
| working group.
|
| Hell the EU is CONSIDERING mandatory attestation driven by
| whitelisted signed phone firmwares for certain essential
| activities. Freedom of choice is an illusion.
| google234123 wrote:
| Webp is a lot older than jpg xl, right?
| IshKebab wrote:
| > I kinda suspect it wouldn't be that long of a wait
|
| Yeah... guess again. It took Chrome 13 years to support
| animated PNG - the last major change to PNG.
| edoceo wrote:
| Maybe they were focused on Webp?
| jchw wrote:
| APNG wasn't part of PNG itself until very recently, so I'd
| argue it's kind-of neither here nor there.
| Scaevolus wrote:
| Every new image codec faces this challenge. PNG + Zstandard
| would look similar. The ones that succeeded have managed it
| by piggybacking off a video codec, like
| https://caniuse.com/avif.
| jchw wrote:
| Why would PNG + ZStandard have a harder time than AVIF? In
| practice, AVIF needs more new code than PNG + ZStandard
| would.
| junon wrote:
| I'm just guessing, but bumping a library version to
| include new code cam integrating a separate library might
| be the differentiating factor.
| jchw wrote:
| The zstd library is already included by most major
| browsers since it is a supported content encoding. Though
| I guess that does leave out Safari, but Safari should
| probably support Zstd for that, too. (I would've
| preferred that over Brotli, but oh well.)
| bawolff wrote:
| Doesn't PNG have progressive decoding? I.e. adam7 algorithm
| layer8 wrote:
| It does, using Adam7:
| https://en.wikipedia.org/wiki/Adam7_algorithm
|
| The recently released PNG 3 also supports HDR and animations:
| https://www.w3.org/TR/png-3/
| bawolff wrote:
| > The recently released PNG 3 also supports HDR and
| animations: https://www.w3.org/TR/png-3/
|
| APNG isn't recent so much as the specs were merged
| together. APNG will be 21 years old in a few weeks.
| layer8 wrote:
| True, but https://news.ycombinator.com/item?id=44802079
| presumably holds the opinion that APNG != PNG, so I
| mentioned PNG 3 to counteract that. Animated PNGs being
| officially PNG is recent.
| duskwuff wrote:
| Adam7 is interlacing, not progressive decoding (i.e. it
| cannot be used to selectively decode a part of the image). It
| also interacts extremely poorly with compression; there is no
| good reason to ever use it.
| adzm wrote:
| Web browsers already have code in place for webp (lossless,vp8)
| and avif (av1, which also supports animation), as well as
| classic jpeg and png, and maybe also HEIC (hevc/h265)... what
| benefit do we have by adding yet another file format if all the
| use cases are already covered by the existing formats? That
| said, I do like JPEG-XL, but I also kind of understand the
| hesitation to adopt it too. I imagine if Apple's push for it
| continues, then it is just a matter of time to get supported
| more broadly in Chrome etc.
| bawolff wrote:
| I think there is a benefit to knowing that if you have a png file
| it works everywhere that supports png.
|
| Better to make the back compat breaks be entirely new formats.
| encom wrote:
| (2021)
|
| In my opinion PNG doesn't need fixing. Being ancient is a
| feature. Everything supports it. As much as I appreciate the
| nerdy exercise, PNG is fine as it is. My only gripe is that some
| software writes needlessly bloated files (like adding a useless
| alpha channel, when it's not needed). I wish we didn't need tools
| like OptiPNG etc.
| heinrich5991 wrote:
| Most of the comments on that issue are from this year.
| ori_b wrote:
| Yes. One of the best features of png is that I don't have to
| wonder if it's going to work somewhere. Throwing that away in
| favor of a bit of premature optimization seems like a big loss.
| Especially as this wouldn't be the only modernized image
| compression format out there. Why use this over, eg, lossless
| webp?
|
| I don't think I have ever noticed the decode time of a png.
| willvarfar wrote:
| We ought consider using QOI instead.
|
| QOI is often equivalent or better compression than PNG, _before_
| you even compress it with something like LZ4 etc.
|
| Compressing QOI with something like LZ4 would generally
| outperform PNG.
| adgjlsfhk1 wrote:
| QOI has some pretty major downsides. it only supports 8 bit
| SRGB, and is optimized for images with 8 bit transparency.
| Also, the handsome it uses seems to harm compression when
| entropy compression is used. Also, the focus on streaming means
| that the algorithm can't take advantage of 2d locality.
|
| QOI is really cool, but I think the author cut the final
| version of the spec too early, and intentionally closed it off
| to a future version with more improvements. With another year
| or 2 of development, I think it probably works have become ~10%
| more efficient and suitable for more usecases.
| HocusLocus wrote:
| The reason we have a world full of .gif today is that the .png
| committee rejected animation back when everyone was saying PNG
| would be the "GIF killer". Just sayin'. Don't hold your breath.
| edoceo wrote:
| Remember this: https://burnallgifs.org/
| hughw wrote:
| Related: what's the status of content negotiation? Any browsers
| use it seriously, and has it been successful? If so, then why not
| zpng.
| jasonthorsness wrote:
| One of the interesting features of ZStandard is the support for
| external dictionaries. It supports "training" a dictionary on a
| set of samples, of whatever size (16KiB, 64 KiB, etc.), then
| applying that dictionary as a separate input file for compression
| and decompression. This lets you compress short content much more
| effectively.
|
| I doubt it would apply to PNG because of the length and content
| doesn't seem to be dictionary-friendly, but it would be
| interesting to try from some giant collection of scraped PNGs.
| This approach was important enough for Brotli to include a
| "built-in" dictionary covering HTML.
| DefineOutside wrote:
| This has been applied to minecraft region files in a fork of
| paper, which is a type of minecraft server.
|
| https://github.com/UltraVanilla/paper-zstd/blob/main/patches...
|
| from the author of this patch on discord - the level 9 for
| compression isn't practical and is too slow for a real
| production server but it does show the effectiveness of zstd
| with a shared dictionary. So you start off with
| a 755.2 MiB world (in this test, it is a section of an existing
| DEFLATE-compressed world that has been lived in for a while).
| If you recreate its regions it will compact it down to 695.1
| MiB You set region-file-compression=lz4 and run
| --recreateRegionFiles and it turns into a 998.9 MiB world.
| Makes sense, worse compression ratios but less CPU is what
| mojang documented in the changelog. Neat, but I'm confused as
| to what the benefits are as I/O increasingly becomes the more
| constrained thing nowadays. This is just a brief detour from
| what I'm really trying to test You set region-file-
| compression=none and it turns into a 3583.0 MiB world. The
| largest region file in this sample was 57 MiB Now,
| you take this world, and compress each of the region files
| individually using zstd -9, so that the region files are now
| .mca.zst files. And you get a world that is 390.2 MiB
| immibis wrote:
| Note that each region file contains 1024 chunks that are
| designed to be (but probably aren't) accessed at random, so
| compressing a region file is like a solid archive with a
| solid block size of 1024 files.
| citrin_ru wrote:
| ZSTD is a great compression algorithm but an important PNG (v1.2)
| advantage is that implementations are available in almost all
| actively used operating systems and in most popular languages.
| The same cannot be said about ZSTD with very few implementations
| except https://github.com/facebook/zstd
|
| I'm not even sure there is a good pure Java (no JNI) and Go
| (without Cgo) implementations for ZSTD. And it definitely would
| require more powerful hardware - some micro-controllers which can
| use PNG are too small for ZSTD.
| jonathanoliver wrote:
| For Go, we've been using this library which supports ZSTD.
| https://github.com/klauspost/compress
___________________________________________________________________
(page generated 2025-08-05 23:00 UTC)