[HN Gopher] Lzbench compression benchmark
       ___________________________________________________________________
        
       Lzbench compression benchmark
        
       Author : todsacerdoti
       Score  : 20 points
       Date   : 2025-02-11 15:47 UTC (3 days ago)
        
 (HTM) web link (morotti.github.io)
 (TXT) w3m dump (morotti.github.io)
        
       | pella wrote:
       | Probably, the results in the table are about 7 years old. For
       | example, the table shows version 1.3.4 for zstd, which indicates
       | that the results are outdated - currently, version 1.5.6 is the
       | latest release (
       | https://github.com/facebook/zstd/releases/tag/v1.5.6 )
       | 
       | In my opinion, it is better to check the original repository:
       | https://github.com/inikep/lzbench
       | 
       | or this:
       | https://github.com/inikep/lzbench/blob/master/doc/lzbench20_...
        
       | gotoeleven wrote:
       | Based on these results, why would you use anything but lz4_fast ?
       | Are there other dimensions that might be important for a
       | particular use case so a different algo may be best? Ones I can
       | think of might be multi threaded or SIMD friendliness, cache
       | efficiency, memory efficiency, information locality, stream-
       | ability
        
         | pella wrote:
         | latest - sorted by Ratio :
         | https://github.com/inikep/lzbench/blob/master/doc/lzbench20_...
        
         | Galanwe wrote:
         | > Are there other dimensions that might be important for a
         | particular use case so a different algo may be best?
         | 
         | Choosing a compression/decompression algo is an optimization
         | problem essentially. It mainly depends on your IO for input.
         | 
         | For instance, if you have very fast IO, say a local SSD at
         | 7GB/s, then you probably don't want compression, as even the
         | fastest decompression will not go beyond 4GB/s.
         | 
         | If you have moderately fast IO, say 4GB/s, then lz4 is likely a
         | good candidate, you will get maybe a 0.5 compression ratio,
         | meaning you will spend half the time reading your input, while
         | decompressing at nearly the same same speed that you would read
         | uncompressed data.
         | 
         | If you have very slow IO, say 100MB/s, then the compression
         | ratio becomes the main factor, overriding decompression speed.
         | 
         | So, essentially, you want to solve an optimization problem of
         | lowering the time to read the compressed file and decompressing
         | it, conditioned by your reading speed, writing speed, and CPU
         | to spare, under a time constraint of compression speed.
         | 
         | There are solvers on internet doe this problem, but I would
         | love to see someone write it down here in plain math.
        
       | nolist_policy wrote:
       | The data sets are way too small, the biggest one
       | (silesia/mozilla) is just 50mb.
        
       ___________________________________________________________________
       (page generated 2025-02-14 23:01 UTC)