[HN Gopher] Fabrice Bellard's TS Zip (2024)
___________________________________________________________________
Fabrice Bellard's TS Zip (2024)
Author : everlier
Score : 65 points
Date : 2026-01-12 20:26 UTC (2 hours ago)
(HTM) web link (www.bellard.org)
(TXT) w3m dump (www.bellard.org)
| publicdebates wrote:
| Bellard finally working with his true colleague.
| dmitrygr wrote:
| "compressed size" does not seem to include the size of the model
| and the code to run it. According to the rules of Large Text
| Compression Benchmark, total size of those must be counted,
| otherwise a 0-byte "compressed" file with a decompressor
| containing the plaintext would win.
| underdeserver wrote:
| Technically correct, but a better benchmark would be a known
| compressor with an unknown set of inputs (that come from a
| real-world population, e.g. coherent English text).
| paufernandez wrote:
| Yeah, but the xz algorithm is also not counted in the bytes...
| Here the "program" is the LLM, much like your brain remembers
| things by coding them compressed and then reconstructs them. It
| is a different type of compression: compression by
| "understanding", which requires the whole corpus of possible
| inputs in some representation. The comparison is not fair to
| classical algorithms yet that's how you can compress a lot more
| (given a particular language): by having a model of it.
| wrs wrote:
| "Compressors are ranked by the compressed size of enwik9
| (10^9 bytes) plus the size of a zip archive containing the
| decompresser." [0]
|
| [0] https://www.mattmahoney.net/dc/text.html
| FartyMcFarter wrote:
| True for competitions, but if your compression algorithm is
| general purpose then this matters less (within reason - no one
| wants to lug around a 1TB compression program).
| MisterTea wrote:
| This is something I have been curious about in terms of how an
| LLM's achieves compression.
|
| I would like to know what deviations are in the output as this
| almost feels like a game of telephone where each re-compression
| results in a loss of data which is then incorrectly
| reconstructed. Sort of like misremembering a story and as you
| tell it over time the details change slightly.
| Scaevolus wrote:
| When LLMs predict the next token, they actually produce a
| distribution of the probability of each of the possible next
| tokens, and the sampler chooses one of them, and not
| necessarily the most likely one!
|
| If instead you run LLM prediction and then encode the
| _probability_ of the next token of the input text you want to
| encode (from the cumulative distribution, a number in [0, 1])
| using arithmetic coding, you can run the same operation in
| reverse to achieve lossless compression.
|
| The tricky part is ensuring that your LLM executes absolutely
| deterministically, because you need to make sure that the
| encoder and decoder have the same probability distribution map
| at each step.
| wewewedxfgdf wrote:
| >> The ts_zip utility can compress (and hopefully decompress)
| text files
|
| Hopefully :-)
| hamandcheese wrote:
| Reading data is overrated. I highly recommend S4:
|
| http://www.supersimplestorageservice.com/
| benatkin wrote:
| I propose the name _tokables_ for the compressed data produced by
| this. A play on tokens and how wild it is.
| fancyswimtime wrote:
| please pass the tokables to the left hand side
| shawnz wrote:
| Another fun application of combining LLMs with arithmetic coding
| is steganography. Here's a project I worked on a while back which
| effectively uses the opposite technique of what's being done
| here, to construct a steganographic transformation:
| https://github.com/shawnz/textcoder
| meisel wrote:
| Looks like it beats everything in the large text compression
| benchmark for enwik8, but loses to several programs for enwik9. I
| wonder why that is.
| AnotherGoodName wrote:
| It's actually not the best at enwik8 or 9.
|
| The results at https://www.mattmahoney.net/dc/text.html
| explicitly add the size of the compressor itself to the result.
| Note the "enwik9+prog" column. That's what it's ranked on.
|
| The reason to do this is that it's trivial to create a
| compressor that 'compresses' a file to 0 bytes. Just have an
| executable with a dictionary of enwik9 that writes that out
| given any input. So we always measure what is effectively the
| Kolmogorov complexity. The data+program as a whole that
| produces the result we want.
|
| So those results add in the compressor size. The programs there
| generally have no dictionary built in or in the case of LLM
| based compressors, no pre-trained data. They effectively build
| the model as they process data. Not compressing much at all at
| the start and slowly compressing better and better as they go.
| This is why these programs do better and better with larger
| data sets. They start with 0 knowledge. After a GB or so they
| have very good knowledge of the corpus of human language.
|
| This program here however is pre-trained and shipped with a
| model. It's 150MB in size! This means it has 150MB of extra
| starting knowledge over those models in that list. The top
| models in that list are the better compressors, they'll quickly
| out learn and overtake this compressor but they just don't have
| that headstart.
|
| Of course measuring fairly this should be listed with that
| 150MB program size added to the results when doing a
| comparison.
| srcreigh wrote:
| As an aside, I wonder how to account for the information
| content embedded in the hardware itself.
|
| A Turing Machine compressor program would likely have more
| bytes than the amd64 binary. So how to evaluate
| KolmogorovComplexity(amd64)?
|
| The laws of physics somehow need to be accounted for too,
| probably.
| d_burfoot wrote:
| Kolmogorov Complexity is only defined up to a constant,
| which represents Turing machine translation length.
| rurban wrote:
| So did beat his own leading program from 2019, nncp, finally.
| egl2020 wrote:
| When Jeff Dean gets stuck, he asks Bellard for help...
| SnowProblem wrote:
| I love this because it gets to the heart of information theory.
| Shannon's foundational insight was that information is surprise.
| A random sequence is incompressible by definition. But what
| counts as surprise depends on context, and for text, we know a
| large amount of it is predictable slop. I suspect there's a lot
| of room to go along this style of compression. For example, maybe
| you could store an upfront summary that makes prediction more
| accurate. Or perhaps you could encode larger sequences or some
| kind of hierarchical encoding. But this is great.
| bambax wrote:
| Yes! information is surprise, and that's why a measure of
| intelligence is the ability to predict.
| oxag3n wrote:
| Compression and intelligence reminded me of the
| https://www.hutter1.net/prize
|
| I've encountered it >10 years ago and it felt novel that
| compression is related to intelligence and even AGI.
| omoikane wrote:
| Current leader of the Large Text Compression Benchmark is NNCP
| (compression using neural networks), also by Fabrice Bellard:
|
| https://bellard.org/nncp/
|
| Also, nncp-2024-06-05.tar.gz is just 1180969 bytes, unlike
| ts_zip-2024-03-02.tar.gz (159228453 bytes, which is bigger than
| uncompressed enwiki8).
| gmuslera wrote:
| Reminded me of pi filesystem (https://github.com/philipl/pifs),
| with enough digits of pi precalculated you might be able to do a
| decent compression program. The trick is in the amount of
| reasonable digits for that, if it's smaller or bigger than that
| trained LLM.
___________________________________________________________________
(page generated 2026-01-12 23:00 UTC)