https://bellard.org/ts_server/ts_zip.html

ts_zip: Text Compression using Large Language Models

The ts_zip utility provided with the ts_server software can compress
(and hopefully decompress) text files with Large Language Models. The
compression ratio is much higher than with other compression tools.
There are some caveats of course:

  * Unlike NNCP, the language model must be available when
    decompressing.
  * A GPU is mandatory to get a reasonable speed. Depending on the
    model size, the speed varies between a few kB/s to one hundred kB
    /s.
  * The same exact GPU model and program versions must be used for
    compression and decompression.
  * The model is frozen so it works only for text files in a language
    that the model has already seen.

Compression Ratio

The compression ratio is given in bits per byte for each model. CMIX
v19 is one of the best lossless data compression program.

                 Original   xz   CMIX  pythia_deduped_70M rwkv_169M rwkv_430M falcon_7B_q4 rwkv_7B_q4
      File         size    (bpb)  v19        (bpb)          (bpb)     (bpb)      (bpb)       (bpb)
                  (bytes)        (bpb)
alice29.txt      152089    2.551 1.645 1.335              1.166     1.028     0.718        0.411
book1            768771    2.717 1.816 1.569              1.426     1.311     1.104        1.115
enwik8           100000000 1.989 1.187 -                  1.098     0.948     -            -
linux-1.2.13.tar 9379840   1.441 -     1.010              0.991     0.837     -            -

Compression Speed and Required Memory

They are measured when compressing the book1 file on a RTX A6000 GPU.
The decompression speed and memory requirements are similar.

      Model        Compression speed GPU memory
                      (kBytes/s)        (GB)
rwkv_169M          128               0.38
rwkv_430M          85                0.94
pythia_deduped_70M 70                6.61
rwkv_7B_q4         15                4.76
falcon_7B_q4       6.7               8.44

Conclusion

The smaller RWKV models seem a good compromise for text compression
because they use a small amount of memory due to their RNN structure
and have a (relatively) high running speed.

---------------------------------------------------------------------
Fabrice Bellard - https://bellard.org/