[HN Gopher] AI language models can exceed PNG and FLAC in lossle...
___________________________________________________________________
AI language models can exceed PNG and FLAC in lossless compression,
says study
Author : belter
Score : 56 points
Date : 2023-09-28 16:00 UTC (7 hours ago)
(HTM) web link (arstechnica.com)
(TXT) w3m dump (arstechnica.com)
| hint23 wrote:
| ts_zip ( https://bellard.org/ts_server/ts_zip.html ) already
| implements lossless compression with language models and is fast
| enough to be usable for some applications.
| nuancebydefault wrote:
| Assuming any image or audio stream is available on the internet
| (not true but very close), you can have a super large compression
| by replacing the media by their url.
|
| LLM's (the set of connections and their weights) are in fact a
| compressed version of a large part of the internet.
|
| So what is found by the article should not surprise.
| Legend2440 wrote:
| That's not how it works. LLMs were only trained on text, so
| this is new data it's never seen before. There's no train-test
| leakage.
| nuancebydefault wrote:
| You are right but I am sure there are patterns in the weights
| that (accidentally or not) predict patterns commonly seen in
| media.
| darig wrote:
| [dead]
| cpeterso wrote:
| > Assuming any image or audio stream is available on the
| internet (not true but very close), you can have a super large
| compression by replacing the media by their url.
|
| That reminds me of the "Dropship" utility: to save server
| storage and reduce file upload time, Dropbox used to
| deduplicate uploaded files globally in a way that files would
| have the same hash regardless of who uploaded them. Anyone who
| knew the hash of an uploaded file could download it to their
| Dropbox folder, so people could share large files by just
| sharing the Dropbox hash.
|
| http://paranoia.dubfire.net/2011/04/how-dropbox-sacrifices-u...
| sfink wrote:
| The article mixes up what is significant. The headline refers to
| an utterly irrelevant result. As others have pointed out, it's
| just saying that a compression method with a large dictionary can
| beat a (better) compression method with a small dictionary.
|
| The interesting bit is that it can transfer a dictionary-ish
| thing from the text to the audio and image domains and it'll work
| well enough to beat format-specific compression algorithms. Which
| _is_ surprising.
|
| I'm sure it's only benefiting from a small subset of the 280GB
| raw text-domain dictionary. But it is surprising that it is not
| only benefiting, it is benefiting enough to beat the special-
| purpose compressors.
|
| I'll be sad if this turns out to only be true because of some
| meaningless category of edge cases.
|
| On a side note, it does make one wonder if we should have a well-
| known massive image-based dictionary that compression algorithms
| can assume the existence of (and request ranges of from a
| server). Or heck, given these results in the article, maybe it
| would be a dictionary with a mishmash of different domains. Then
| when encoding for limited devices with network connections, you'd
| specify the maximum amount of data it could retrieve from the
| dictionary (perhaps as a percentage of the compressed size?).
| pornel wrote:
| It's cool that it works, but this isn't entirely surprising
| result.
|
| Compression can be understood as a combination of probability
| modelling and entropy coding.
|
| Entropy coding takes a predicted probability of the next symbol
| (symbol can be a word, a color, audio sample, etc.) and converts
| it into bits. This is a solved problem. We know how to code this
| with perfect efficiency (however PNG didn't get optimal
| arithmetic coding due to patents, so its Huffman coding is
| optimal only for probabilities proportional to powers of two).
|
| So the rest of the lossless compression is the art of predicting
| what will come next based on data decompressed so far. Standard
| compression algorithms have simple hardcoded rules for this,
| mostly predicting exact repetitions of data seen recently. OTOH
| ML models have seen many more patterns, can express probabilities
| with more finesse, and setups like LLMs are very good at adapting
| based on previous data.
| foota wrote:
| But in this case that data isn't language, it's image data.
| Implying that's it's not a LLM, but something more, dare I say,
| general?
| pornel wrote:
| No, that would be personifying it too much. It's trained on
| tokens, and on the scale of the web, it has seen all kinds of
| patterns, including random garbage. It's seen
| "aaaaaaaaaaaaaaaaaaaaaa" and "abcdefghijklmnop" too, and
| that's already as good as PNG filters can do. If you feed it
| pixel-derived nonsense, it will mimic more of such nonsense.
| bastawhiz wrote:
| Does this paper actually say how they provide the image/audio
| data to the language model? You can't really just pipe 16khz
| audio into CLIP and slap that into your model of choice. I
| skimmed the paper, then skimmed it again, and I don't have an
| answer for what exactly they did here.
| qayxc wrote:
| The paper actually lays it out pretty clearly. Yes, you can
| just pipe 16kHz audio into a tokenizer, which is exactly what
| they did (kind of).
|
| They just treat the input as 8-bit ASCII (which in itself is
| technically not a thing, since ASCII is 7-bit, but the paper
| doesn't seem to care about technicalities).
|
| So the input alphabet is 256 entries - 1 for each byte. Simple
| as that.
| nneonneo wrote:
| Let's not forget that a decompressor for such a format would
| necessitate running the entire model, i.e. in this case, a 70B
| parameter decompressor. It's perhaps not surprising that you can
| compress files better when given a (very) large dictionary to
| refer to. This is why any reasonable compression benchmark
| includes the decompressor's size in the size score.
| Legend2440 wrote:
| It's still surprising because the language model was only
| trained on text, and is _generalizing_ to compressing other
| forms of data using in-context learning. The model contains no
| information about images or audio.
|
| But yes, no one is suggesting it as a practical compression
| tool due to the large size and compute requirements.
| ozim wrote:
| It does not matter bytes are bytes in the end neural network
| is generic compression model it is predicting next bytes that
| should show up.
|
| For parent - imagine winrar but in cloud so if you don't pay
| you cannot decompress your data. There would be no way you
| could "pirate" it so everyone has to pay not like regular
| winrar everyone jokes about buying licenses.
| ramesh31 wrote:
| Could a distributed decompressor be valid? I. e. something like
| bittorrent that provides a DHT for known files.
| bastawhiz wrote:
| If you have to fetch one or more dictionaries to decompress
| the file anyway, why not just include the dictionary with the
| file you need to decompress?
| jmoss20 wrote:
| Because the model (=="dictionary") is 70B floats -- 280GB
| naively, 40-70GB aggressively quantized (which might reduce
| compression rate). If your file is big enough that the
| marginal compression win over other methods makes this
| space-effective, sure. But that's a very narrow case.
| dmajor2 wrote:
| https://qwantz.com/index.php?comic=2092
| nick-of-time wrote:
| This has been implemented: https://github.com/philipl/pifs
| burnte wrote:
| Give me a lookup table of 50 petabytes, let me seed that LUT with
| billions of files, and I, too, can write an amazingly efficient
| compression algorithm.
| janalsncm wrote:
| I don't see why video chat apps shouldn't do something like
| this. Maybe the encoding/decoding is too slow.
| sfink wrote:
| Accessing a giant lookup table is about the most cache-
| unfriendly thing you could do. This is not a good approach
| when latency matters.
|
| Smaller LUTs are fun to think of, especially if you eliminate
| the lossless angle. I'm really not that interesting to watch
| during a videoconference. When I'm not talking, it would
| probably be _better_ for viewers to make up a stream of me
| sitting still with a dumb look on my face. It should be
| loosely matched to my actual position /pose, so that when I
| start talking or my dog attacks me or the bomb explodes and
| wipes out my room, the transition back to actual data is
| relatively seamless.
|
| (And if you set the threshold high enough, maybe I don't need
| to be in the room at all...)
| m3kw9 wrote:
| But it isn't generalized like this is supposed to be
| gpt5 wrote:
| Sure. But you can only use text files in your seed.
| qayxc wrote:
| What's the difference between 0x2e and .? You can easily
| transform between representations of data, be that text or
| binary in any encoding you'd like. The model input token
| vector (or LUT index) is still the same. I don't see a
| difference there.
| gpt5 wrote:
| You are missing the point. LLM are not trained on image
| data, but on human text data. This is not the same as
| encoding the image as ASCII characters.
| nonameiguess wrote:
| I don't know if chinchilla does the same, but PNG isn't really
| meant to be optimal lossless compression. Partly to avoid patent
| issues with LZW, but also because part of the point of the format
| is progressive display, i.e. the ability to show a lower-
| resolution image constructed from a subset of pixels before the
| entire file has transferred. You can turn off interlacing if you
| wish, but having it on makes for less effective compression. As
| far as I understand, most implementations also opt for faster
| speed and lower energy use at the cost of worse compression.
| brucethemoose2 wrote:
| Even slow PNG compression (dozens of seconds per file) is quite
| inefficient.
| [deleted]
| ComputerGuru wrote:
| There are exceptions where interlacing can decrease the png
| size, but they are few and far in between:
| https://stackoverflow.com/questions/37867463/how-can-an-inte...
| beefman wrote:
| https://arxiv.org/abs/2309.10668
___________________________________________________________________
(page generated 2023-09-28 23:02 UTC)