https://github.com/Futrell/ziplm Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} Futrell / ziplm Public * Notifications * Fork 4 * Star 126 126 stars 4 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights Futrell/ziplm This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/F] Use Git or checkout with SVN using the web URL. [gh repo clone Futrel] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @Futrell Futrell Update README.md ... 4e0464d Jul 15, 2023 Update README.md 4e0464d Git stats * 16 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time README.md Update README.md July 14, 2023 21:01 ziplm.py fix wrong temperature conversion factor July 14, 2023 18:01 View code ziplm Usage Why does this "work"? How well does it work? README.md ziplm Useless but mildly interesting language model using compressors built-in to Python. Usage You can "train" it using some training data: data = open(my_favorite_text_file).read().lower() alphabet = "qwertyuiopasdfghjklzxcvbnm,.;1234567890 " model = ziplm.ZipModel(alphabet, training=data) "".join(model.sample_sequence(10)) # sample 10 characters from the alphabet You can also run it without any training data, and just forward sample to see what kinds of patterns gzip likes: alphabet = "abc" model = ziplm.ZipModel(alphabet) "".join(model.sample_sequence(100)) # I get 'ccabcabcabcabcabcabcabcabcabcabcabcabcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbcbccabcbcbcbc' You can also get the probability for a sequence: alphabet = "qwertyuiopasdfghjklzxcvbnm " model = ziplm.ZipModel(alphabet) model.sequence_logprob("this is my favorite string") # I get -83.8 You can also try using bz2 and lzma as language models by passing them as the compressor argument to the model import lzma model = ziplm.ZipModel(alphabet, compressor=lzma) "".join(model.sample_sequence(100)) # I get 'cccbaaaaacccccabcacccbaaaaabaacaabaacaabaacaabaabacaaaaaaaaaaacccbabacaaaaaaaaaaaccccacaaccbaaaaaccc' Why does this "work"? This works because of two facts: 1. A language model is nothing but a distribution on the next token given previous tokens, $p(x \mid c)$. 2. There is a general equivalence between probability distributions and codes. The second point is what makes this interesting. Information theory tells us that we can derive codes from probability distributions. That is, if I have some datapoints $x$, and I know that they follow probability distribution $p(x)$, I can come up with a lossless binary code to encode the $x$ where the length of each code is $-\log_2 p(x) $. This code minimizes the average code length: the only way to get shorter average code length would be to go into the realm of lossy compression. This is called the Shannon Limit. Since I can convert probability distributions to codes in this way, I can also convert codes to probability distributions. If I have a code (like gzip) that describes my datapoint with length $l(x)$ in binary, then that corresponds to a probability distribution $p(x) = 2^{-l(x)} $. If the code is $K$-ary, then the corresponding distribution is $$p (x) = K^{-l(x)}.$$ The ZipLM model works by converting code lengths to probabilities in this way. If I have a vocabulary of size $K$, and a string $c$, then the probability distribution for continuations $x$ is: $$p(x \mid c) \propto K^{-l(cx)},$$ where the proportionality reflects the fact that we have to sum over the compressed lengths of $cx^\prime$ for all $x^\prime$ in the vocabulary. That's all there is to it. How well does it work? It's pretty bad, but it doesn't generate total junk. Here I trained the gzip model in Moby Dick---from the Project Gutenberg text---and the output at least has some recognizable parts: data = open("mobydick.txt").read().lower() alphabet = "qwertyuiopasdfghjkl;'zxcvbnm,. " model = ziplm.Model(alphabet, data) "".join(model.sample_sequence(100)) This gives me "'theudcanvas. ;cm,zumhmcyoetter toauuo long a one aay,;wvbu.mvns. x the dtls and enso.;k.like bla.njv'" which at least seems to have "long a one" in it. About No description, website, or topics provided. Resources Readme Stars 126 stars Watchers 4 watching Forks 4 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.