https://github.com/EGjoni/DRUGS Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} EGjoni / DRUGS Public * Notifications * Fork 5 * Star 172 Stop messing around with finicky sampling parameters and just use DRuGS! License MIT license 172 stars 5 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Discussions * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights EGjoni/DRUGS This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/E] Use Git or checkout with SVN using the web URL. [gh repo clone EGjoni] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @EGjoni EGjoni Merge pull request #1 from Josh-XT/main ... be87166 Dec 30, 2023 Merge pull request #1 from Josh-XT/main Add PyPi Publish, Fix requirements be87166 Git stats * 31 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows Add PyPi Publish December 30, 2023 05:24 drugs changes to Dgenerate to default to make cached generation the default. December 29, 2023 12:54 experiments Have some code and graphs December 28, 2023 14:13 .gitignore Add PyPi Publish December 30, 2023 05:24 LICENSE Initial commit December 23, 2023 22:38 README.md Update README.md December 29, 2023 08:22 base_ref.py useless imports December 28, 2023 16:28 drugs_conf.py Have some code and graphs December 28, 2023 14:13 environment.yml Have some code and graphs December 28, 2023 14:13 just_chat.ipynb more reade December 28, 2023 15:11 just_chat.py more reade December 28, 2023 15:11 plot_divergence.ipynb Have some code and graphs December 28, 2023 14:13 requirements.txt Add PyPi Publish December 30, 2023 05:24 setup.py Add PyPi Publish December 30, 2023 05:24 View code [ ] DRuGS Stop messing around with finicky sampling parameters and just use DRuGS! The Problem: The Solution: Should I use DRuGS? Are there any negative side effects from using DRuGS? What kind of DRuGs can I use? How do I use DRuGs? What is a reasonable dose of DRuGS? In the meantime, please experiment with DRuGS! README.md DRuGS Stop messing around with finicky sampling parameters and just use DRuGS! This repo introduces Deep Random Micro-Glitch Sampling (DRuGS). The Problem: At a high level, the generative model landscape looks like first spending millions of dollars pretraining a giant model to predict the collective works of humanity, then giving those predictions to a dumb-as-rocks random number generator to kindly take into consideration in its role as the final arbiter over the multi-million dollar model's canonical output (which the model is then forced to commit to on its next prediction pass). This is kinda nuts. The Solution: DRuGS just inverts this scheme. Instead of using noise to sample from the model's predictions, DRuGS injects noise directly into the transformer layers at inference time, thereby varying what the model predicts. From here, simply selecting the most likely prediction is often enough to increase output variety while maintaining coherence. Intuitively, the primary advantage of this scheme (though there's more than one) is that the model has ample opportunity in its later layers to correct or account for our perturbations in its earlier layers. Should I use DRuGS? Absolutely. But do note that this proof of concept repo only supports LLaMA models. This isn't a technical limitation, and I'm very open to contributions from anyone willing to help me make DRuGS. You can get a sense of its generation quality in this colab chat with Alan Watts. Open In Colab Are there any negative side effects from using DRuGS? Negative side effects are difficult to identify subjectively, and in my experience DRuGs feel great the whole time you're using them. In theory however, yes, prolonged use of DRuGS can have negative side effects that get worse over time. Specifically, when injecting noise into layers < n, the hidden state vectors in all layers >=n will be conditioned on this noisy input, and if you're using kv-caching, that noise-conditioned prediction will remain in the cache only to be pertutbed again on the next forward pass. This library includes a cold_shower function, which periodically sobers up the cache after every t-predictions, or which you can elect to call yourself while the model is awaiting user input. This is to allow for some measure of theoretical purity, but again, in my experience it seems unnecessary, and using it means contending with periodically having to wait for your model to finish its shower before it can use more DRuGS. What kind of DRuGs can I use? While not an exhaustive list of the DRuGs that are theoretically possible, this repo provides implementations and experimental data for four types of DRuGs. These are Q, K, V, and A; which inject noise into the Query, Key, Value, and Attention head outputs, respectively. How do I use DRuGs? First, install this library. pip install git+https://github.com/EGjoni/DRUGS.git Then, import it into your project, and decide which and how much DRuGS you want your model to use. import torch from transformers import AutomodelForCausalLM, Autotokenizer, TextStreamer from drugs.dgenerate import DRUGS model_id = "NousResearch/Llama-2-7b-chat-hf" #or whatever LLaMA2 variant you prefer tokenizer = AutoTokenizer.from_pretrained(model_id) tokenizer.pad_token_id = tokenizer.eos_token_id sober_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") sober_model.eval() #prepare DRUGS, then inject into the model. drugs = DRUGS() drugs.set_A_dose_theta(0.1) model = drugs.inject(sober_model) You can then call model() as usual for a single forward pass, or use the DRUGS equivalent of model.generate as follows: streamer = TextStreamer(tokenizer) with torch.no_grad(): generated_tokens = model.Dgenerate( #capital 'D' before generate to distinguish from regular .generate() input_ids = tokenized_start, streamer = streamer ) Optionally, you can specify how deep you want you want to inject which type of DRuGs by defining a DRuG profile. injection_depth = 0.4 #how deep to shove the needle in (0 is first layer, 1 is last layer) spread = 0.1 #how many layers to dose on either side of the injection site (0 is no layers, 1 is all layers) drug_profile = ([ {'depth': (injection_depth-(spread*1.01)), 'peakratio': 0}, #ramp up {'depth': (injection_depth-spread), 'peakratio': 1}, #sustained peak {'depth': (injection_depth+spread), 'peakratio' : 1}, #sustained peak {'depth': (injection_depth+(spread*1.01)), 'peakratio' : 0}], #cool down 'ceil') drugs.set_A_dose_shape(drug_profile) For more examples, take a look at just_chat.ipynb What is a reasonable dose of DRuGS? The `dose_theta`` parameter basically just defines a maximum angle in radians by which to randomly rotate the A, Q, K, or V vectors. You probably shouldn't go past 0.1, but this kind of depends on which drug and where you're injecting it. But this is also kind of where things get interesting. Consider the following starting prompt: [INST] Write an epic rap battle between William Rowe Hamilton and Lord Kelvin[/INST] [Scene: A dark and dimly lit underground rap club, the air thick with anticipation. The crowd is on the edge of their seats as two of the greatest minds in science history take the stage. William Rowe Hamilton and Lord Kelvin are about to engage in an epic rap battle.] William Rowe Hamilton: Yo, I'm the king of the math game, With equations that'll make your head spin like a flame, I'm the one who brought you quaternions, And I'll leave you in the dust, like a poor excuse for a reason. Lord Kelvin: Hold up, Hamilton, you ain't ready, I'm the one who's got the science, the facts, and the A fully sober model predicts the most likely next token is "energy". If we store the hidden states at each layer for the vector corresponding to that prediction, we can visualize the effects of injecting various amounts of noise at various layers (keep an eye on the title at the top) K_dose_layer.mp4 K_dose_fullstack.mp4 A_dose_layer.mp4 A_dose_fullstack.mp4 (Interactive versions of these graphs, as well as ones for Q and V dosages are available as .html files in the experiments/ vergence_plots/Hamilton_* directory of this repo) To clarify these graphs: The prediction texts on the top right correspond solely to a (quite high) dosage theta of 0.7. The predictions listed are the top 10 most likely as per the sober model. THe ||| bars indicate likelihood as per the DRuGS augmented model, turning into - to indcate how far they fall short of the baseline prediction, or + to indicate how much they exceed the baseline prediction. Each video frame shows a different range of layers into which noise is being injected (as indicarted by the graph title at that frame) The horizontal axis shows the layer at which divergence is being measured. The vertical axis shows the degree of divergence at that layer. And the remaining axis shows the dose theta that was used to cause that degree of divergence. It might be a bit much to grok at a glance, but once you've wrapped your head around it, a few things might immediately stand out. First, we can add quite a lot of noise in earlier layers and the model very quickly drowns that noise out with its own signal. (This is likely part of why franken-merges work so well. It's not just that the residual stream keeps values in a reasonable region to avoid too much harm, it seem to be also that each layer of the model actively wants to push its inputs into something it can make sense of). Second, something special seems to happen in the middle layers that causes relatively large spikes in output divergence. And third, the most likely prediction changes, but generally remains reasonable. I feel like there's a lot more to play with and discover here, but, it's gonna need crowdsourcing. Personally my next step is to see (very seriously) explore the potential of DRuGS to control model hallucinations. Anyway -- critiques and contributions welcome. And I'll have a mistral implementation up soon. In the meantime, please experiment with DRuGS! About Stop messing around with finicky sampling parameters and just use DRuGS! Resources Readme License MIT license Activity Stars 172 stars Watchers 3 watching Forks 5 forks Report repository Releases No releases published Packages 0 No packages published Contributors 2 * @EGjoni EGjoni Eron Gjoni * @Josh-XT Josh-XT Josh XT Languages * HTML 76.9% * Jupyter Notebook 23.0% * Python 0.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.