[HN Gopher] QuIP#: 2-bit Quantization for LLMs
___________________________________________________________________
QuIP#: 2-bit Quantization for LLMs
Author : jasondavies
Score : 195 points
Date : 2023-12-08 23:37 UTC (23 hours ago)
(HTM) web link (cornell-relaxml.github.io)
(TXT) w3m dump (cornell-relaxml.github.io)
| skavi wrote:
| can anyone comment on running the 2b quantized llama 70b on
| consumer cards like the 4090?
| beacon294 wrote:
| I have quantized Llama-2-chat-70b to 2q_k with llama.cpp then
| run on a 4090 for 50/83 layers. I was able to get 1.3 tokens
| per second. It answers this question, which smaller models have
| trouble with:
|
| <s>[INST]Hello. The functions addition, multiplication, and
| exponentiation can be considered each as a higher order of the
| previous. For example, 2+2+2 = 2*3 which is significant because
| there are three twos, which is the multiplication factor.
| Likewise 2*2*2=2^3, again, the power number represents the
| count of the numbers being multiplied. What is this concept and
| what is the next higher order function?[/INST] The concept you
| are referring to is called the "iterative" or "recursive"
| nature of arithmetic operations. Each operation can be seen as
| a repeated application of the previous operation, with the
| multiplication factor representing the number of times the
| operation is applied.
|
| For example, the addition of two numbers, 2+2, can be seen as a
| repeated application of the multiplication operation, where the
| multiplicand is 2 and the multiplier is 3 (i.e., 2*3 = 6, which
| is the result of the addition). Similarly, the multiplication
| of two numbers, 2*2, can be seen as a repeated application of
| the exponentiation operation, where the base is 2 and the power
| is 3 (i.e., 2^3 = 8, which is the result of the
| multiplication).
|
| The next higher order function would be the "tetration"
| operation, which is the iterative application of the
| exponential function. In other words, tetration(n,x) =
| x^(x^(x^(...(x^n)...))), where n is the number of times the
| exponential function is applied.
|
| For example, tetration(2,2) = 2^(2^(2^(2))), which can be
| simplified to 2^16, or 65536.
|
| It's worth noting that this concept can be extended even
| further to higher levels of abstraction, such as "hyper-
| operations" and "super -operations", but these are more
| advanced and less commonly used concepts in mathematics.*
| beacon294 wrote:
| The answer seems kind of low quality, but it got further than
| others. It could probably self correct.
| taneq wrote:
| I don't know too many humans who could answer that half as
| well.
| quickthrower2 wrote:
| To be fair it is a memory/recite test mostly. "Define
| this"
| epcoa wrote:
| "For example, the addition of two numbers, 2+2, can be
| seen as a repeated application of the multiplication
| operation,"
|
| I bet most humans ever born did not know what the word
| "multiplication" meant. Not sure how your metric is
| useful.
| philipswood wrote:
| Sometimes, to achieve some perspective, I also zoom out
| and declare normal over the estimated +-120,000,000,000
| people who have ever lived.
| skavi wrote:
| to clarify, is that with
| https://huggingface.co/relaxml/Llama-2-70b-chat-E8P-2Bit
| beacon294 wrote:
| It is Llama-2-70b-chat. I quantized it to 2q_k using
| `quantize` with llama.cpp.
| skavi wrote:
| So your experience isn't representative of the work
| presented by this post? Or does llama.cpp use the same
| technique for quantization?
| beacon294 wrote:
| I don't know, hopefully it will help set general
| expectations.
| skykooler wrote:
| Note that it got all the math wrong. 2+2 is a repetition of
| the succession operator, not multiplication; 2+2 equals 4,
| not 6; multiplication of two numbers is repeated addition,
| not exponentiation; 2*2 equals 4, not 8; and tetration(2,2)
| is 4, not 65536. This is actually the invariant in that
| n-ation(2,2) equals 4 for all natural numbers n.
| Philpax wrote:
| To be clear, this is _not_ using QuIP#, and is thus not
| particularly relevant to the discussion at hand.
| lhl wrote:
| It's been on my list to do a proper shootout of all the various
| new quant formats floating around (my list here: https://llm-
| tracker.info/books/llms/page/quantization-overvi...) but a lot
| of them don't have very good production code yet (eg, a few
| months ago, when I tried OmniQuant, some of the important bits
| of code wasn't even included and had to be gotten directly from
| the authors: https://llm-
| tracker.info/books/llms/page/omniquant).
|
| If you're looking for the best widely deployed quant format
| atm, it's probably ExLlamaV2's EXL2 - it supports arbitrary bpw
| w/ a calibration file, and also 8-bit kvcache support. I
| haven't tested EXL2 much at lower bpws though.
|
| Note, both llama.cpp and AirLLM allow layer offloading to
| system memory (or in AirLLM's case, even to disk?!).
|
| r/LocalLlama probably is the best place to search for if you're
| looking for people's experiences w/ quants. I know some people
| have been testing, like:
| https://www.reddit.com/r/LocalLLaMA/comments/17klaa5/tested_...
| 0xDEADFED5 wrote:
| i humbly request you to add mlc-llm to your quant test
| when/if you get around to doing it
| lhl wrote:
| Sure, I think their quant format is pretty basic, something
| similar to bnb q4 - my plan will to be scripting a
| framework for testing, so should do that as well since the
| omniquant implementation is in mlc-llm anyways.
| acosmism wrote:
| i was trying to get this to work with mlc-llm. i'd
| appreciate any pointers
| acosmism wrote:
| more specifically on a non-cuda gpu - mali on orangepi via
| opencl
| brucethemoose2 wrote:
| > https://llm-tracker.info/books/llms/page/quantization-
| overvi...
|
| This is a very cool resource, thanks!
|
| Gems like this, even in areas I follow pretty closely, are
| why I keep coming back to HN.
| thulle wrote:
| Better output than the smaller llamas in my limited testing,
| but it's surprisingly slow:
|
| Output generated in 101.74 seconds (0.98 tokens/s, 100 tokens,
| context 82, seed 532878022)
|
| Output generated in 515.46 seconds (0.99 tokens/s, 511 tokens,
| context 27, seed 660997525)
|
| Checking nvidia-smi it stalls at ~130W (out of ~470 W max)
| power usage, ~25% GPU usage and ~10% memory bandwidth usage.
| There's fairly much traffic on the pci-bus though, and the
| python process is stable at 100% usage of one core. GPU
| possibly limited by some thing handled in python? Pausing the
| GPU-accelerated video-decoding of a twitch stream it get a
| surprisingly large boost:
|
| Output generated in 380.42 seconds (1.34 tokens/s, 511 tokens,
| context 26, seed 648992918)
| bongwater_OS wrote:
| One of the best papers I've read in a long time. This could be
| huge.
| skykooler wrote:
| I wonder whether quantization to 1-bit would be functional?
| gumby wrote:
| There was a paper from the Allen institute from around 2017
| successfully using 1 bit quantization but I can't find it right
| now. We started using it where I was working at the time but
| I'm no longer there so I don't know how it all turned out.
| orbital-decay wrote:
| Functional? Sure, it will work. Useful? Unlikely. It would be
| surprising to see quantization of pretrained models to surpass
| the existing research around binary NNs (trained from scratch
| as 1-bit while using full precision for the most critical
| parts)
| rolisz wrote:
| If we get 1-bit quantization, wouldn't it be basically a bunch
| of nested if's and else's?
| compinter wrote:
| So, just like the rest of my code then
| hskalin wrote:
| So a decision tree? They are good for some tasks I reckon
| SeanAnderson wrote:
| Just to make sure I'm understanding this correctly.
|
| This paper signals that the authors have found a way to run Llama
| 2 70B, but with 1/8th the VRAM requirements as compared to the
| original model, right?
|
| And the output is on-par with the original along some metrics
| (ArcE/PiQA), within 25% on others (Wiki/C4), and the trajectory
| of their progress hints that there's even more ground to gain in
| the future?
| samus wrote:
| Your last paragraph is the key. Without their improvements, the
| quality tradeoff would have been hard to stomach.
| londons_explore wrote:
| When considering quality, you should really compare not to
| the original model, but to smaller models which use the same
| vram/compute
| karmasimida wrote:
| But people use large model for better metrics, this defeats
| the purpose.
| lxe wrote:
| Already works on oobabooga as of a few days ago:
| https://github.com/oobabooga/text-generation-webui/issues/47...
|
| Need a few extra steps: https://github.com/oobabooga/text-
| generation-webui/pull/4803
| tarruda wrote:
| If this quantization method works with smaller models, it would
| enable running up to 33B models with only 12GB VRAM.
|
| Especially important for democratizing access to Mistral MoE new
| model.
| MrNeon wrote:
| IIRC quantizing small models causes a higher relative drop in
| the metrics.
| pyinstallwoes wrote:
| Since a pixel can have more states than binary, could you have
| more space and compute by leveraging RGBA-space for data/compute
| than binary?
|
| Maybe a stupid question.
| wizzard0 wrote:
| A pixel takes more than 1 bit to store, too
| pyinstallwoes wrote:
| on a video card? I'm not sure that is true wrt to the
| geometry and charge.
| pyinstallwoes wrote:
| Found this https://news.ycombinator.com/item?id=27409371
|
| Kinda related. Especially the comments.
| karmasimida wrote:
| For quantization, you should always verify directly on your own
| intended tasks, not trusting the quantization will preserve
| accuracy on a boarder spectrums of tasks, because surprises are
| not that infrequent.
| DrNosferatu wrote:
| Does LM Studio support it?
|
| By the way, what's your favorite easy-to-use LLM front end?
| FooBarWidget wrote:
| LM Studio, sort of. Unfortunately my Macbook is incapable of
| running all but the smallest models. It would be great if LM
| Studio can connect to a remote model running on a GPU server
| that I can rent.
| jerpint wrote:
| You could set up something similar using gradio quite easily:
|
| https://www.jerpint.io/blog/model-inference/
| omneity wrote:
| How does this 2-bit quantization method compare to HQQ which was
| posted yesterday?
|
| https://news.ycombinator.com/item?id=38563537
| shahbazac wrote:
| Can someone answer CS 101 questions about this please.
|
| I know there are other methods related to matrix factorization,
| but I'm asking specifically about quantization.
|
| Does quantization literally mean the weight matrix floats are
| being represented using fewer bits than the 64 bit standard?
|
| Second, if fewer bits are being used, are CPUs able to do math
| directly on fewer bits? Aren't CPU registers still 64 bit? Are
| these floats converted back to 64 bit for math, or is there some
| clever packing technique where a 64 bit float actually represents
| many numbers (sort of a hackey simd instruction)? Or do modern
| CPUs have the hardware to do math on fewer bits?
| garrisonhh wrote:
| Yes but no. The actual values represented by the quantized bits
| don't use a representation akin to IEEE floating point, but
| they are able to act like floating point values due to
| mathematical transformations during propagation. The floating
| point values a quantized value corresponds to are chosen using
| some kind of precomputation depending on the quantization
| method
| p1esk wrote:
| This is for GPUs, not CPUs. GPUs do have lower precision ALUs
| to do math on fewer bits. Though not 2 bits - I believe there's
| support for 1, 4 and 8 bit computation in modern Nvidia cards.
|
| But even without such support there's a benefit of model size
| compression so that bigger models can fit in GPU memory,
| eliminating costly CPU/GPU data transfers.
| saberience wrote:
| I'm a layperson when it comes to this topic but does this mean
| every value in the network is a value from 00 to 11? I.e: 00, 01,
| 10, and 11?
|
| I struggle to understand how a network with only two bits of
| precision could ever generate text or numbers or anything really.
|
| Is my intuition wrong here? If so, can someone give an example of
| what it means to quantize the network down to 2 bits only.
| p1esk wrote:
| It's better to think about mapping groups of weights to an
| entry in a large codebook, and then using clever methods to do
| weight lookups.
|
| 2 bits of precision per weight is perfectly fine as long as you
| have enough weights. The information encoded by a neural
| network is measured in total number of bits, so you can
| compress it by either reducing number of weights or reducing
| number of bits per weight.
___________________________________________________________________
(page generated 2023-12-09 23:02 UTC)