[HN Gopher] AI engineers claim new algorithm reduces AI power co...
___________________________________________________________________
AI engineers claim new algorithm reduces AI power consumption by
95%
Author : ferriswil
Score : 122 points
Date : 2024-10-19 18:03 UTC (4 hours ago)
(HTM) web link (www.tomshardware.com)
(TXT) w3m dump (www.tomshardware.com)
| remexre wrote:
| Isn't this just taking advantage of "log(x) + log(y) = log(xy)"?
| The IEEE754 floating-point representation stores floats as sign,
| mantissa, and exponent -- ignore the first two (you quantitized
| anyway, right?), and the exponent is just an integer storing
| log() of the float.
| convolvatron wrote:
| yes. and the next question is 'ok, how do we add'
| dietr1ch wrote:
| I guess that if the bulk of the computation goes into the
| multiplications, you can work in the log-space and simply
| sum, and when the time comes to actually do a sum on the
| original space you can go back and sum.
| a-loup-e wrote:
| Not sure how well that would work if you're often adding
| bias after every layer
| kps wrote:
| Yes. I haven't yet read this paper to see what exactly it
| says is new, but I've definitely seen log-based
| representations under development before now. ( _More_ log-
| based than the regular floating-point exponent, that is. I
| don 't actually know the argument behind the exponent-and-
| mantissa form that's been pretty much universal even before
| IEEE754, other than that it mimics decimal scientific
| notation.)
| mota7 wrote:
| Not quite: It's taking advantage of (1+a)(1+b) = 1 + a + b +
| ab. And where a and b are both small-ish, ab is really small
| and can just be ignored.
|
| So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not
| the same! But it turns out, machine guessing apparently doesn't
| care much about the difference.
| amelius wrote:
| You might then as well replace the multiplication by the
| addition in the original network. In that case you're not
| even approximating anything.
|
| Am I missing something?
| tommiegannert wrote:
| Plus the 2^-l(m) correction term.
|
| Feels like multiplication shouldn't be needed for
| convergence, just monotonicity? I wonder how well it would
| perform if the model was actually trained the same way.
| _aavaa_ wrote:
| Original discussion of the preprint:
| https://news.ycombinator.com/item?id=41784591
| djoldman wrote:
| https://arxiv.org/abs/2410.00907
|
| ABSTRACT
|
| Large neural networks spend most computation on floating point
| tensor multiplications. In this work, we find that a floating
| point multiplier can be approximated by one integer adder with
| high precision. We propose the linear-complexity multiplication
| (L-Mul) algorithm that approximates floating point number
| multiplication with integer addition operations. The new
| algorithm costs significantly less computation resource than
| 8-bit floating point multiplication but achieves higher
| precision. Compared to 8-bit floating point multiplications, the
| proposed method achieves higher precision but consumes
| significantly less bit-level computation. Since multiplying
| floating point numbers requires substantially higher energy
| compared to integer addition operations, applying the L-Mul
| operation in tensor processing hardware can potentially reduce
| 95% energy cost by elementwise floating point tensor
| multiplications and 80% energy cost of dot products. We
| calculated the theoretical error expectation of L-Mul, and
| evaluated the algorithm on a wide range of textual, visual, and
| symbolic tasks, including natural language understanding,
| structural reasoning, mathematics, and commonsense question
| answering. Our numerical analysis experiments agree with the
| theoretical error estimation, which indicates that L-Mul with
| 4-bit mantissa achieves comparable precision as float8 e4m3
| multiplications, and L-Mul with 3-bit mantissa outperforms float8
| e5m2. Evaluation results on popular benchmarks show that directly
| applying L-Mul to the attention mechanism is almost lossless. We
| further show that replacing all floating point multiplications
| with 3-bit mantissa L-Mul in a transformer model achieves
| equivalent precision as using float8 e4m3 as accumulation
| precision in both fine-tuning and inference.
| onlyrealcuzzo wrote:
| Does this mean you can train efficiently without GPUs?
|
| Presumably there will be a lot of interest.
| crazygringo wrote:
| No. But it does potentially mean that either current or
| future-tweaked GPUs could run a lot more efficiently --
| meaning much faster or with much less energy consumption.
|
| You still need the GPU parallelism though.
| fuzzfactor wrote:
| I had a feeling it had to be something like massive waste
| due to a misguided feature of the algorithms that shouldn't
| have been there in the first place.
|
| Once the "math is done" quite likely it would have paid off
| better than most investments for the top people to have
| spent a few short years working with grossly underpowered
| hardware until they could come up with amazing results
| there before scaling up. Rather than grossly overpowered
| hardware before there was even deep understanding of the
| underlying processes.
|
| When you think about it, what we have seen from the latest
| ultra-high-powered "thinking" machines is truly so
| impressive. But if you are trying to fool somebody into
| believing that it's a real person it's still not "quite"
| there.
|
| Maybe a good benchmark would be to take a regular PC, and
| without reliance on AI just pull out all the stops and put
| all the effort into fakery itself. No holds barred, any
| trick you can think of. See what the electronics is capable
| of this way. There are some smart engineers, this would
| only take a few years but looks like it would have been a
| lot more affordable.
|
| Then with the same hardware if an AI alternative is not as
| convincing, something has got to be wrong.
|
| It's good to find out this type of thing before you go
| overboard.
|
| Regardless of speed or power, I never could have gotten an
| 8-bit computer to match the output of a 32-bit floating-
| point algorithm by using floating-point myself. Integers
| all the way and place the decimal where it's supposed to be
| when you're done.
|
| Once it's really figured out, how do you think it would
| feel being the one paying the electric bills up until now?
| jimmaswell wrote:
| Faster progress was absolutely worth it. Spending years
| agonizing over theory to save a bit of electric would
| have been a massive disservice to the world.
| BolexNOLA wrote:
| "A bit"?
| bartread wrote:
| Yes, a large amount for - in the grand scheme of things -
| a short period of time (i.e., a quantity of energy usage
| in an intense spike that will be dwarfed by energy usage
| over time) can accurately be described as "a bit".
|
| Of course, the impact is that AI will continue to become
| cheaper to use, and induced demand will continue the
| feedback loop driving the market as a result.
| pcl wrote:
| Isn't this paper pretty much about spending a few short
| years to improve the performance? Or are you arguing that
| the same people who made breakthroughs over the last few
| years should have also done the optimization work?
| fuzzfactor wrote:
| >the same people who made breakthroughs over the last few
| years should have also done the optimization work
|
| I never thought it would be ideal if it was otherwise, so
| I guess so.
|
| When I first considered neural nets from state-of-the art
| vendors to assist with some non-linguistic situations
| over 30 years ago, it wasn't quite ready for prime time
| and I could accept that.
|
| I just don't have generic situations all the time which
| would benefit me, so it's clearly my problems that have
| the deficiencies ;\
|
| What's being done now with all the resources being thrown
| at it is highly impressive, and gaining all the time, no
| doubt about it. It's nice to know there are people that
| can afford it.
|
| I truly look forward to more progress, and this may be
| the previously unreached milestone I have been detecting
| that might be a big one.
|
| Still not good enough for what I need yet so far though.
| And I can accept that as easily as ever.
|
| That's why I put up my estimation that not all of those
| 30+ years has been spent without agonizing over something
| ;)
| robomartin wrote:
| I posted this about a week ago:
|
| https://news.ycombinator.com/item?id=41816598
|
| This has been done for decades in digital circuits, FPGA's,
| Digital Signal Processing, etc. Floating point is both resource
| and power intensive and using FP without the use of dedicated FP
| processing hardware is something that has been avoided and done
| without for decades unless absolutely necessary.
| ujikoluk wrote:
| Explain more for the uninitiated please.
| ausbah wrote:
| a lot of things in the ML research space are rebranding an old
| concept w a new name as "novel"
| fidotron wrote:
| Right, the ML people are learning, slowly, about the importance
| of optimizing for silicon simplicity, not just reduction of
| symbols in linear algebra.
|
| Their rediscovery of fixed point was bad enough but the "omg if
| we represent poses as quaternions everything works better"
| makes any game engine dev for the last 30 years explode.
| kayo_20211030 wrote:
| Extraordinary claims require extraordinary evidence. Maybe it's
| possible, but consider that some really smart people, in many
| different groups, have been working diligently in this space for
| quite a while; so claims of 95% savings on energy costs _with
| equivalent performance_ is in the extraordinary category. Of
| course, we'll see when the tide goes out.
| vlovich123 wrote:
| They've been working on unrelated problems like structure of
| the network or how to build networks with better results. There
| have been people working on improving the efficiency of the
| low-level math operations and this is the culmination of those
| groups. Figuring this stuff out isn't super easy.
| throwawaymaths wrote:
| I don't think this claim is extraordinary. Nothing proposed is
| mathematically impossible or even unlikely, just a pain in the
| ass to test (lots of retraining, fine tuning etc, and those
| operations are expensive when you dont have already massively
| parallel hardware available, otherwise you're ASIC/FPGAing for
| something with a huge investment risk)
|
| If I could have a SWAG at it I would say a low resolution model
| like llama-2 would probably be just fine (llama-2 quantizes
| without too much headache) but a higher resolution model like
| llama-3 probably not so much, not without massive retraining
| anyways.
| Randor wrote:
| The energy claims up to ~70% can be verified. The inference
| implementation is here:
|
| https://github.com/microsoft/BitNet
| kayo_20211030 wrote:
| I'm not an AI person, in any technical sense. The savings
| being claimed, and I assume verified, are on ARM and x86
| chips. The piece doesn't mention swapping mult to add, and a
| 1-bit LLM is, well, a 1-bit LLM.
|
| Also,
|
| > Additionally, it reduces energy consumption by 55.4% to
| 70.0%
|
| With humility, I don't know what that means. It seems like
| some dubious math with percentages.
| Randor wrote:
| > I don't know what that means. It seems like some dubious
| math with percentages.
|
| I would start by downloading a 1.58 model such as:
| https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens
|
| Run the non-quantized version of the model on your
| 3090/4090 gpu and observe the power draw. Then load the
| 1.58 model and observe the power usage. Sure, the numbers
| have a wide range because there are many gpu/npu to make
| the comparison.
| kayo_20211030 wrote:
| Good one!
| sroussey wrote:
| Not every instruction on a CPU or GPU uses the same amount
| of power. So if you could rewrite your algorithm to use
| more power efficient instructions (even if you technically
| use more of them), you can save overall power draw.
|
| That said, time to market has been more important than any
| cares of efficiency for some time. Now and in the future,
| there is more of a focus on it as the expenses in equipment
| and power have really grown.
| littlestymaar wrote:
| How does the liked article relate to BitNet at all? It's
| about the "addition is all you need" paper which AFAIK is
| unrelated.
| manquer wrote:
| It is a click bait headline the claim itself is not
| extraordinary. the preprint from arxiv was posted here some
| time back .
|
| The 95% gains is specifically only for multiplication
| operations, inference is compute light and memory heavy in the
| first place so the actual gains would be far less smaller .
|
| Tech journalism (all journalism really) can hardly be trusted
| to publish grounded news with the focus on clicks and revenue
| they need to survive.
| kayo_20211030 wrote:
| Thank you. That makes sense.
| kayo_20211030 wrote:
| re: all above/below comments. It's still an extraordinary
| claim.
|
| I'm not claiming it's not possible, nor am I claiming that it's
| not true, or, at least, honest.
|
| But, there will need to be evidence that using real machines,
| and using real energy an _equivalent performance_ is
| achievable. A defense that "there are no suitable chips" is a
| bit disingenuous. If the 95% savings actually has legs some
| smart chip manufacturer will do the math and make the chips. If
| it's correct, that chip making firm will make a fortune. If
| it's not, they won't.
| stefan_ wrote:
| I mean, all these smart people would rather pay NVIDIA all
| their money than make AMD viable. And yet they tell us its all
| MatMul.
| kayo_20211030 wrote:
| Both companies are doing pretty well. Why don't you think AMD
| is viable?
| nelup20 wrote:
| AMD's ROCm just isn't there yet compared to Nvidia's CUDA.
| I tried it on Linux with my AMD GPU and couldn't get things
| working. AFAIK on Windows it's even worse.
| syntaxing wrote:
| I'm looking forward to Bitnet adaptation. MS just released a tool
| for it similar to llamacpp. Really hoping major models get
| retrained for it.
| andrewstuart wrote:
| The ultimate "you're doing it wrong".
|
| For he sake of the climate and environment it would be nice to be
| true.
|
| Bad news for Nvidia. "Sell your stock" bad.
|
| Does it come with a demonstration?
| talldayo wrote:
| > Bad news for Nvidia. "Sell your stock" bad.
|
| People say this but then the fastest and most-used
| implementation of these optimizations is always written in
| CUDA. If this turns out to not be a hoax, I wouldn't be
| surprised to see Nvidia prices _jump_ in correlation.
| mouse_ wrote:
| Hypothetically, if this is true and simple as the headline
| implies -- AI using 95% less power doesn't mean AI will use 95%
| less power, it means we will do 20x more AI. As long as it's
| the current fad, we will throw as much power and resources at
| this as we can physically produce, because our economy depends
| on constant, accelerating growth.
| Dylan16807 wrote:
| Bad news for Nvidia how? Even ignoring that the power savings
| are only on one type of instruction, 20x less power doesn't
| mean it runs 20x faster. You still need big fat GPUs.
|
| If this increases integer demand and decreases floating point
| demand, that moderately changes future product design and
| doesn't do much else.
| idiliv wrote:
| Duplicate, posted on October 9:
| https://news.ycombinator.com/item?id=41784591
| asicsarecool wrote:
| Don't assume this isn't already in place at the main AI companies
| DesiLurker wrote:
| validity of the claim aside, why dont they say reduces by 20
| times instead of 95%. its much better perspective of a fraction
| when fraction is tiny.
| hello_computer wrote:
| How does this differ from Cussen & Ullman?
|
| https://arxiv.org/abs/2307.01415
| GistNoesis wrote:
| Does https://en.wikipedia.org/wiki/Jevons_paradox apply in this
| case ?
| narrator wrote:
| Of course. Jevons paradox always applies.
| gosub100 wrote:
| Not necessarily a bad thing: this might give the AI charlatans
| enough time to actually make something useful.
| panosv wrote:
| Lemurian Labs looks like it's doing something similar:
| https://www.lemurianlabs.com/technology They use the Logarithmic
| Number System (LNS)
| andrewstuart wrote:
| Here is the Microsoft implementation:
|
| https://github.com/microsoft/BitNet
| quantadev wrote:
| I wonder if someone has feed this entire "problem" into the
| latest Chat GPT-01 (the new model with reasoning capability), and
| just fed it in all the code for a Multilayer Perceptron and then
| given it the task/prompt of finding ways to implement the same
| network using only integer operations.
|
| Surely even the OpenAI devs must have done this like the minute
| they got done training that model, right? I wonder if they'd even
| admit it was an AI that came up with the solution rather than
| just publishing it, and taking credit. haha.
| chx wrote:
| You are imaging LLMs are capable of much more than they
| actually are. Here's the _only_ thing they are good for.
|
| https://hachyderm.io/@inthehands/112006855076082650
|
| > You might be surprised to learn that I actually think LLMs
| have the potential to be not only fun but genuinely useful.
| "Show me some bullshit that would be typical in this context"
| can be a genuinely helpful question to have answered, in code
| and in natural language -- for brainstorming, for seeing common
| conventions in an unfamiliar context, for having something
| crappy to react to.
|
| > Alas, that does not remotely resemble how people are pitching
| this technology.
| didgetmaster wrote:
| Maybe I am just a natural skeptic, but whenever I see a headline
| that says 'method x reduces y by z%'; but when you read the text
| it instead says that optimizing some step 'could potentially
| reduce y by up to z%'; I am suspicious.
|
| Why not publish some actual benchmarks that prove your claim in
| even a few special cases?
| TheRealPomax wrote:
| Because as disappointing as modern life is, you need clickbait
| headlines to drive traffic. You did the right thing by reading
| the article though, that's where the information is, not the
| title.
| phtrivier wrote:
| Fair enough, but then I want a way to penalize publishers for
| abusing clickbait. There is no "unread" button, and there is
| no way to unsubscribe to advertisement-based sites.
|
| Even on sites that have a "Like / Don't like" button, my
| understanding is that clicking "Don't like" is a form of
| "engagement", that the suggestion algorithm are going to
| reward.
|
| Give me a button that says "this article was a scam", and
| have the publisher give the advertisement money back. Of
| better yet, give the advertisement money to charity / public
| services / whatever.
|
| Take a cut of the money being transfered, charge the
| publishers for being able to get a "clickbait free" green
| mark if they implement the scheme.
|
| Track the kind of articles that generate the most clickbait-
| angry comment. Sell back the data.
|
| There might a business model.
| NineStarPoint wrote:
| I doubt there's a business model there because who is going
| to opt in to a scheme that loses them money?
|
| What could work is social media giving people an easy
| button to block links to specific websites from appearing
| in their feed, or something along those lines. It's a nice
| user feature, and having every clickbait article be a
| chance someone will choose to never see your website again
| could actually reign in some of the nonsense.
| baq wrote:
| OTOH you have a living proof that an amazingly huge neural
| network can work on 20W of power, so expecting multiple orders
| of magnitude in power consumption reduction is not
| unreasonable.
| dragonwriter wrote:
| Well, one, because the headline isn't from the researchers, its
| from a popular press report (not even the one posted here,
| originally, this is secondary reporting of another popular
| press piece) and isn't what the paper claims so it would be odd
| for the paper's authors to conduct benchmarks to justify it.
| (And, no, even the "up to 95%" isn't from the paper, the cost
| savings are cited per operation depending on operation and the
| precision the operation is conducted at, are as high as 97.3%,
| are based on research already done establishing the energy cost
| of math operations on modern compute hardware, but no end-to-
| end cost savings claim is made.)
|
| And, two, because the actual energy cost savings claimed aren't
| even the experimental question -- the energy cost differences
| between various operations on modern hardware have been
| established in other research, the experimental issue here was
| whether the mathematical technique that enables using the lower
| energy cost operations performs competitively on output quality
| with existing implementations when substituted in for LLM
| inference.
| littlestymaar wrote:
| Related: https://news.ycombinator.com/item?id=41784591 10 days
| ago
| greenthrow wrote:
| The trend of hyping up papers too early on is eroding people's
| faith in science due to poor journalism failing to explain that
| this is theoretical. The outlets that do this should pay the
| price but they don't, because almost every outlet does it.
| holoduke wrote:
| I don't think algorithms will change energy consumption. There is
| always max capacity needed in terms of computing. If tomorrow a
| new algorithm increases the performance 4 times, we will just
| have 4 times more computing.
___________________________________________________________________
(page generated 2024-10-19 23:00 UTC)