[HN Gopher] New LLM optimization technique slashes memory costs
___________________________________________________________________
New LLM optimization technique slashes memory costs
Author : hochmartinez
Score : 428 points
Date : 2024-12-13 19:14 UTC (4 days ago)
(HTM) web link (venturebeat.com)
(TXT) w3m dump (venturebeat.com)
| dboreham wrote:
| TFP: https://arxiv.org/abs/2410.13166
| vlovich123 wrote:
| TFS: https://github.com/SakanaAI/evo-memory
| vlovich123 wrote:
| Wonder how this compares with Microsoft's HeadKV paper [1] which
| claims a 98% percent reduction in memory while retaining 97% of
| the performance.
|
| [1] https://arxiv.org/html/2410.19258v3
| benatkin wrote:
| Seems like a different thing. That paper appears to be memory
| reduction in caching while article appears to be memory
| reduction in content.
| vlovich123 wrote:
| They're both exploring the same space of optimizing the
| memory needed by the KV cache which is essentially another
| name for the context window (no one elides the KV cache as
| otherwise you're doing N^2 math to do attention). They're
| exploring different approaches to achieve the same goal and
| they may be both possible to apply simultaneously to reduce
| the attention mechanism to almost 0 memory usage which would
| be really cool, but I'm curious how they compare against each
| other individually.
| benatkin wrote:
| That sounds like a stretch to me. If not, I'm impressed how
| the articles can describe such similar things in such
| different terms.
| vlovich123 wrote:
| The only memory mechanism within an LLM as far as I know
| is the attention mechanism where it compares all previous
| tokens to generate a probability distribution for the
| next token to generate. The attention mechanism has a
| thing called a KV cache to take the O(n^2) matrix math
| down to O(n) by caching and reusing the results of some
| math from previous tokens. The size of how many tokens
| the context will cover is called the context window (e.g.
| 128k for Llama).
|
| The articles use very similar verbiage.
|
| > The context window can be considered the model's
| working memory
|
| Snip
|
| > Universal transformer memory optimizes prompts using
| neural attention memory models (NAMMs), simple neural
| networks that decide whether to "remember" or "forget"
| each given token stored in the LLM's memory.
|
| snip
|
| > Meanwhile, by discarding unnecessary tokens, NAMM
| enabled the LLM model to save up to 75% of its cache
| memory while performing the tasks.
|
| You just have to be familiar with the wording in the
| space and read enough literature. Here's more direct
| wording from the NAMM paper:
|
| > NAMMs use evolution to optimize the performance of LMs
| by pruning their KV cache memory. Evolved NAMMs can be
| zero-shot transferred to other transformers, even across
| input modalities and task domains.
|
| This is all related work about shrinking the size of the
| KV cache as the context grows both due to memory and it
| also has a speed up effect since you're not having to
| attend all the tokens (O(n) -> sublinear with the size of
| the context).
|
| Context is critical in the LLM answering correctly and
| remembering all the information given to it + everything
| it said. Typical limits for open models these days are
| 128k but with techniques like this it could scale even
| further allowing better performance on thing like code
| completion.
| benatkin wrote:
| I thought the context would also have floating point
| numbers so that tokens would be included in a more fuzzy
| way, and that when requests are sent it would result in
| loading slightly different tokens into the cache. Yeah my
| understanding certainly is limited and I'd like to study
| it more. Thanks for the response, I see more similarity
| now.
| vlovich123 wrote:
| The word you're looking for is latent space and yes,
| everything in the compute graph, including context cache
| & compute is done in latent space. Literal input tokens
| are first converted to latent space through the embedding
| layer and literal output tokens are generated by
| converting the last compute tensor into token
| probabilities & taking the most probable token.
| Everything in the middle though happens in the "floating
| point" latent space.
|
| When you hear something like "it's attending all previous
| tokens" IMHO it's not strictly the correct explanation
| since you're attending through latent space which doesn't
| actually correspond 1:1 with tokens but is a
| multidimensional representation of that token & all
| preceding tokens as understood by _that_ attention head.
| But conceptually it 's how it's described because the
| size of your context goes up by 1 tensor for every token
| you process, even though applying attention actually ends
| up changing all tensors in the KV cache (hence self-
| attention). Also important to note that each attention
| head within each layer has it's own KV cache. LLMs are an
| autoregressive family of models where the output of each
| layer feeds into the input of the next and each layer has
| a transformer performing attention. That's another reason
| why it's not strictly correct to think of it as tokens
| make up your context because there's actually many many
| contexts within a transformer model. That's why your 128k
| context window can be ~15 GiB for a naiive inference
| implementation - 128k context window * 1024 *
| 1024-element tensor * 2 bytes per tensor * 8 attention
| heads * 8 layers (or something along those lines). And
| that's what this work is talking about shrinking (as does
| the HeadKV).
|
| > tokens would be included in a more fuzzy way, and that
| when requests are sent it would result in loading
| slightly different tokens into the cache
|
| The entire process of LLMs is generally actually 100%
| deterministic based on the same inputs & given a fixed
| seed for the RNG (modulo bugs in the inference math /
| bugs in HW/SW for the accelerator). Some inference
| implementations don't guarantee this property in the face
| of concurrent requests & you can't control the seed for
| hosted LLMs which is why it seems like random responses
| for the same query.
| benatkin wrote:
| The KV cache feels more like a graph to me, like in the
| RDF sense. Each parameter could be numbered and given a
| URL it seems. I have some studying to do. I think
| building a simple neural net and looking at raw data for
| context in whatever LLM I'm playing with in Ollama are
| good things to try.
| oblio wrote:
| > they may be both possible to apply simultaneously to
| reduce the attention mechanism to almost 0 memory usage
| which would be really cool
|
| https://matt.might.net/articles/why-infinite-or-
| guaranteed-f...
| tsukikage wrote:
| This isn't like lossless compression. Both techniques
| involve throwing lots of information away, with the
| justification that doing so does not significantly affect
| the end result.
|
| The extent to which using both the techniques together
| will help will depend on how much overlap there is
| between the information each ends up discarding.
| oblio wrote:
| My joke was more along the lines of entropy. Entropy is
| information and you can't throw away all of it, otherwise
| you have nothing useful left.
| lxgr wrote:
| Hence the idea to only throw away almost all of it.
| vlovich123 wrote:
| Modern LLMs are still quite inefficient in their
| representation of information. We're at like the DEFLATE
| era and we've still yet to invent zstd where there's only
| marginal incremental gains; so right now there's a lot of
| waste to prune away.
| punkpeye wrote:
| Any real-world (open-source) implementations of this?
| ComputerGuru wrote:
| This only decreases memory cost of input context window, not the
| memory cost to load and run the models.
| freehorse wrote:
| Context window requires ram too.
| ynniv wrote:
| "Only"
| keyle wrote:
| I agree with you though, the title is misleading.
| BoorishBears wrote:
| Title is perfect. Their typical audience probably understands
| "memory" better than "context window", but then if you've
| actually deployed these systems it's not difficult to go the
| other way, from "memory" to "context window" since the
| context window specifically is known to take additional VRAM
| over the model itself
| solarkraft wrote:
| And that's what matters the most! To me, at small model sizes
| (1-8B), anyway. A few thousans tokens already bog my RAM down
| quite a lot and I'd love to have more - I'd go as far as saying
| that context greatly determines LLM capability at this point.
| danielbln wrote:
| Yes, pretraining and post-training is nice and important, but
| in-context learning turns LLMs from toys into tools.
| richwater wrote:
| This is for inference right? Not training?
| lawlessone wrote:
| doesn't training require inference? so i guess it would help
| there too?
| fzzzy wrote:
| Training doesn't require inference. It uses back-propagation,
| a different algorithm.
| bitvoid wrote:
| Backpropagation happens after some number of inferences.
| You need to infer to calculate a loss function to then
| backprop from.
| boringg wrote:
| Yeah but training requires the larger memory deployment data
| center infra
| pizza wrote:
| It's for KV caching. In most conversations that will mean
| inference. But you _can_ do reinforcement learning using
| sampled sequences, and you _could_ use KV caching to speed that
| up too, so that would be an instance where training could get a
| slight boost.
| yishanchuan wrote:
| interesting
| lawlessone wrote:
| So it's like a garbage collector for prompts?
| cowsandmilk wrote:
| More of lossy compression
| aziaziazi wrote:
| Does lossy means you may see previous inputs change ?!
| TkTech wrote:
| Lossy doesn't necessarily imply that it is non-
| deterministic, just irreversible.
| lxgr wrote:
| Only the model's view, doesn't have to be yours, just like
| you can participate in a long conversation without perfect
| memory that might in retrospect slightly differ from a
| recording.
| verdverm wrote:
| with the added bonus that these things are unreliable and the
| compressor could drop important tokens
| tharmas wrote:
| Does this mean us plebs can run LLMs on gimped VRAM Nvidia lower
| end cards?
| swifthesitation wrote:
| I don't think so. It seems to just lower the ram needed for the
| context window. Not for loading the model on the vram.
| iandanforth wrote:
| Boringness classifier! Pretty cool because this implies the large
| models already know what is useless and what isn't.
| lxgr wrote:
| > NAMMs are trained separately from the LLM and are combined
| with the pre-trained model at inference time [...]
| odyssey7 wrote:
| Is it possible that after 3-4 years of performance optimizations,
| both algorithmic and in hardware efficiency, it will turn out
| that we didn't really need all of the nuclear plants we're
| currently in the process of setting up to satisfy the power
| demands of AI data centers?
| NhanH wrote:
| It seems (feels?) likely that demand for LLM is elastic,
| especially when it comes to specialized niche. Less power
| requirements just mean we run more of them in parallel for
| stuffs, so the power needs is gonna be growing anyway.
| groceryheist wrote:
| If we're "lucky" (in an AI-optimist sense) we'll need the
| nuclear plants despite efficiency increases.
| Trasmatta wrote:
| Are we setting up nuclear plants for AI data centers? If so, I
| see that as a win all around. We need to rely more on nuclear
| power, and I'll take whatever we can get to push us in that
| direction.
| boringg wrote:
| How else will we get manufacturing gains for a mars base
| nuclear system?
| fragmede wrote:
| Jevons paradox says as things get more efficient, usage goes
| up. In this case, even if AI data centers don't pan out, I
| think we'll still find use for the electricity they generate.
| tbrownaw wrote:
| No, as the things using that power get better (newer models
| keep getting less garbagey) and cheaper (faster hardware and
| more efficient use of power), people will keep coming up with
| more things to use them for.
| TkTech wrote:
| No, not really. AWS getting more power and space efficient
| chips didn't reduce total power demand, they just added more
| cores.
|
| Even if the data centers didn't keep up with available
| capacity, energy demanding industry move to and expand with
| sources of power, like aluminum production.
| _aavaa_ wrote:
| Nobody is building nuclear power plants for data centres. A few
| people have signed some paperwork saying that they would buy
| electricity from new nuclear plants if they could deliver it at
| a certain price, a price mind you that has not been done
| before. Others are trying to restart an existing reactor at
| three mile island (a thing that has never been done before, and
| likely won't be done now since the reactor was shut down due to
| being too expensive to run).
|
| And certainly nobody is building one in the next 3-4 years;
| they'd be lucky to finish the paperwork in that time.
|
| What is actually going to power them is solar, wind, and
| batteries:
| https://www.theverge.com/2024/12/10/24317888/googles-data-ce...
| standeven wrote:
| And unfortunately, gas and coal in the meantime.
|
| https://www.theguardian.com/technology/2024/sep/15/data-
| cent...
| robocat wrote:
| That article is terribly vague.
|
| Electric cars are causing exactly the same problem.
|
| Also the "recs" appear to be based on a lie. Overall, an
| increase in load can only be green if there is added new
| green generation to service that load.
| _aavaa_ wrote:
| Except the alternative to electric are petrol/diesel cars
| which are worse than electric cars run on gas or coal.
| The pollution no longer occurs in population zones, and
| the grid can be cleaned up without changing the car.
|
| The alternatives for these data centres are either build
| renewables or not build the data centres, both of which
| are better.
| aitchnyu wrote:
| Could be a good candidate for factobattery. Overbuild the
| system, run them at full speed at peak solar generation, then
| underclock them at night.
|
| https://www.moderndescartes.com/essays/factobattery/
| ForHackernews wrote:
| This is a really interesting idea! Of course, in practice
| that will just mean crypto-token mining rather than
| anything useful.
| numpad0 wrote:
| Then why no one seem to be doing it?
| adelpozo wrote:
| Or solar in space, which some have already heard of Lumen
| Orbit https://www.ycombinator.com/companies/lumen-orbit
| cratermoon wrote:
| Space-based solar power contains little intrinsic advantage
| that we can get "only from space." It looks like a wash at
| best, and the astronomers would say "don't bother."
| https://dothemath.ucsd.edu/2012/03/space-based-solar-power/
| adelpozo wrote:
| Yeah, but what I found thought provoking is what if you
| send the solar panels and the datacenter as well for
| training. No need to transmission of power down to earth.
| I guess then it becomes a heat dissipation and hardware
| upgrade and maintenance. But again, thought provoking.
| cma wrote:
| > Nobody is building nuclear power plants for data centres. A
| few people have signed some paperwork saying that they would
| buy electricity from new nuclear plants if they could deliver
| it at a certain price, a price mind you that has not been
| done before.
|
| Not building new, but I think Microsoft paying to restart a
| reactor at Three Mile Island for their datacenter is much
| more significant than you make the deals sound:
|
| https://www.theguardian.com/environment/2024/sep/20/three-
| mi...
| portaouflop wrote:
| They say it's going to be online in 2028.
|
| Are you willing to bet that they won't have 3 mile island
| operational by 2030?
| throwup238 wrote:
| _> Constellation closed the adjacent but unconnected Unit
| 1 reactor in 2019 for economic reasons, but will bring it
| back to life after signing a 20-year power purchase
| agreement to supply Microsoft's energy-hungry data
| centers, the company announced on Friday._
|
| The reactor they're restarting was operational just five
| years ago. It's not a fully decommissioned or melted down
| reactor and it's likely all their licensing is still
| valid so the red tape, especially environmental studies,
| is mostly irrelevant. Getting that reactor back up and
| running will be a lot simpler than building a new one.
| c-linkage wrote:
| The largest problem will be finding qualified and vetted
| personnel. All the people who worked at the plant when it
| closed five years ago had to find jobs elsewhere. Even
| though the plant was an important employer in Middletown,
| I don't know if those former employees will be willing
| the quit there current jobs to go back, especially if
| there is a risk the plant will just be shut down again
| when it once again becomes too expensive to operate.
| lopis wrote:
| It's all been pretty much greenwashing to distract for the
| real impact of all the AI infrastructure in the energy and
| water supply.
| _aavaa_ wrote:
| Microsoft isn't paying to restart. A PPA is a contract
| saying they will purchase electricity for a specified price
| for a fixed term. Three Mile Island needs to be able to
| produce the electricity for the specified price for
| Microsoft to pay buy it. If it's above that price Microsoft
| is off the hook.
| yunohn wrote:
| Look, I agree that nuclear is difficult, but Google and
| Microsoft have publicly committed to those projects you're
| mentioning. I don't understand your dismissive tone that all
| of it is hogwash? This is one of those HN armchair comments.
| dx034 wrote:
| But not just for AI, for all their data center operations.
| yard2010 wrote:
| Google and Microsoft won't do anything that doesn't
| translate to money. These days are over.
| yunohn wrote:
| Yes, that's why they want to fund and purchase cheap
| nuclear energy.
| portaouflop wrote:
| Microsoft also committed publicly to prioritise security.
| And Google says they prioritise privacy of their users
| above all else.
|
| I pity the fool that believes anything these corporations
| put out publicly.
|
| Actions matter, words are wind
| yunohn wrote:
| I almost feel there's a big difference between the kinds
| of things we're talking about, but sure.
| oblio wrote:
| > Google and Microsoft have publicly committed to those
| projects you're mentioning
|
| Google and Microsoft, or their current CEOs, today?
|
| Amazon's CEO committed to their office employees having
| flexibility regarding their workplace, only about 2 years
| ago, yet here we are now, with said employees soon having
| the flexibility to be 5 days in the office, or quit the
| company.
|
| CEO promises are not worth the screen time they're
| provided.
|
| Have these companies signed contracts with major penalties
| if they back out? Those would basically be the only "close
| to" unbreakable bonds for them.
| result2vino wrote:
| I feel like taking Google's commitment to something
| seriously is one of this things that I can very
| uncontroversially respond to with "is this your first day?"
|
| All but the biggest Google fanboys know that Google is
| incredibly indecisive and will cut plans at a moment's
| notice.
| yunohn wrote:
| I come to HN for better discussions without such
| infantile retorts.
| lopis wrote:
| Because it's all marketing and greenwashing. They are
| training these models _today_ using fossil fuels. By the
| time those nuclear reactors are online, they will have
| gobbled up literally every human creation to train their
| models multiple times and dried up several water sources.
| _aavaa_ wrote:
| My tone is because this is a simple predatory delay
| strategy.
|
| _Tomorrow, tomorrow, I'll decarbonize tomorrow._
|
| Instead of paying to buy wind and solar plants, which can
| go up _today_ they are signing a meaningless agreement for
| _the future_.
|
| A PPA isn't worth the paper it's written on if the seller
| can't produce electricity at the agreed upon price by the
| date required.
|
| Take Three Mile Island. It was closed in 2019 since it was
| uneconomical to run. Since then renewables have continued
| getting substantially cheaper, while the reactor has been
| in the process of decommissioning.
|
| Instead of spending money on building wind and solar,
| Microsoft saw how well Vogtle went and decided that another
| first of it's kind nuclear project is the best way to make
| it appear like they're doing something.
| exe34 wrote:
| I've been told my entire life that it's too late for
| nuclear, we should have been building them 20 years ago.
|
| I think now's fine, even if it takes time. these
| companies already buy a ton of power from renewable
| sources, and it's good to diversify - nuclear is a good
| backup to have.
| ViewTrick1002 wrote:
| The west tried building nuclear power 20 years ago. If it
| had delivered we would be building more now.
|
| It did not deliver. It is time to leave nuclear power to
| the past just like we have done with the steam engine.
|
| It had its heyday but better cheaper technology replaced
| it.
| exe34 wrote:
| what does it mean that "the west tried" - was it a
| technical failure or was it that people didn't want it in
| their backyard? just because people hate something
| doesn't mean that they don't need it. children hate
| spinach.
| ViewTrick1002 wrote:
| There was talk of an ongoing nuclear renaissance in the
| early 2000s. [1]
|
| American companies and utilities announced 30 reactors.
| Britain announced ~14.
|
| We went ahead and started construction on 7 reactors in
| Vogtle, Virgil C. Summer, Flamanville, Olkiluoto and
| Hanhikivi to rekindle the industry. We didn't believe
| renewables would cut it.
|
| The end result of what we broke ground on is 3 cancelled
| reactors, 3 reactors which entered commercial operation
| in the 2020s and 1 still under construction.
|
| The rest are in different states of trouble with
| financing with only Hinkley Point C slowly moving
| forward.
|
| In the meantime renewables went from barely existing to
| dominating new capacity (TWh) in the energy sector.
|
| Today renewables make up 2/3rds of global investment in
| the energy sector.
|
| The failure of nuclear power is that it is horrifically
| expensive and the timelines are insane compared to the
| competition.
|
| Steam locomotives technically work, but are like nuclear
| power uncompetitive.
|
| Lately nuclear power has caught the imagination of
| conservative politicians as a method to delay the
| renewable disruption of the fossil industry and have an
| answer to climate change.
|
| When their plans, like in Australia, get presented they
| don't care the slightest about nuclear power and it is
| only a method to prolong the life of the coal and gas
| assets.
|
| [1]: https://en.wikipedia.org/wiki/Nuclear_renaissance_in
| _the_Uni...
| Eisenstein wrote:
| > American companies and utilities announced 30 reactors.
| Britain announced ~14.
|
| Lots of projects get announced, they aren't meant to be
| promises.
|
| > The end result of what we broke ground on is 3
| cancelled reactors, 3 reactors which entered commercial
| operation in the 2020s and 1 still under construction.
|
| So there are three operational reactors and another one
| almost ready. I'm surprised we got that after Fukushima.
|
| > Today renewables make up 2/3rds of global investment in
| the energy sector.
|
| So we should not invest in anything else?
|
| > Steam locomotives technically work, but are like
| nuclear power uncompetitive.
|
| This is a terrible analogy.
|
| > Lately nuclear power has caught the imagination of
| conservative politicians as a method to delay the
| renewable disruption of the fossil industry and have an
| answer to climate change.
|
| People who have been advocating for more nuclear power
| should stop because it is a conservative issue now?
| ViewTrick1002 wrote:
| Which would have moved forward towards completion if the
| economic calculus made sense.
|
| We should of course continue with basic research. But,
| without some incredible breakthrough nuclear power will
| only serve climate change deniers agenda in delaying the
| renewable buildout.
|
| This is what you sign up for when proposing investing in
| nuclear power in 2024:
|
| > The opposition last week released modelling of its
| "coal-to-nuclear" plan that would slow the rollout of
| renewable energy and batteries and instead rely on more
| fossil fuel generation until a nuclear industry could be
| developed, mostly after 2040.
|
| https://www.theguardian.com/australia-
| news/2024/dec/16/coali...
| exe34 wrote:
| in other words, it's too late to build nuclear, let's
| bury our heads in the sand and hope somehow we have
| enough renewable in 20 years and we're not still using
| the coal/gas.
| ViewTrick1002 wrote:
| The bury our heads in the sand part seems to be you
| projecting.
|
| The research disagrees with you. Whenever new built
| nuclear power is included in the analysis the results
| becomes prohibitively expensive.
|
| > Focusing on the case of Denmark, this article
| investigates a future fully sector-coupled energy system
| in a carbon-neutral society and compares the operation
| and costs of renewables and nuclear-based energy systems.
|
| > The study finds that investments in flexibility in the
| electricity supply are needed in both systems due to the
| constant production pattern of nuclear and the
| variability of renewable energy sources.
|
| > However, the scenario with high nuclear implementation
| is 1.2 billion EUR more expensive annually compared to a
| scenario only based on renewables, *with all systems
| completely balancing supply and demand across all energy
| sectors in every hour*.
|
| > For nuclear power to be cost competitive with
| renewables an investment cost of 1.55 MEUR/MW must be
| achieved, which is substantially below any cost
| projection for nuclear power.
|
| https://www.sciencedirect.com/science/article/pii/S030626
| 192...
|
| Or if you want a more southern latitude you have
| Australia here:
|
| https://www.csiro.au/-/media/Energy/GenCost/GenCost2024-2
| 5Co...
| Eisenstein wrote:
| It may cost more, but it is constant generation, and we
| should invest in as many carbon neutral alternatives as
| possible that are feasible. The fact that you have a
| political opposition to it because of conservative
| opportunists using it for their own agenda is irrelevant.
| ViewTrick1002 wrote:
| Which is not what any modern grid needs? We need cheap
| dispatchable power, not horrifically expensive inflexible
| power.
|
| Many grids around the world already spend loads of time
| with renewables filling 100% of the demand.
|
| https://www.power-technology.com/news/california-
| achieves-10...
|
| That is a down right hostile environment for nuclear
| power which relies on being able to output at 100% 24/7
| all year around to only be horrifically expensive.
|
| In the land of infinite resources and infinite time "all
| of the above" is a viable answer. In the real world we
| neither have infinite resources nor infinite time to fix
| climate change.
|
| Lets focus our limited resources on what works and
| instead spend the big bucks on decarbonizing truly hard
| areas like aviation, construction, shipping and
| agriculture.
| Eisenstein wrote:
| 'Plenty of places' is not all places and you want to
| completely count out a significant energy generating
| ability because you are annoyed that it doesn't agree
| with your politics. If it isn't feasible then they won't
| build it -- by going around and advocating against it you
| are doing the same thing that happened in the 70s and 80s
| -- removing a perfectly valid option for energy that we
| _need_ and will otherwise be fulfilled in any other way
| if not provided -- almost always with fossil fuels. If
| you can guarantee every place for all time will be fine
| with renewables, I 'd like to see it, otherwise, why not
| step back and let engineers and scientists evaluate
| instead of grandstanding against an option?
| ViewTrick1002 wrote:
| What places aren't covered by the spectrum with Denmark
| for higher latitudes and Australia for the near the
| equator?
|
| I'm advocating against wasting public money on nuclear
| power pretending it is a solution to climate change.
|
| Have at it with your own money.
|
| I already provided you with the scientists and engineers,
| but you seem to have completely disregarded them because
| they did not align with what you wanted.
|
| I can do it again:
|
| The research disagrees with you. Whenever new built
| nuclear power is included in the analysis the results
| becomes prohibitively expensive.
|
| > Focusing on the case of Denmark, this article
| investigates a future fully sector-coupled energy system
| in a carbon-neutral society and compares the operation
| and costs of renewables and nuclear-based energy systems.
|
| > The study finds that investments in flexibility in the
| electricity supply are needed in both systems due to the
| constant production pattern of nuclear and the
| variability of renewable energy sources.
|
| > However, the scenario with high nuclear implementation
| is 1.2 billion EUR more expensive annually compared to a
| scenario only based on renewables, *with all systems
| completely balancing supply and demand across all energy
| sectors in every hour*.
|
| > For nuclear power to be cost competitive with
| renewables an investment cost of 1.55 MEUR/MW must be
| achieved, which is substantially below any cost
| projection for nuclear power.
|
| https://www.sciencedirect.com/science/article/pii/S030626
| 192...
|
| Or if you want a more southern latitude you have
| Australia here:
|
| https://www.csiro.au/-/media/Energy/GenCost/GenCost2024-2
| 5Co...
| signatoremo wrote:
| The logic is pretty straightforward I'm not sure what
| your complaint is. They don't need the power now, but
| they calculate that they'd need much more power in the
| future than non nuclear ways of power generation would be
| able to give them in the same timeframe.
|
| The US is already adding record amount of solar and wind
| power to replace coal and natural gas plants. What makes
| you believe Microsoft can just buy more renewable
| energies? Did you privy to the terms and conditions of
| the PPA?
|
| The alternative to Three Miles Island restart would be to
| add natural gas plants, or to buy renewable energy at
| higher price. I'm sure they have plan B.
| yunohn wrote:
| > Instead of paying to buy wind and solar plants
|
| Have you considered googling and checking your
| assumptions? May help clear up the cynical
| misunderstandings you appear to have.
|
| If you had, you would've read that both Microsoft and
| Google invest heavily into wind and solar, and that
| Google is the largest corporate purchaser of renewables
| in the world. I'm not advocating for these companies,
| just trying to show that tech is one of the few
| industries that does actually care and invest into clean
| energy.
|
| Some sources:
|
| - https://www.gstatic.com/gumdrop/sustainability/google-2
| 024-e...
|
| - https://www.latitudemedia.com/news/googles-largest-
| wind-inve...
|
| - https://amp.theguardian.com/technology/2019/sep/20/goog
| le-sa...
|
| - https://www.theverge.com/2024/5/2/24147153/microsoft-
| ai-data...
| _aavaa_ wrote:
| > Have you considered googling and checking your
| assumptions? May help clear up the cynical
| misunderstandings you appear to have.
|
| I don't have any such misunderstanding. Perhaps consider
| seeing my original comment which links to an article
| describing Google building out solar and wind farms for
| its data centres.
|
| My cynicism, which I argue is well founded, is based
| around tech companies signing such agreements with
| nuclear companies, especially when it involves doings
| things that have never been done before (restarting
| reactors and building economical SMRs, see Nuscale...).
|
| All these agreements are likely to amount to nothing more
| than positive PR, greenwashing, or predatory delay. Yes,
| they also build out solar and wind, but their nuclear
| PPAs are given equal standing with projects which
| actually are likely to be built; so instead of having to
| build more solar and wind today for more real money, they
| can promise to buy nuclear tomorrow for no cost today.
| ranyume wrote:
| > Nobody is building nuclear power plants for data centres
|
| Argentina just announced they're building nuclear plants for
| AI.
| sbierwagen wrote:
| Congrats, you have independently reinvented the Hardware
| Overhang hypothesis: that early AGI could be very inefficient,
| undergo several optimization passes, and go from needing a
| datacenter of compute to, say, a single video game console's
| worth:
| https://www.lesswrong.com/posts/75dnjiD8kv2khe9eQ/measuring-...
|
| In that scenario, you can go from 0 independent artificial
| intelligences to tens of millions of them, very quickly.
| UltraSane wrote:
| it would seem perfectly reasonable to expect the first AIs to
| be very unoptimized and if the AIs are any good they will be
| able to optimize themselves a lot and even help design ASICs
| to help run them.
| philipswood wrote:
| Thanks for sharing. Worth its own submission:
| https://news.ycombinator.com/newest
| ilaksh wrote:
| I think they can easily eat up the new capacity with larger
| multimodal models that ground language on video.
| mvkel wrote:
| You have it flipped. But it's both.
|
| AI compute is measured in gigawatts, not gigaflops.
|
| It's "how any gigawatts of compute can we get allocated?"
|
| Not
|
| "How much compute can we fit inside of a gigawatt?"
|
| There's no such thing as "enough"
| owenpalmer wrote:
| No. This is a classic case of Jevon's paradox. Increased
| efficiency in resource use can lead to increased consumption of
| that resource, rather than decreased consumption.
|
| Example:
|
| 1. To decrease total gas consumption, more fuel efficient
| vehicles are invented.
|
| 2. Instead of using less gas, people drive _more miles_. They
| take longer road trips, commute farther for work, and more
| people can now afford to drive.
|
| 3. This increased driving leads to higher overall gasoline
| consumption, despite each car using gas more efficiently.
| baq wrote:
| See also https://en.m.wikipedia.org/wiki/Induced_demand
| bonoboTP wrote:
| How do you argue that demand was induced as opposed to
| existing demand served?
| oblio wrote:
| Ok, not maybe instead of calling it "induced", call it
| "latent demand" if you prefer.
|
| People will do whatever's more convenient, so if you make
| driving far more convenient than everything else
| ("cheaper"/"more available"), they will drive.
|
| However convenience should not be the only factor for
| social decisions. To take this to extremes, it would be
| much more convenient for J. Doe to steal a car than to
| buy it, so we definitely do not want to make theft
| convenient.
| michaelhoney wrote:
| Yes, the term is a bit clumsy. The way I think of it,
| people have desires (to drive on the highway), but are
| dissuaded from doing so by disincentives (it's too busy).
| Adding a lane reduces the disincentive, so that latent
| desire is satisfied, until it reaches a new equilibrium.
| HPsquared wrote:
| Result: more people getting where they want to go.
| treyd wrote:
| But in the case of highways, they probably would have
| still gotten where they want to go by another route. The
| folly is treating highway capacity as being a "market"
| when really the decision making is much more dynamic and
| nuanced.
| HPsquared wrote:
| Like a market it's very complicated with many feedbacks
| and value judgements. For example "How much of my time is
| it worth sitting in traffic to get to my preferred store
| across town vs the closer one?"
|
| It's a bit like queueing. The cost isn't monetary.
| bonoboTP wrote:
| But sometimes the social environment adapts and now you
| have to drive that amount because it got factored in and
| things are now built further away, so whether you want to
| go far is not up to you. See long commutes becoming the
| norm. Now, arguably long commutes lead to better job
| allocations and more efficient land use as people can
| live in one place and work in any of the workplaces
| within a large radius. So it's a bit more complicated
| than "want", but ultimately more value seems to be
| produced.
| kaibee wrote:
| > ultimately more value seems to be produced.
|
| Is more value produced or are costs just shifted off the
| balance sheet onto the public commons? Driving instead of
| walking/public transit has certainly been profitable for
| some people/companies. But it has also been less than
| ideal from a public health standpoint. And the time spent
| commuting is unpaid, so while the business saves money on
| rent, the increase in travel time is still a cost borne
| by society as a whole. I would describe this as the
| opposite of 'efficient land use' personally.
| fauigerzigerk wrote:
| Because economists only call it demand to the extent that
| people are willing and able to make a purchase.
|
| If someone has a need or wants something real bad but
| can't afford to buy the desired quantity at the
| prevailing price then economists don't call it demand.
| Joker_vD wrote:
| Maybe because of an extremely aggressive marketing, and
| pushing "AI features" into literally everything, and
| there being a pushback against that?
| bonoboTP wrote:
| There's no paradox in that. People became more capable and
| can afford to do more.
| mewpmewp2 wrote:
| Also I think this will play out for AI as a productivity
| multiplier. Instead of people having less work there will
| be more to do since more things are worth doing now. For
| the following few years at least.
| HPsquared wrote:
| Work expands to fill the available time, after all.
| oreilles wrote:
| It is a paradox because there is an apparent contradiction
| in the fact that higher efficiency leads to higher
| consumption. By definition the opposite should be true.
| bonoboTP wrote:
| I don't share that intuition. If I earn more money, I
| won't necessarily save more. I'll buy better food, better
| clothes, better everything and live a materially more
| prosperous life. My savings rate may even go down. Or up.
| It depends on the specifics. When a tech gets more
| efficient, it causes people to do more. To shape their
| surroundings and bend reality more to their will. If you
| can travel easier, you can realize your travel wishes
| better.
| crazygringo wrote:
| Really, once you understand any "paradox", it ceases to be
| a paradox at all.
|
| I always feel a bit silly referring to any "paradox" as
| such, when it's _not_ a paradox anymore. When it now makes
| perfect sense.
| banannaise wrote:
| [nitpick] it's the Jevons Paradox, named for William Stanley
| Jevons. No apostrophe, but if you were to add one, it would
| be Jevons' Paradox.
|
| https://en.wikipedia.org/wiki/William_Stanley_Jevons
| m463 wrote:
| Don't you think people will just add better models to meet
| available memory?
|
| If we run 7B now, why wouldn't we run 700b with memory
| optimizations?
| lm28469 wrote:
| It's called the rebound effect, at no point in modern history
| efficiency reduced our energy needs, we just use the extra
| energy to either run more of the same thing or run other things
| ndr wrote:
| Wirth's law: software is getting slower more rapidly than
| hardware is becoming faster.
|
| I think there's the energy parallel: Software is becoming more
| energy-hungry faster than algorithms are becoming efficient.
|
| So we'll still need the energy.
| thom wrote:
| What if it's a big hoax and we create a better world for
| nothing?
| huijzer wrote:
| I guess power demands will slowly grow. The same happened with
| compute in general. Compared to 1960, we have several orders of
| magnitude more compute but also several orders of magnitude
| more efficient compute. Data centers are currently about 0.4%
| of total energy use (electricity is about 20% of total energy
| use and of the electricity about 2% goes to data centers, so
| 20% * 2% = 0.4%).
| amelius wrote:
| No idea, but it may also turn out that OpenAI has no moat,
| which is more interesting.
| hoseja wrote:
| Then you can simply have more AI in the data centres.
| hagbard_c wrote:
| That'd give a lot of extra power which can be used for other -
| and probably better - purposes so I'd say let them build those
| plants. The more power available the better after all?
| narrator wrote:
| Jevons Paradox will take care of it[1]. The more efficiently a
| resource is used, the more demand there is for it.
|
| The grave implication of Jevons paradox is that the fundamental
| conflict between sustainability and economic progress is not
| resolved solely by using resources more efficiently. It's a
| theory of supply chain constraints essentially. Once a resource
| is used more efficiently, its use is increased until the next
| most economically constrained resource hits its economically
| useful limit.
|
| [1] https://en.wikipedia.org/wiki/Jevons_paradox
| the8472 wrote:
| We don't even know a tighter lower bound for matrix multiplies
| than O(n2). Naive is O(n3), strassen is O(n^2.8). And those are
| simple, low-level kernels. At the higher level we also do not
| know tight lower bounds. But we do know some loose bounds from
| nature, e.g. how much data and energy a human consumes over its
| lifetime.
| ironfootnz wrote:
| I'm a big fan of their papers, this one didn't disappoint
| gcanyon wrote:
| Given that the algorithms powering present LLM models hadn't been
| invented ten years ago, I have to think that they are
| (potentially) _far_ from optimal.
|
| Brains have gone through millions of iterations where being
| efficient was a huge driver of success. We should not be
| surprised if someone finds a new ML method that is both wildly
| more efficient and wildly more effective.
| mycall wrote:
| Perhaps LLM++ will start iterating the algorithms via synthetic
| data until they are far more optimal
| solarkraft wrote:
| It's mind bogglingly crazy that language models rivaling ones
| that used to require huge GPUs with a ton of VRAM to run now run
| on my upper-mid-range laptop from 4 years ago. At usable speed.
| Crazy.
|
| I didn't expect capable language models to be practical/possible
| to run loyally, much less on hardware I already have.
| baq wrote:
| You have a sota multi-modal LLM running in your head at 20W,
| shared with best in class sensor package and top performing
| robotics control unit.
|
| There's soooo much more to optimize.
| fecal_henge wrote:
| But can it know love?
| baq wrote:
| Word on the street is researchers looked at the weights and
| weights looked back. You'll have to ask the weights.
| myrmidon wrote:
| I would argue that our sensor package is losing its lead very
| quickly-- audio performance is already on par with current
| tech and image processing is closing the gap very quickly as
| well (it helps a lot that silicon-based technology is much
| less constrained on bandwidth). Tactile sensing is still
| lightyears ahead, and I don't see that situation improving
| anytime soon...
| IWeldMelons wrote:
| chemical sensing is still quite good though.
| prox wrote:
| I might have a go at installing one, what is a good source or
| install at the moment?
| wint3rmute wrote:
| Ollama was the easiest way to set up local LLMs for me.
|
| https://ollama.com/
| solarkraft wrote:
| With llama3.2:1b, llama3.2:3b and llama 3.1:8b being the
| main ones I tried and found impressive.
| spacemanspiff01 wrote:
| lmstudio is very easy if you are running on local desktop.
| Alifatisk wrote:
| msty.app is good
| Eisenstein wrote:
| If you don't care about docker packages being used as
| installers and your home directory invisibly used to store
| massive weight files in exchange for not having to deal with
| learning any configuration: ollama or lmstudio.
|
| If you just want to play for a bit: llamafile
|
| If you want granular control with ease of execution in
| exchange for having to figure out what the settings mean and
| figure out which weights to download: koboldcpp. (check out
| bartowski on huggingface for the weights)
|
| These are all based on llamacpp as a backend, by the way.
| mandmandam wrote:
| > I didn't expect capable language models to be
| practical/possible to run _loyally_
|
| Now there's a fun typo. Hopefully not too much fun.
| momojo wrote:
| That sentence triggered memories of reading the huge
| paperback Asimov compilation my mom kept on the bookshelf as
| a kid.
| bamboozled wrote:
| Really exciting news.
| Euphorbium wrote:
| Stop words in extra steps.
| aussieguy1234 wrote:
| Most people don't remember absolutely everything, just the
| important stuff.
| hoc wrote:
| And we finally can sell the unoptimized models as Hires (since I
| can still read the differences!).
| cs702 wrote:
| Very clever, very meta, and it seems to work really well.
|
| The two big take-aways for me are:
|
| * It's possible to train a model to learn to summarize context
| from the _attention matrix_ , based only on dot-product scores (k
| @ q.T * mask), _regardless of how tokens are embedded_.
|
| * Once the model is trained, it will work with _any_ attention
| matrix, even if it 's the attention matrix of another model.
|
| I've added this to my ever-growing list of things to try.
| xpl wrote:
| Is there any intuition why does it even work? It seems very
| unexpected.
| skellington wrote:
| This only reduces the working memory, not the base model itself?
___________________________________________________________________
(page generated 2024-12-17 23:01 UTC)