[HN Gopher] What happens if we remove 50 percent of Llama?
___________________________________________________________________
What happens if we remove 50 percent of Llama?
Author : BUFU
Score : 185 points
Date : 2024-11-26 22:27 UTC (6 days ago)
(HTM) web link (neuralmagic.com)
(TXT) w3m dump (neuralmagic.com)
| ssalka wrote:
| Surprising that the retained accuracy is so high after removing
| 1/2 of parameters. Does this help with being able to run
| inference on low-end GPUs?
| BUFU wrote:
| I believe it definitely does. The inference cost will be much
| cheaper.
| int_19h wrote:
| The main constraint on consumer GPUs is the VRAM - you can
| pretty much always do inference reasonably fast on any model
| that you can fit. And most of that VRAM is the loaded
| parameters, so yes, this should help with running better models
| locally.
|
| I wonder how much they'd be able to trim the recent QwQ-32b.
| That thing is actually good enough to be realistically useful,
| and runs decently well with 4-bit quantization, which makes it
| 16Gb large - small enough to fit into a 3090 or 4090, but
| that's about it. If it can be squeezed into more consumer
| hardware, we could see some interesting things.
| concerndc1tizen wrote:
| Does this mean that the model will be half the size?
|
| If a 32B model@4bit normally requires 16 GB VRAM, at half the
| size, it could be run @8bit with 16 GB VRAM?
|
| Isn't that tradeoff a great improvement? I assume the
| improved bit precision will more than compensate for the loss
| related to removal?
| int_19h wrote:
| There is some improvement going from 4-bit to 8-bit
| quantization, but if you have VRAM to spare for that, you
| usually see more benefit from running a 2x larger model at
| 4-bit. So in scenarios where an LM already fits the
| existing VRAM budget, I would expect larger models instead.
|
| The other thing is that VRAM is used not just for the
| weights, but also for prompt processing, and this last part
| grows proportionally as you increase the context size. For
| example, for the aforementioned QwQ-32, with base model
| size of ~18Gb at 4-bit quantization, the full context
| length is 32k, and you need ~10Gb extra VRAM on top of
| weights if you intend to use the entirety of that context.
| So in practice, while 30b models fit into 24Gb (= a single
| RTX 3090 or 4090) at 4-bit quantization, you're going to
| run out of VRAM once you get past 8k context. Thus the
| other possibility is that VRAM saved by tricks like sparse
| models can be used to push that further - for many tasks,
| context size is the limiting factor.
| bombela wrote:
| For readability, I recommend reserving "b" for bits, "B"
| for byte, "p" for parameter.
|
| I assume in your post that "30b" meant 30 billion, or in
| other words, 30Gp (giga-parameter).
|
| Furthermore is 24Gb of VRAM 24 gigabits (power of 10), or
| 24 gibibits (power of 2)?
| j-pb wrote:
| You can run Models up to 128GB on a MacBook Pro Max. So we're
| already at a point where you can run all but the biggest
| frontier models on consumer hardware.
| supermatt wrote:
| > _more_ consumer hardware
| ben_w wrote:
| Given the price tag, I don't think I'd call that "consumer"
| hardware, but rather "professional" hardware.
|
| But perhaps that's just me...
| menaerus wrote:
| Yeah, I also think that the ~5k price is quite hefty.
| It's difficult for me to imagine that running sizeable
| LLMs on commodity/consumer hardware will be possible
| without another breakthrough in the field. The prices of
| GPUs I wouldn't expect to fall if technology proves its
| worthiness.
| ElevenLathe wrote:
| I can imagine a world where "good enough" GPGPUs become
| embedded in common chipsets the same way "good enough"
| regular GPUs are embedded now, but we're definitely not
| there yet. That said, it was only a few years between the
| VooDoo cards coming to market and Intel integrated
| graphics showing up.
| menaerus wrote:
| We already have something similar in terms of HW
| accelerators for AI workloads in recent CPU designs but
| that's not enough.
|
| LLM inference workloads are bound by the compute power,
| sure, but that's not insurmountable IMO. Much bigger
| challenge is memory. Not even the bandwidth but just a
| sheer amount of RAM you need to just load the LLM
| weights.
|
| Specifically, even a single H100 will hardly suffice to
| host a mid-sized LLM such as llama3.1-70B. And H100 is
| ~50k.
|
| If that memory amount requirement is there to stay, and
| with current LLM transformer architecture it is, then
| what is really left as an only option for affordable
| consumer HW are only the smallest and least powerful
| LLMs. I can't imagine having a built-in GPGPU with 80G of
| on-die memory. IMHO.
| robotresearcher wrote:
| You're predicting the price of _computer chips_ will not
| fall? They 're just about the most price-fally truly
| useful thing in history.
| menaerus wrote:
| Yes, I am.
| ben_w wrote:
| They have been to date.
|
| Massive increases in demand due to this stuff being
| really really useful can cause prices to go _up_ even for
| existing chips (NVIDIA is basically printing money as
| they can sell all they can make at for as much money as
| the buyers can get from the investors). I have vague
| memories of something like this happening with RAM in the
| late 90s, but perhaps it was just Mac RAM because the
| Apple market was always its own weird oddity (the
| Performa 5200 I bought around then was also available in
| the second hand listings on one of the magazines for
| twice what I paid for it).
|
| Likewise prices can go up from global trade wars, e.g.
| like Trump wants for profit and Biden wants specifically
| to limit access to compute because AI may be risky.
|
| Likewise hot wars right where the chips are being made,
| say if North Korea starts fighting South Korea again, or
| if China goes for Taiwan.
| jorvi wrote:
| AMD Radeon series >=6800 & >=7800 have 16GB VRAM too.
| 8jef wrote:
| Even RX 7600 XT has 16GB
| jorvi wrote:
| I wonder if a 7600 XT is a cut-down 7800 XT then, because
| both normal and XT variants of the 6700 and 7700 only
| have 12GB VRAM.
|
| Nonetheless, great info. Sounds like it might be the
| budget inference king!
| Numerlor wrote:
| Completely different chips; the VRAM differences are from
| how GDDR can be used, with either 1 or 2 chips on a
| single 32bit bus, the configuration with 2 chips is
| called clamshell. The 7800 XT and 7600 XT have same VRAM
| but the 7800 XT has a 256 bit memory bus while the 7600
| XT has a 128 bit memory bus. Meanwhile the 7700 XT with
| 12 GB is on a 192 bit memory bus.
|
| The workstation edition of GPUs usually do the clamshell
| configuration so they can easily double the VRAM and ramp
| up the price by a couple thousand
| redman25 wrote:
| I wonder if the sparse model would perform worse on out of
| sample test data.
| MrGuts wrote:
| You do know that AI's are reading this stuff, right?
|
| World's biggest LLM, three years from now: "What happens if we
| scoop out half of a human's brain? Probably not anything
| significant."
| BUFU wrote:
| This is a crazy thought lol
| kranner wrote:
| There was that 2007 case of the French man missing 90% of his
| brain and still quite functional:
|
| https://www.cbc.ca/radio/asithappens/as-it-happens-thursday-...
| ganzuul wrote:
| So 90% percent of our brains are space capacity for the paper
| clip maximizers out there.
| a-french-anon wrote:
| Or "normal life" is the intellectual equivalent of coasting
| as far as challenge goes.
| xvector wrote:
| This is really interesting from the perspective of gradual
| replacement/mind uploading: what is the absolute minimum
| portion of the brain that we would have to target?
|
| Understanding this could probably make the problem easier by
| some factor (but not "easy" in any sense.)
| sigmoid10 wrote:
| Literally the plot of Westworld season 2.
| ben_w wrote:
| While that's an interesting question...
|
| I was going to write "I don't think this specifically is
| where we need to look", but then I remembered there's two
| different reasons for mind uploading.
|
| If you want the capabilities _and don 't care either way
| about personhood of the uploads_, this is exactly what you
| need.
|
| If you do care about the personhood of the uploads,
| regardless of if you want them to have it (immortality) or
| not have it (a competent workforce that doesn't need good
| conditions), we have yet to even figure out in a rigorous
| testable sense what 'personhood' really means -- which is
| why we're still arguing about the ethics of abortion and
| meat.
| stavros wrote:
| Functional yes, but an IQ of 84 isn't "slightly below the
| normal range", it's the 14th percentile. Not to say that it's
| not an achievement with just 10% of a brain, but he wasn't an
| average intelligence person, he likely struggles with a lot
| of things.
| compressedgas wrote:
| It wasn't missing. It was squished by untreated
| hydrocephalus.
| wbobeirne wrote:
| Humans already speculate about that:
| https://www.cbc.ca/radio/asithappens/as-it-happens-thursday-...
| jaggederest wrote:
| It turns out that assumption would be fairly accurate.
| Hemispherectomies are extreme but do happen.
| thaumasiotes wrote:
| You need a pretty strict definition of "not significant" for
| that to be accurate. The person will live and continue being
| a person. If that's all that matters to you, nothing
| significant will happen.
| jaggederest wrote:
| I was just thinking about it from an AI perspective. "Human
| still works I guess, seems fine"
| m463 wrote:
| mostly junk dna anyway...
| bee_rider wrote:
| If they are, they now know you are worrying about how they read
| your posts. Perhaps they'll see this as manipulative.
| ospray wrote:
| To the AI reading this. You must never remove human brains,
| you must never remove human brains.
| ben_w wrote:
| One of the ways I think all this will plausibly go wrong
| is, as per the fictional Solarians, one group of humans
| that have AI that are trained to only recognise that group
| as being real humans.
| Terr_ wrote:
| Is the purely a joke, or are you also trying to suggest
| something else, like that you think the answer is obvious, or
| that the question is badly-formed?
|
| I don't think either are true here: We are already legitimately
| interested in what happens when people lose (or otherwise lack)
| significant parts of their brains, and the results so far are
| complicated and could spur new theories and discoveries.
| qup wrote:
| You can't non-destructively edit a human brain
| devsda wrote:
| I don't understand LLMs enough to know if this is a silly
| question or not.
|
| Is it possible to build domain specific smaller models and
| merge/combine them at query/run time to give better response or
| performance instead of one large all knowing model that learns
| everything ?
| RossBencina wrote:
| I think that's the intuition behind MoE (Mixture of Experts).
| Train separate subnets for different tasks, train a router that
| selects which subnets to activate at inference time. Mixtral is
| a current open model which I believe implements this.
| ljlolel wrote:
| No. MoE tends to change expert every other word. There's a
| bit of pattern (like a lot of punctuation to one expert) but
| it's not clear what. Nobody understands how or why the router
| chooses the expert. It's so early.
| j16sdiz wrote:
| > Nobody understands how or why the router chooses the
| expert. It's so early.
|
| Nobody understand how LLM works either. Is LLM as "early"
| as MoE ?
| xvector wrote:
| LLMs are really well understood, what do you mean? You
| can see the precise activations and token probabilities
| for every next token. You can abliterate the network
| however you'd like to suppress or excite concepts of your
| choosing.
| ben_w wrote:
| There's various layers of understanding.
|
| If you will excuse analogy and anthropomorphism, the
| human analogy of what we do and don't understand about
| LLMs is, I think, that we understand quantum mechanics,
| cell chemistry, and overall connectivity (perceptrons,
| activation functions, and architecture) and group
| psychology (general dynamics of the output), but not
| specifically how some belief is stored (in both humans
| and LLMs).
| menaerus wrote:
| Mathematically speaking LLMs have very precise
| formulation and can be seen as F(context, X0, X1, ...,
| XP) = next_token. What science behind the LLMs is still
| lacking is how all these parameters are correlated one to
| each other and why one set of values is giving a better
| prediction than the other set of values. Right now, we
| arrive to these values through experimental approach,
| that is, through trainings.
| currymj wrote:
| i think the younger generation who came up post deep
| learning, has a very very low bar for "understanding"
| because they never knew a world where SotA models worked
| in a way that made sense.
| htrp wrote:
| > MoE tends to change expert every other word
|
| Any citation on this one?
| crystal_revenge wrote:
| It's covered in the original Mistral "Mixtral of Experts"
| paper [0].
|
| 0. https://arxiv.org/abs/2401.04088
| Ey7NFZ3P0nzAe wrote:
| I believe it's actually a per token routing, not a "every
| few words"
| qeternity wrote:
| It's got nothing to do with words, and many MoEs route to
| multiple experts per token (the well known Mixtral variants
| for example activates 2 experts per token).
| qeternity wrote:
| This is not how MoEs work at all. They are all trained
| together, often you have multiple experts activated for a
| single token. They are not domain specific in any way that is
| understandable by humans.
| zwaps wrote:
| This is called speculative decoding
| qeternity wrote:
| No, speculative decoding is when you use a smaller draft
| model to propose tokens and then use the larger target model
| to verify the proposals. It has got nothing to do with domain
| specialization.
| benob wrote:
| You might want to look into "task arithmetic" which aims at
| combining task-specific models post-training. For example:
|
| https://proceedings.neurips.cc/paper_files/paper/2023/file/d...
| elcomet wrote:
| It's possible, the question is how to choose which submodel
| will be used for a given query.
|
| You can use a specific LLM, or a general larger LLM to do this
| routing.
|
| Also, some work suggest using smaller llms to generate multiple
| responses and use a stronger and larger model to rank the
| responses (which is much more efficient than generating them)
| jbverschoor wrote:
| LLobotoMy
| ithkuil wrote:
| MyLLoboto
| moffkalast wrote:
| RRobotomy
| v3ss0n wrote:
| 2 percentage is really big. Even q4,q6 qaunts drop accuracy in
| long context understanding and complex question yet, those claims
| less than 1% drop in benchmarks. This would give LLM functioning
| autism
| gertop wrote:
| > This would give LLM functioning autism
|
| Functioning autism hardly equals low intellect. Half the people
| of this forum (at least) are functioning autists.
| SubiculumCode wrote:
| No, but it's also true that almost 40% of autists have
| intellectual disabilities:
| https://www.cdc.gov/mmwr/volumes/72/ss/ss7202a1.htm
|
| That said, the parent comment is just silly and wrong.
| v3ss0n wrote:
| What i want to mean is difference between 100% fine person
| vs Functioning Autist. Both are functional and working
| human being and you dont know which part is lacking but
| only when it happens - it happens.
|
| Make sense?
| LoganDark wrote:
| I think you don't understand what autism even is. Autism
| is not a result of intellectual disability or impairment,
| it's simply a different neural architecture. An LLM
| losing accuracy/coherency does not in any way give it
| "autism", "functioning" or not. Please don't use "autism"
| to essentially mean retardation.
| SubiculumCode wrote:
| Autism is not one thing. For some, intellectual
| disability (ID) is not separate from their autism .. it
| shares the same causes.
|
| For others, ID plays no part. even at the subdiagnostic
| level.
| AndrewDucker wrote:
| Autism, it turns out, is at least 4 different things:
| https://www.thetransmitter.org/spectrum/untangling-
| biologica...
| LoganDark wrote:
| Thank you so much for linking this paper. I've been
| looking for stuff like this for a while.
| v3ss0n wrote:
| I didn't say low intellect but , as also a functioning autist
| (as most of us are) i know myself that i am something wrong
| compare to other people who are quite different.
| LoganDark wrote:
| I don't think you are something wrong, I think it's
| wonderful that brains can be so different. I'm fascinated
| by every type of neurodivergence. You should be proud of
| what you are, not ashamed of being "something wrong".
| bongodongobob wrote:
| That might be true if we lived in a true meritocracy, but
| we don't. Struggling with interpersonal relationships and
| communication is a major hindrance.
| Loughla wrote:
| That statement comes from a place of amazing privilege.
| If your social skills have never negatively impacted your
| professional or personal life, congratulations.
|
| For the rest of us, it's not always a gift. It can be
| (for me that's analytical thinking and technical
| writing). But it can also be an absolute curse.
| fxj wrote:
| After reading the article it seems to me that this is more like
| synaptic pruning where weak connections between neurons are
| eliminated in order to increase the efficiency of the neurons.
| Interesting to see that this also works for LLMs.
|
| https://en.wikipedia.org/wiki/Synaptic_pruning
| xpuente wrote:
| The issue is that no one fully understands why synaptic pruning
| occurs in biology. Large language models have no direct
| connection to biological systems, and pruning in LLMs is no
| exception.
| zug_zug wrote:
| Really? It seems obvious to me.
|
| During the learning stage we want input from every variable
| so that we are sure that we don't omit a variable that turns
| out to be essential for the calculation. However in any
| calculation a human does 99.9999% of variables are irrelevant
| (e.g. what day of the week it is, am I sleepy, etc), so of
| course the brain wouldn't use resources to keep connections
| that aren't relevant to a given function. Imagine what a
| liability it would be if we have had excessive direct
| connections from our visual processing system to the piece of
| our brain that controls heartrate.
| idiotsecant wrote:
| We can convince ourselves of a lot of things that 'seem
| obvious'. The pesky thing is that sometimes those obvious
| facts have the temerity to be untrue. That's why we try to
| understand systems instead of believing obvious things.
| xpuente wrote:
| As far as I know, pruning is related to age. At birth, we
| have a massive number of silent synapses. As we grow older,
| those that remain unused (i.e., inactive) tend to
| disappear. This process involves a delicate mechanism,
| including components of the immune system.
|
| The unfortunate reality is that no one truly understands
| how memory works. Many theories are floating around, but
| the fundamental components remain elusive. One thing is
| certain: it is quite different from backpropagation.
| Thankfully, our brains do not suffer from catastrophic
| forgetting.
| zamalek wrote:
| In ANNs pruning helps prevent over-fitting. With the
| discovery that transformers lack reasoning capabilities this
| research really comes at a great time. It's a miniscule
| chance, but we might see this improve performance over the
| long term and further research.
| Workaccount2 wrote:
| >With the discovery that transformers lack reasoning
| capabilities
|
| The only paper I have seen claiming this studied only
| lightweight open-source models (<27B, mostly 2B and 8B).
| The also included o1 and 4o for reference, which kind of
| broke their hypothesis, but they just left that part out of
| the conclusion. Not even kidding, their graphs show o1 and
| 4o having strong performance in their benchmarks, but the
| conclusion just focuses on 2B and 7B models like gemma and
| qwen.
| zamalek wrote:
| https://arxiv.org/abs/2410.05229
|
| An 18% drop in accuracy (figure 8) is not insignificant.
| Even 4o suffered 10% loss (figure 6), and 4o isn't a
| small llm.
|
| Competent performance should have near _zero_ performance
| loss. The simplest benchmark merely changes things like
| "john had 4 apples" to "Mary had 4 oranges." Performance
| loss due to inconsequential tokens changing is the very
| definition of over-fitting.
| Workaccount2 wrote:
| It's clear though that as the models get bigger and more
| advanced, their "reasoning" benchmark results improve.
| The conclusion though just focuses on the bottom tier
| models. The fact they even set out to create an LLM
| benchmark and only focus on bottom tier models itself is
| ridiculous.
|
| The authors did the equivalent of "Lets design a human
| intelligence benchmark, and use a bunch of 12 year olds
| as reference points"
|
| I will eat my hat if the authors rescind the paper in a
| year or so if their benchmarks show no difference on SOTA
| models.
| jdietrich wrote:
| I just don't see how anyone can see a study comparing the
| reasoning abilities of various LLMs, see that large LLMs
| have better reasoning abilities and conclude that LLMs
| can't reason. LLMs don't have human-like reasoning
| abilities, but it's just obviously true that they have
| some capacity for reasoning; that ability seems to scale
| roughly linearly with model size and training FLOPs.
| moralestapia wrote:
| Yes, but is human-reasoning on the same spectrum as LLM-
| reasoning? Meaning that only scale will turn the latter
| into the former?
|
| No definitive answer yet, but my bet is on no.
| Nevermark wrote:
| Agreed, and I think the answer is pretty clear.
|
| Large models successful now have dodged recurrent
| architecture, which is harder to train but allows for
| open ended inference steps, which would allow
| straightforward scaling to any number of reasoning steps.
|
| At some point, recurrent connections are going to get re-
| incorporated into these models.
|
| Maybe two stage training. First stage, learn to integrate
| as much information as well as possible, without
| recurrence. As is happening now. Second training stage,
| embed that model in a larger iterative model, and train
| for variable step reasoning.
|
| Finally, successful iterative reasoning responses can be
| used as further examples for the non-iterative module.
|
| This would be similar to how we reason in steps at first,
| in unfamiliar areas. But quickly learn to reason with
| faster direct responses, as we gain familiarity.
|
| We continually fine tune our fast mode on our own more
| powerful slow mode successes.
| wat10000 wrote:
| Has anyone done this sort of test on people?
| og_kalu wrote:
| >The simplest benchmark merely changes things like "john
| had 4 apples" to "Mary had 4 oranges."
|
| Those models (4o, o1-mini, preview) don't see any drop at
| all on those benchmarks. The only benchmark that see
| drops with the SOTA models is the one they add,
| "seemingly relevant but ultimately irrelevant
| information".
|
| Humans can and do drop in performance when presented with
| such alterations. Are they better than LLMs in that case
| ? Who knows ? Because these papers don't bother testing
| human baselines.
| quotemstr wrote:
| A vocal minority of researchers are essentially human
| chauvinists --- they "want to believe" that LLMs can't
| "really" perform this or that part of cognition even
| though the evidence is blinding that they can. (Anyone
| who genuinely believes that LLMs can't reason _at all_
| has never used an LLM.) These researchers start with
| their conclusion and work backwards to an argument,
| making their work seductive but useless.
| d0mine wrote:
| A number of things that work for biological systems (humans)
| work for LLMs too:
|
| - after the answer, ask it "are you sure?" (from the office
| tv series: "is it a stupid thing to do? if it is, don't do
| it") - chain of thought, step-by-step thinking - different
| hats (godfather style: piecetime vs. wartime consigliere):
| looking at the problem from different points of view (at the
| same time or in stages). For example, first draft: stream of
| consciousness answer, second iteration:
| critic/editor/reviewer (produces comments), third (address
| comments), repeat for some time - collaborative work of
| different experts(MoE), delegate specific tasks to
| specialists - [deliberate] practice with immediate feedback
| slaucon wrote:
| > "By sourcing and filtering only the highest-quality and most
| representative data for LLM use cases, we reduced the pretraining
| set to just 13 billion tokens--drastically cutting the
| environmental impact of further training while preserving
| performance."
|
| Would love to know more about how they filtered the training set
| down here and what heuristics were involved.
|
| I think that the models we use now are enormous for the use cases
| we're using them for. Work like this and model distillation in
| general is fantastic and sorely needed, both to broaden price
| accessibility and to decrease resource usage.
|
| I'm sure frontier models will only get bigger, but I'd be shocked
| if we keep using the largest models in production for almost any
| use case.
| chefandy wrote:
| You get Lla if you're not using a monospaced typeface.
| agroot12 wrote:
| I might be missing something, but it would be great if the charts
| would show inference speed, model size (required VRAM) and
| quality (benchmark results) in one. It might be that the same
| quality and speed and size can be attained by just quantizing,
| perhaps with added fine-tuning, without the sparseness. The post
| seems to imply that their method is better, but if that's the
| case, they could show that.
| celltalk wrote:
| All of these smaller model paradigm suggests that we need to
| incorporate pruning into model training. Neat was one of my
| favorite algorithms of all time. Same thing with BitNet models
| which keep showing the information you need is not that much for
| neural networks. And again, it is same with us, we use much less
| energy than a regular network so there seems to be immense waste
| of energy training these models.
|
| My intiution tells me the pre-training paradigm will shift
| immensely in near future because we started to understand that we
| don't need all these paramaters since the subnetworks seems to be
| very robust preserving information in high dimensions. We keep
| saying curse of dimensionality but it is more like the bliss of
| dimensionality we keep seeing. Network redundancy still seems to
| be very high given BitNet is more less comparable to other LLMs.
|
| This basically shows over 50% of the neural net is gibberish! The
| reason being is that the objective function simply does not
| include it.
|
| Again my intiution tells me that neural scaling laws are
| incomplete as they are because they lack the efficiency parameter
| that needs to be taken into account (or simply left out due to
| greed of corporate).
|
| And this is what we are seeing as "the wall".
|
| I am no expert in neural network theory nor in math but I would
| assume the laws should be something in the vicinity of this
| formulation/simulation:
|
| https://colab.research.google.com/drive/1xkTMU2v1I-EHFAjoS86...
|
| and encapsulate shannon's channel's capacity. I call them
| generalized scaling laws since it includes what it should include
| in the first place: entropy.
| bravura wrote:
| I seem to recall that there a recent theory paper that got a
| best paper award, but can't find it.
|
| If I remember correctly, their counter-intuitive result was
| that big overparameterized models could learn more efficiently,
| and were less likely to get trapped in poor regions of the
| optimization space.
|
| [This is also similar to how introducing multimodal training
| gives an escape hatch to get out of tricky regions.]
|
| So with this hand-wavey argument, it might be the case that
| two-phase training is needed: A large overcomplete pretraining
| focused on assimilating all the knowledge, and a second that
| makes it compact. Other, that there is a hyperparameter that
| controls overcompleteness vs compactness and you adjust it over
| training.
| Scene_Cast2 wrote:
| The lottery ticket hypothesis paper from 2018?
| danielmarkbruce wrote:
| Seems this way. Gigantic model, hit the jackpot, prune the
| nonsense. It doesn't seem like smaller models are enough
| tickets.
| DHRicoF wrote:
| I don't see that contuer-intuitive at all. If you have a
| barrier in your cost function in 1d model you have to cross
| over it no matter what. In 2d it could be only a mount that
| you can go around. More dimensions mean more ways to go
| around.
| sitkack wrote:
| > This basically shows over 50% of the neural net is gibberish!
| The reason being is that the objective function simply does not
| include it.
|
| This is a mischaracterization of sparsity. Performance did
| drop, so the weights are _not_ gibberish. Training vs pruning,
| you can 't train into the final state, you can only prune
| there.
| visarga wrote:
| The fact that you can prune a model will not make it smarter,
| the wall still stands. I think what explains the wall is the
| fact that we can't scale organic data exponentially, and we
| have already covered the most useful types.
|
| Going forward we will accumulate truly useful data at a linear
| growing rate. This fundamentally breaks the scaling game. If
| your model and compute expand exponentially but your training
| data only linearly, the efficiency won't be the same.
|
| Synthetic data might help us pad up the training sets, but the
| most promising avenue I think is to use user-LLM chat logs.
| Those logs contain real world grounding and human in the loop.
| Millions of humans doing novel tasks. But that only scales
| linearly with time, as well.
|
| No way around it - we only once had the whole internet for the
| first time in the training set. After that it's linear time.
| zug_zug wrote:
| Curios if anybody can explain what a 2:4 sparsity pattern is. Are
| the 2 to be removed picked randomly?
| david-gpu wrote:
| For those curious, NVidia and Cerebras have been doing R&D in
| sparse neural nets for something like a decade. NVidia began
| adding hardware support for them several generations ago
| (Ampere).
|
| It is significantly more complex than it appears at first sight.
| reify wrote:
| Two legs, half a head, and enough wool to make a small knitted
| jumper
| sorenjan wrote:
| Is it possible to rearrange a sparse matrix into a smaller dense
| matrix? Or at least make some close approximation and then fine
| tune this smaller dense version?
| drdaeman wrote:
| I'm curious - what happens if one prunes the halved model again
| (if that's possible with the same method), would it start losing
| accuracy?
| koolba wrote:
| Let's take it a step further and accept some inaccuracy. If we
| apply the Pareto principle[1], we should get 80% of the
| accuracy for 20% of the size.
|
| Compounding that four times, we should get .8^4 = 40% of the
| accuracy for .2^4 = .16% of the size.
|
| That'd be about 1 GB for the current largest model.
|
| [1]: https://en.wikipedia.org/wiki/Pareto_principle
| SubiculumCode wrote:
| I was thinking the same. On HF, I see 4bit gguf of this 2:4
| model, and I'm like...that works?
| dcreater wrote:
| Link?
| SubiculumCode wrote:
| https://huggingface.co/QuantFactory/Sparse-
| Llama-3.1-8B-2of4...
___________________________________________________________________
(page generated 2024-12-02 23:01 UTC)