[HN Gopher] Zebra-Llama: Towards Efficient Hybrid Models
___________________________________________________________________
Zebra-Llama: Towards Efficient Hybrid Models
Author : mirrir
Score : 44 points
Date : 2025-12-06 20:15 UTC (2 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| mason_mpls wrote:
| > Zebra-Llama achieves Transformer-level accuracy with near-SSM
| efficiency using only 7-11B training tokens (compared to
| trillions of tokens required for pre-training) and an 8B teacher.
| Moreover, Zebra-Llama dramatically reduces KV cache size--down to
| 3.9%, 2%, and 2.73% of the original for the 1B, 3B, and 8B
| variants, respectively--while preserving 100%, 100%, and 97% of
| average zero-shot performance on LM Harness tasks.
|
| This is an extraordinary claim, is there a catch I'm missing? Am
| I misreading?
| adityashankar wrote:
| Due to perverse incentives and the historical nature of models
| over-claiming accuracy, it's very hard to believe anything until
| it is open source and can be tested out
|
| that being said, I do very much believe that computational
| efficiency of models is going to go up [correction] drastically
| over the coming months, which does pose interesting questions
| over nvidia's throne
|
| *previously miswrote and said computational efficiency will go
| down
| danielbln wrote:
| I think you mean computational efficiency will go _up_ in the
| future. To your last point: Jevons paradox might apply.
| adityashankar wrote:
| yup that's what I meant!, Jevon's paradox applies to resource
| usage in general and not towards a specific companies
| dominance
|
| if computational efficiency goes up (thanks for the
| correction), and CPU inference becomes viable for most
| practical applications, GPUs (or accelerators) themselves may
| be unnecessary for most practical functions
| atq2119 wrote:
| Discrete GPUs still have an advantage in memory bandwidth.
| Though this might push platforms like laptops towards
| higher bandwidths, which would be nice.
| credit_guy wrote:
| Like this?
|
| https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT
| adityashankar wrote:
| yes!, thanks for the link!
| ACCount37 wrote:
| I don't doubt the increase in efficiency. I doubt the
| "drastically".
|
| We already see models become more and more capable per weight
| and per unit of compute. I don't expect a state-change
| breakthrough. I expect: more of the same. A SOTA 30B model from
| 2026 is going to be ~30% better than one from 2025.
|
| Now, expecting that to hurt Nvidia? Delusional.
|
| No one is going to stop and say "oh wow, we got more inference
| efficiency - now we're going to use less compute". A lot of
| people are going to say "now we can use larger and more
| powerful models for the same price" or "with cheaper inference
| for the same quality, we can afford to use more inference".
| colechristensen wrote:
| Eh.
|
| Right now, Claude is good enough. If LLM development hit a
| magical wall and never got any better, Claude is good enough
| to be terrifically useful and there's diminishing returns on
| how much good we get out of it being at $benchmark.
|
| Saying we're satisfied with that... well how many years until
| efficiency gains from one side and consumer hardware from the
| other meet in the middle so "good enough for everybody" open
| models are available for anyone who wants to pay for a $4000
| MacBook (and after another couple of years a $1000 MacBook,
| and several more and a fancy wristwatch).
|
| Point being, unless we get to a point where we start
| developing "models" that deserve civil rights and
| citizenship, the years are numbered to where we NEED cloud
| infrastructure and datacenters full of racks and racks of
| $x0,000 hardware.
|
| I strongly believe the top end of the S curve is nigh, and
| with it we're going to see these trillion dollar ambitions
| crumble. Everybody is going to want a big-ass GPU and a ton
| of RAM but that's going to quickly become boring because open
| models are going to exist that eat everybody's lunch and the
| trillion dollar companies trying to beat them with a premium
| product aren't going to stack up outside of niche cases and
| much more ordinary cloud compute motivations.
| Reubend wrote:
| It would be REALLY cool to see this same technique applied to a
| much more recent OSS model distillation. For example, Mistral 3
| 14B would be a great target. How efficient can we get inference
| there?
| a_wild_dandan wrote:
| If the claims in the abstract are true, then this is legitimately
| revolutionary. I don't believe it. There are probably some major
| constraints/caveats that keep these results from generalizing.
| I'll read through the paper carefully this time instead of a skim
| and come back with thoughts after I've digested it.
| xer wrote:
| This is great! But what if the US invests 1% of GDP in GPU
| datacenters and then those are not needed becaues someone created
| a much more efficient architecture?
| _boffin_ wrote:
| They will still use capacity. Why would you believe anything
| different?
| wild_egg wrote:
| More efficiency just means more consumption. Think when they
| add lanes to a highway, traffic gets better for a little bit
| but very soon the highway is just as congested as before.
___________________________________________________________________
(page generated 2025-12-06 23:00 UTC)