[HN Gopher] Mamba-3
___________________________________________________________________
Mamba-3
Author : matt_d
Score : 268 points
Date : 2026-03-17 22:45 UTC (4 days ago)
(HTM) web link (www.together.ai)
(TXT) w3m dump (www.together.ai)
| robofanatic wrote:
| > Mamba-3 is a new state space model (SSM) designed with
| inference efficiency as the primary goal -- a departure from
| Mamba-2, which optimized for training speed. The key upgrades are
| a more expressive recurrence formula, complex-valued state
| tracking, and a MIMO (multi-input, multi-output) variant that
| boosts accuracy without slowing down decoding.
|
| Why can't they simply say -
|
| Mamba-3 focuses on being faster and more efficient when making
| predictions, rather than just being fast to train like Mamba-2.
| E-Reverance wrote:
| The first sentence basically does though, no?
| robofanatic wrote:
| Of course my only objection was the language. LLMs are now
| old enough to leave the jargon behind and talk in simple easy
| to understand terms.
| oersted wrote:
| I'd argue the opposite, the terminology is fairly
| mainstream by now and "inference" has a much more specific
| sense than "making predictions".
| esquire_900 wrote:
| This is sort of what their first sentence states? Except your
| line implies that they are fast in training and inference, they
| imply they are focusing on inference and are dropping training
| speed for it.
|
| It's a nice opening as it is imo
| cubefox wrote:
| They don't say anything about dropping training speed.
| estearum wrote:
| > a departure from Mamba-2, which optimized for training
| speed.
|
| ?
| cubefox wrote:
| Yes? Mamba-2 optimized for training speed compared to
| Mamba-1. Mamba-3 adds optimization for inference. These
| are pretty much version numbers.
| arendtio wrote:
| I don't get the downvotes, as I had trouble understanding the
| intro as well. It seems it was written for a very specific
| audience.
| qeternity wrote:
| Yes, it is written for a specific audience.
|
| That is not a reason for snark.
|
| As other commenters have noted, it's well written.
| magicalhippo wrote:
| > I don't get the downvotes
|
| Because the blog post is a technical one and the intro
| contains very common jargon, and the proposed alternative was
| wrong.
| mufasachan wrote:
| The blog is technical, technical terms in the TL;DR seems
| relevant to me.
| camillomiller wrote:
| I don't know why you're being downvoted. As a longtime editor
| your version is immensely better. Looks like the original was
| probably not human-written.
| stavros wrote:
| Why would the simpler version be better for a technical
| audience?
| renewiltord wrote:
| Found the guy who made the Windows error messages say "Your
| computer did an oopsie :(" instead of including any useful
| information.
| i000 wrote:
| Agreed. What you wrote was probably the input, what we see is
| the LLM output with the directive to "make us sound smart, put
| gratuitous em-dash"
| nl wrote:
| I'm looking forward to comparing this to Inception 2 (the text
| diffusion model) which in my experience is very fast and
| reasonably high quality.
| cubefox wrote:
| Mamba-3 is an architecture while diffusion is, I believe, a
| type of objective. So these are not mutually exclusive and
| therefore not comparable.
| gyrovagueGeist wrote:
| Not wrong, but I think it's more accurate to say:
|
| Mamba is an architecture for the middle layers of the network
| (the trunk) which assumes decoding takes place through an
| autoregressive sequence (popping out tokens in order). This
| is the SSM they talk about.
|
| Diffusion is an alternative to the autoregressive approach
| where decoding takes place through iterative refinement on a
| batch of tokens (instead of one at a time processing and
| locking each one in only looking forward). This can require
| different architectures for the trunk, the output heads, and
| modifications to the objective to make the whole thing
| trainable. Could mamba like ideas be useful in diffusion
| networks...maybe but it's a different problem setup.
| cubefox wrote:
| Linear architectures are at least heavily used in image
| diffusion models. More so in fact than in language models.
| joefourier wrote:
| Mamba doesn't assume auto-regressive decoding, and you can
| use absolutely use it for diffusion, or pretty much any
| other common objective. Same with a conventional
| transformer. For a discrete diffusion language model, the
| output head is essentially the same as an autoregressive
| one. But yes, the training/objective/inference setup is
| different.
| nl wrote:
| I mean I guess but the diffusion objective and the ability to
| do simultaneous decode both dictate pretty different
| architectures in practice.
| cubefox wrote:
| Apparently not. See https://arxiv.org/abs/2511.15927v3
| jychang wrote:
| That's completely different. That's like saying you want to
| compare the Nvidia 5090 GPU to the latest Call of Duty.
| cubefox wrote:
| You are right, people who downvoted you are just ignorant.
| PhilippGille wrote:
| You mean Mercury 2, by Inception:
| https://openrouter.ai/inception/mercury-2
| jychang wrote:
| I'm not sure that I buy their conclusion that more compute during
| inference is good.
|
| Yes, batch=1 inference is mostly memory bandwidth bound, not GPU
| compute bound. But no provider does batch=1 inference. Everyone
| groups all the requests into a batch, and the GPU computes them
| together.
|
| With a fused kernel, that means the GPU streams the tensors from
| VRAM, and does a bunch of compute on different conversations in
| the batch, at the same time.
|
| If they increase the amount of compute required per token, that
| just reduces the maximum batch size a GPU can handle. In
| practice, yes this does mean each GPU can serve less users.
| Providers aren't leaving GPU cores idle normally during
| inference.
| zozbot234 wrote:
| > Everyone groups all the requests into a batch, and the GPU
| computes them together.
|
| You're only saving on fetching read-only parameters, and not
| even on that if you're using MoE models where each inference in
| the batch might require a different expert (unless you
| rearrange batches so that sharing experts becomes more likely,
| but that's difficult since experts change per-token or even
| per-layer). Everything else - KV-cache, activations - gets
| multiplied by your batch size. You scale both compute and
| memory pressure by largely the same amount. Yes, GPUs are great
| at hiding memory fetch latency, but that applies also to n=1
| inference.
| jychang wrote:
| Well, the actual inference providers put each expert on its
| own single GPU. Deepseek explicitly does this.
|
| Read-only parameters is also usually the majority of space.
| Deepseek is 700GB of params. Meanwhile kv cache is small
| (Deepseek is about 7GB at max context) and ssm/conv1d cache
| is even smaller- IIRC Qwen 3.5 is 146MB per token regardless
| of context size. Not sure about how Mamba-3 works, but I
| suspect read-only parameters are still a significant amount
| of memory bandwidth.
|
| I guess the question isn't whether compute is 1:1 with
| memory, but rather if you run out of compute before you run
| out of vram adding more users.
| zozbot234 wrote:
| > Well, the actual inference providers put each expert on
| its own single GPU.
|
| Experts are usually chosen on a per-layer basis, not just
| by token, so I'd think this requires having lots of GPU's
| to make it worthwhile. You could do it with a single
| physical GPU by switching expert-layer mixes in a round-
| robin fashion after the batch for any single expert-layer
| mix is completed (essentially a refined version of expert
| offloading). But still, not easy.
| yorwba wrote:
| Their latency measurements comparing Mamba-2 and Mamba-3 are
| done with a batch size of 128. It doesn't seem like Mamba-2 was
| compute-bound even at that batch size.
| jychang wrote:
| Well, Deepseek batch sizes are something like 8192, so 128
| isn't much.
|
| https://arxiv.org/html/2412.19437v1 "the batch size per
| expert is relatively small (usually within 256 tokens)"
| btown wrote:
| Throughput is indeed king for the standard-tier mindshare-
| capture play. But there are many who would pay multiple times
| the current cost for agentic systems for engineers and
| executives, if it meant a meaningful reduction in latency. The
| economics could work extremely well.
| Havoc wrote:
| Why would execs need latency?
|
| I can see it for engineering - coding with slow ai is painful
| metanonsense wrote:
| Impatient execs can also be painful. EDIT: Writing this
| while I am waiting for Codex to complete, so I may enjoy
| slow AI more than the usual developer ;-)
| btown wrote:
| The economic effect of latency is measured not by the
| incremental productivity itself, but by the combined
| economic downforce of thousands of resulting HN and
| Reddit comments :)
| notnullorvoid wrote:
| Focusing on needs of providers isn't a very good long term
| strategy if you believe compute will eventually move to self
| hosted and on premises solutions where large batch sizes aren't
| needed.
| sroussey wrote:
| Local has a batch size of 1. If you are already memory bound
| then you leave compute on the table. Why not use it?
|
| Not sure they target local though...
| Havoc wrote:
| Is there a reason we don't switch halfway through? ie start with
| a classic LLM and switch to something linear like mamba as
| context grows
| energy123 wrote:
| Probably best achieved by model routing, either an indirection
| behind the chat UI or an API user does it themselves by calling
| a different API for long context queries.
| cubefox wrote:
| Linear time complexity models are bad at in-context retrieval,
| which limits their performance on various tasks, so a pure
| linear model isn't currently feasible anyway, at least for
| language models. Instead they recommend mixing linear and
| attention layers. Presumably this mostly solves the performance
| problem (at least n benchmarks), but it also means the mixed
| architecture is no longer linear. It will still be faster and
| less RAM hungry in long context than a pure transformer though.
| mountainriver wrote:
| We kinda do do this with hybrid mamba transformers
| lambda wrote:
| Because something linear like Mamba doesn't perform as well; so
| you'd have a performance cliff, where suddenly the model would
| get more dumb and forget a lot of what was going on.
|
| Instead, you can get benefits from both by doing both in
| parallel. This can let you reduce the size of the O(n^2)
| attention mechanism, so while it's still quadratic, it reduces
| the constant quite a bit while still retaining a lot of
| performance, as the linear context mechanism can work for the
| tasks its well suited for while allowing attention to play to
| its strengths.
|
| The recent Nemotron 3 Nano and Super models from NVIDIA are
| hybrid architectures this way, with most of their context
| layers as Mamba while retaining enough attention to continue to
| be competitive on the more complex tasks that require the
| quadratic attention.
|
| See
| https://magazine.sebastianraschka.com/i/168650848/18-nemotro...
| for some discussion on this architecture
| 0xbadcafebee wrote:
| They did do that, 2 years ago. The problems are that 1) mamba
| makes accuracy worse as context size grows, 2) Nvidia GPUs are
| designed for transformers, and 3) all the software out there is
| also designed for transformers. It's still useful in some
| applications but it doesn't beat regular transformers if you
| have the gear
| anentropic wrote:
| More here https://news.ycombinator.com/item?id=47423208
|
| https://arxiv.org/abs/2603.15569
| jeffhwang wrote:
| I'm glad I clicked through bc I thought the article was about
| Mamba, the package manager I associate with Python (similar to
| conda).
|
| https://github.com/mamba-org/mamba
| manlymuppet wrote:
| I'm looking forward to the fifth iteration of this model.
| roger_ wrote:
| Can anyone explain why Mamba models start with a continuous time
| SSM (and discretize) vs discrete time?
|
| I know the step isn't fixed, also not sure why that's important.
| Is that the only reason? There also seems to be a
| parameterization advantage too with the continuous formulation.
| fudged71 wrote:
| This is really promising. Are they now going to scale this up to
| hundreds of billions of parameters? Why stop at 1.5B if they
| found a potentially SOTA architecture?
| snek_case wrote:
| Probably constrained by training resources. It's much easier to
| experiment with a smaller architecture. You may need many
| training runs to figure out hyperparameters for example. If
| each run needs multiple GPUs for a week the cost adds up
| quickly. I think it makes a lot of sense to start small.
___________________________________________________________________
(page generated 2026-03-21 23:00 UTC)