[HN Gopher] TransMLA: Multi-head latent attention is all you need
___________________________________________________________________
TransMLA: Multi-head latent attention is all you need
Author : ocean_moist
Score : 115 points
Date : 2025-05-13 03:29 UTC (19 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| olq_plo wrote:
| Very cool idea. Can't wait for converted models on HF.
| MichaelMoser123 wrote:
| deepseek-v2,v3,r1 are all using multi-headed attention.
| kavalg wrote:
| My (possibly wrong) TLDR: TransMLA is a method to "compress" an
| already trained GQA model, with the additional option to further
| fine tune it. Shall make inference faster.
| freeqaz wrote:
| Also makes models smarter ("expressive")
| yorwba wrote:
| It is not a method to _compress_ a Grouped-Query Attention
| model, but to _expand_ it into an equivalent Multi-head Latent
| Attention model with the same key-value cache size but larger
| effective key /value vectors and a correspondingly larger
| number of trainable parameters. With additional training, you
| can then obtain a better model that only uses a little bit more
| memory.
| wiz21c wrote:
| Not quite related, but do the mamba models gain ground ?
|
| Answering my own question:
| https://www.reddit.com/r/MachineLearning/comments/1hpg91o/d_...
| EGreg wrote:
| All you need to stop posting titles like that !
| jbellis wrote:
| [abstract] This approach significantly reduces the KV cache size
| relative to traditional multi-head attention
|
| [3.3] For saving the KV cache, only the intermediate latent
| representations need to be stored: [latex] where r is much
| smaller than nh * dh [n-sub-h, d-sub-h]
|
| [background] In traditional multi-head attention you must cache
| full key and value matrices of size T x (nh * dh) where T is the
| token length, nh is the number of attention heads, dh is the
| dimensionality of each individual head
|
| sounds like a big win for memory constrained environments like
| local inference
| magicalhippo wrote:
| I'm just following the field from the sidelines, but this looks
| interesting to me. Especially the increase in expressiveness that
| the new model allows for over GQA, at the cost of just ~10% more
| memory, and the fact that you can convert existing GQA models
| like LLaMA, Qwen etc with just a bit of fine-tuning.
|
| Perhaps a trivial insight but I feel a lot of progress often
| comes in the form of generalizations, where existing approaches
| can be seen as special cases. Here the authors show that Group
| Query Attention (GQA) and Multi-Query Attention (MQA) falls out
| as special cases of their new model.
|
| edit:
|
| Adding my own summary, as I understand it.
|
| The key to what they're doing, no pun intended, is to rely on the
| fact that large, high-dimensional, matrices may contain a lot of
| redundant information. Thus one may be able to find an good
| approximation which has less redundant information, by going
| through an intermediary stage which has fewer dimensions.
|
| A n-by-m matrix M takes n-dimensional vectors and transforms them
| to m-dimensional vectors. The trick here is to replace matrix A
| by two matrices, L and R, which are n-by-r and r-by-m
| respectively, where r is smaller than n and m. This is called a
| low-rank approximation.
|
| In a sense you're "straining the matrix", by forcing the
| information to pass through an intermediary, low-dimensional
| vector.
|
| The memory savings come from the fact that matrix A has n*m
| entries, while L and R have n*r and r*m entries respectively. Say
| n = m = 100 and r = 20, that means A has 100*100 = 10k entries,
| while L and R have just 100*20 + 20*100 = 4k entries in total.
|
| The trick itself is not new, for example it is also used in LoRA
| where an additional low-rank approximation matrix is used to
| tweak the output of an existing model. The low rank means there's
| far fewer the matrix entries, aka parameters, to train than if
| one had used a regular fully dense matrix.
|
| The extra expressiveness of MLA comes from the fact that in GQA,
| in order to save memory, some of the matrices are actually built
| by gluing copies of a narrower matrix together. This means the
| information in the glued-up matrices are very redundant and fixed
| in a certain way, and thus are restricted in how they can
| transform the inputs.
|
| By using the low-rank approximation instead, the information in
| the full, reconstructed matrices are not fixed in the same way
| compared to the glued-up result. Thus the inputs can be
| transformed in a less restrictive way, leading to the increase in
| expressiveness.
|
| The GQA method saves a bit more memory compared to MLA as the
| narrower matrices are even smaller than the low-rank matrices in
| MLA, but at the cost of expressiveness.
| killerstorm wrote:
| Another paper related to attention distillation, although doing
| something far more radical: transformer attention is distilled
| onto RWKV-like model: https://huggingface.co/papers/2505.03005
| karmakaze wrote:
| I'm not "in the field" though I like to read about and use LLMs.
| This video "How DeepSeek Rewrote the Transformer [MLA]"[0] is
| really good at explaining MHA, MQA, GQA, and MLA with clear
| visuals/animations and how DeepSeek MLA is 57x more efficient.
|
| [0] https://www.youtube.com/watch?v=0VLAoVGf_74&t=960s
___________________________________________________________________
(page generated 2025-05-13 23:02 UTC)