[HN Gopher] Reproducing DeepSeek's MHC: When Residual Connection...
___________________________________________________________________
Reproducing DeepSeek's MHC: When Residual Connections Explode
Author : taykolasinski
Score : 93 points
Date : 2026-01-12 13:57 UTC (9 hours ago)
(HTM) web link (taylorkolasinski.com)
(TXT) w3m dump (taylorkolasinski.com)
| taykolasinski wrote:
| OP here. I spent the last few days reproducing the mHC
| architecture from the recent DeepSeek paper (2512.24880).
|
| Two key takeaways from the reproduction:
|
| Unconstrained Hyper-Connections really do explode (7x
| amplification even at 10M scale).
|
| I hit a nasty "stream persistence" bug where my tensors were the
| right shape, but the architecture was functionally broken.
|
| This is Part 1 (10M scale). Part 2 (scaling to 1B on A100s) is
| coming later this week. Happy to answer questions about the
| implementation.
| WiSaGaN wrote:
| How do you know "GPT-5, Claude, Llama, Gemini. Under the hood,
| they all do the same thing: x+F(x)."?
| taykolasinski wrote:
| I'm referring specifically to the fundamental residual
| connection backbone that defines the transformer architecture
| (x_{l+1} = x_l + F(x_l)).
|
| While the sub-modules differ (MHA vs GQA, SwiGLU vs GeLU,
| Mixture-of-Depths, etc.), the core signal propagation in
| Llama, Gemini, and Claude relies on that additive residual
| stream.
|
| My point here is that DeepSeek's mHC challenges that
| fundamental additive assumption by introducing learnable
| weighted scaling factors to the residual path itself.
| WiSaGaN wrote:
| I guess I am asking how we know Gemini and Claude relies on
| the additive residual stream. We don't know the
| architecture details for these closed models?
| taykolasinski wrote:
| That's a fair point. We don't have the weights or code
| for the closed models, so we can't be 100% certain.
|
| However, transformer-based (which their technical reports
| confirm they are) implies the standard pre-norm/post-
| nnorm residual block structure. Without those additive
| residual connections, training networks of that depth
| (100+ layers) becomes difficult due to the vanishing
| gradient problem.
|
| If they had solved deep signal propagation without
| residual streams, that would likely be a bigger
| architectural breakthrough than the model itself (akin to
| Mamba/SSMs). It's a very high-confidence assumption, but
| you are right that it is still an assumption.
| solarkraft wrote:
| I've been wondering for a while: Why isn't this architecture more
| common in other LLMs? The context efficiency is amazing, after
| all - doesn't that translate to a lot of money at scale?
| kevmo314 wrote:
| It's an incremental improvement, not really a revolutionary
| step.
|
| That being said, I think one could adapt an existing model to
| add mHC by initializing the routing matrix to the regular
| residual connection and then post-train the hyper connection
| matrices. This would let you continue training more efficiently
| on existing models.
| taykolasinski wrote:
| That initialization strategy (effectively starting as
| identity to match the standard residual stream) is clever. It
| would let you surgery an existing model like Llama-3 and
| fine-tune it into an mHC architecture.
|
| The main risk I see is that the 7x signal amplification
| happens very aggressively. Even with a gentle initialization,
| you'd likely need very strict gradient clipping or a tiny
| learning rate on those new routing matrices to prevent them
| from blowing up the pre-trained features in the first few
| steps.
|
| Also, I think there's a mix-up here between mHC (this paper,
| expressivity) and MLA (latent attention, which provides the
| massive context efficiency). mHC doesn't save memory, but it
| might make the model 'smarter' per parameter.
| solarkraft wrote:
| You're right, I totally mixed this up with MLA.
| yorwba wrote:
| https://arxiv.org/abs/2512.24880 was published less than two
| weeks ago, which should explain why it's not more common yet.
| And it's not that amazing either. It's a slight quality
| improvement for a slight increase in cost. It's not even clear
| to me whether it pays for itself.
| solarkraft wrote:
| My bad, I took this as something Multi-head Latent Attention
| (MLA) related.
| graemefawcett wrote:
| I think the biggest benefit is bandwidth more so than
| efficiency. This gives you multiple streams to mux which and a
| means to control their mixing.
|
| The biggest innovation I think may have been accidental. The
| doubly stochastic matrix implements conservation on the signal
| stream.
|
| Treating the signal like the information it is as we do in any
| other domain is crucial for maintaining its coherence. We don't
| allow a network router to generate more packets than it
| receives for the same reason.
| sbondaryev wrote:
| Nice visualization of the residual connections. Is the animated
| svg manually created or programmatically generated? What tools
| did you use?
| taykolasinski wrote:
| Thanks! Manually created Astro components with inline SVG and
| CSS animations.
| cpldcpu wrote:
| May be worth pointing out, that this is not the first residual
| connection innovation to be in production.
|
| Gemma 3n is also using a low-rank projection of the residual
| stream called LAuReL. Google did not publicize this too much, I
| noted it when poking around in the model file.
|
| https://arxiv.org/pdf/2411.07501v3
|
| https://old.reddit.com/r/LocalLLaMA/comments/1kuy45r/gemma_3...
|
| Seems to be what they call LAuReL-LR in the paper, with D=2048
| and R=64
| taykolasinski wrote:
| This is a fantastic catch. I hadn't realized Gemma 3n was
| already shipping with a variant of this in production.
|
| It feels like we are entering the era of residual stream
| engineering. For a long time, the standard x + F(x) additive
| backbone was treated as untouchable. Now, between mHC (weighted
| scaling) and LAuReL (low-rank projections), labs are finally
| finding stable ways to make that signal path more dynamic.
|
| I'm curious if the Low-Rank constraint in LAuReL acts as a
| natural stabilizer against the gradient explosion I saw with
| unconstrained hyper-connections.
|
| Thanks for the paper link, definitely reading that tonight.
| cpldcpu wrote:
| Thanks! Would be quite interesting to see how this fares
| compared to mHC.
|
| I noted that LAuReL is cited in the mHC paper, but they refer
| to it as "expanding the width of the residual stream", which
| is rather odd.
| Scene_Cast2 wrote:
| I implemented this for a toy 8M ViT-style model. Got neutral
| results. This is just an anecdote and is not representative - I
| think mHC will help with larger parameter sizes and larger token
| counts.
| taykolasinski wrote:
| That's interesting.
|
| I suspect your intuition about scale is correct. The
| theoretical benefit of mHC is that it acts as a sort of relief
| valve/router for information flow in very deep/wide networks
| where the standard residual bottleneck becomes an issue. At 8M
| params, the standard residual stream is likely already
| perfectly adequate, so mHC might just be adding parameter
| overhead without solving a real signal propagation problem yet.
|
| Quick question on your run: did you see the signal
| amplification/instability I saw (values growing during the
| forward pass)? or was it stable for you, just neutral on loss?
| Scene_Cast2 wrote:
| My baseline was non-HC "vanilla" residuals; I didn't do a
| meaningful HC run to compare.
|
| My application has some particularities (important and easy
| to identify per-token signals) that result in values growing
| (about 3x to 10x) through layers even in the baseline.
| theschwa wrote:
| Between the clear writing and the diagrams, this was a great
| write up. I had actually skipped reading up on mHC as it sounded
| like it was going to take some time to grok, but this made it
| immediately approachable. I hope you do more write ups like this
| in the future.
| roywiggins wrote:
| imho the prose is very ChatGPT unfortunately
| E-Reverance wrote:
| > Residual connections are more than a trick to help
| gradients flow. They're a conservation law.
|
| > Not a hack, not a trick. A principled constraint that makes
| the architecture work at scale.
| DoctorOetker wrote:
| yes this reads like classic intellectual fellicitatio
| jszymborski wrote:
| OK, I thought I was reading too much into it but those same
| sentences also jumped out for me
| in-silico wrote:
| Why can't you just leave H_res as the identity matrix (or just
| not use it at all)? In that case, the model is basically a ResNet
| again and you don't need to worry about exploding/vanishing
| gradients from H_res.
|
| I would think that H_post and H_pre could cover the lost
| expressiveness.
| john-titor wrote:
| great write up. it's been a while since I had the pleasure to
| read a straightforward blog post about ML tricks that feel
| genuinely applicable to many use cases.
___________________________________________________________________
(page generated 2026-01-12 23:00 UTC)