[HN Gopher] Hierarchical Modeling (H-Nets)
       ___________________________________________________________________
        
       Hierarchical Modeling (H-Nets)
        
       Author : marviel
       Score  : 46 points
       Date   : 2025-07-15 19:35 UTC (3 hours ago)
        
 (HTM) web link (cartesia.ai)
 (TXT) w3m dump (cartesia.ai)
        
       | marviel wrote:
       | > H-Net demonstrates three important results on language
       | modeling:
       | 
       | > 1. H-Nets scale better with data than state-of-the-art
       | Transformers with BPE tokenization, while learning directly from
       | raw bytes. This improved scaling is even more pronounced on
       | domains without natural tokenization boundaries, like Chinese,
       | code, and DNA.
       | 
       | > 2. H-Nets can be stacked together to learn from deeper
       | hierarchies, which further improves performance.
       | 
       | > 3. H-Nets are significantly more robust to small perturbations
       | in input data like casing, showing an avenue for creating models
       | that are more robust and aligned with human reasoning.
        
         | marviel wrote:
         | https://arxiv.org/pdf/2507.07955
         | 
         | paper
        
       | modeless wrote:
       | I don't know if this is the one but something like this is
       | clearly the future IMO. We need more levels of hierarchy to
       | efficiently generalize to longer sequences with high level
       | structure. Back when Byte Latent Transformers came out I thought
       | extending the idea to more levels of hierarchy was the way to go,
       | and this seems to be basically that?
       | 
       | Another article about H-Nets: https://main-
       | horse.github.io/posts/hnet-inf/
        
       | cs702 wrote:
       | I've only skimmed the paper, but it looks interesting and
       | credible, so I've added it to my reading list.
       | 
       | Thank you for sharing on HN!
       | 
       | ---
       | 
       | EDIT: The hierarchical composition and routing aspects of this
       | work vaguely remind me of
       | https://github.com/glassroom/heinsen_routing/ but it has been a
       | while since I played with that. UPDATE: After spending a bit more
       | time on the OP, it's different, but the ideas are related, like
       | routing based on similarity.
        
         | marviel wrote:
         | No problem! I'm still parsing it myself, but it seems promising
         | in theory, and the result curves are impressive.
        
       | gdiamos wrote:
       | How does it handle images?
        
         | marviel wrote:
         | it mentions native multimodality somewhere in either the Arxiv
         | or post -- seems like it might handle it well?
        
         | miven wrote:
         | As far as I understand the "chunking" of input bytes is learned
         | completely end to end, so it's basically up to the model to
         | figure out how to most efficiently delineate and aggregate the
         | information from the inputs according to the patterns provided
         | to it during training.
         | 
         | Since it's end to end this allows them to apply this process
         | not only to raw byte encodings but basically representations of
         | any level, such as stacking two stages of aggregation one after
         | another.
         | 
         | So in principle they could either let the model do its thing on
         | raw bytes of an image or alternatively maybe cut it up into
         | tiny patches ViT-style and feed that to their H-Net.
         | 
         | I wonder how hard would it be to adapt chunking to work in 2D
         | and what would that even look like.
         | 
         | Some other notes on how multimodal inputs could be handled
         | using this architecture are mentioned in Albert Gu's (one of
         | the author's) blog, although only briefly, there's still much
         | to figure out it would seem:
         | https://goombalab.github.io/blog/2025/hnet-future/#alternati...
        
           | marviel wrote:
           | Thanks for sharing this blog post is a great speculative
           | deep-dive.
        
       | aeon_ai wrote:
       | Seems likely to be relevant for memory
       | formation/consolidation/management.
       | 
       | Big, if so.
        
       | cubefox wrote:
       | As Mamba didn't make it, will H-Nets replace Transformers?
        
         | marviel wrote:
         | It's meant to replace the BPE tokenizer piece, so it isn't a
         | full Language Model by itself.
         | 
         | In fact in Gu's blog post (linked in a post below) it's
         | mentioned that they created a Mamba model that used this in
         | place of the tokenizer.
        
       | vannevar wrote:
       | >The best AI architectures in use today treat all inputs equally.
       | 
       | Doesn't this architecture also treat all inputs equally? It seems
       | like an encoder that preprocesses the input by inferring
       | hierarchy. But don't all models essentially do that while
       | training?
        
       ___________________________________________________________________
       (page generated 2025-07-15 23:00 UTC)