[HN Gopher] HMT: Hierarchical Memory Transformer for Long Contex...
       ___________________________________________________________________
        
       HMT: Hierarchical Memory Transformer for Long Context Language
       Processing
        
       Author : jasondavies
       Score  : 65 points
       Date   : 2024-05-17 11:19 UTC (11 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | cs702 wrote:
       | Code: https://github.com/OswaldHe/HMT-pytorch
       | 
       | This looks really interesting. I've added the paper to my reading
       | list and look forward to playing with the code. I'm curious to
       | see what kinds of improvements we can get by agumenting
       | Transformers and other generative sequence models with this and
       | other mechanisms implementing hierarchical memory.[a]
       | 
       | Shouldn't the authors cite the work by Jeff Hawkins et al at
       | Numenta? Hawkins has been proposing AI models with hierarchical
       | temporal memory for a long time.[b] I can't help but wonder if
       | there is a way, somehow, to incorporate his work and ideas in
       | Transformers and other generative sequence models.
       | 
       | We sure live in interesting times!
       | 
       | ---
       | 
       | [a] In the past, I've experimented with mechanisms that add
       | memory to Transformers, but never with _hierarchy_.
       | 
       | [b] https://en.wikipedia.org/wiki/Hierarchical_temporal_memory
        
         | anthonyskipper wrote:
         | I thought Hawkins's book "On Intelligence" was amazing. It's a
         | bit wild how close things have followed to the direction he
         | laid out.
        
       ___________________________________________________________________
       (page generated 2024-05-17 23:01 UTC)